Crowdsourcing

You have heard of outsourcing to lower costs to testing. But have you heard of crowdsourcing? This is a new spin where a company has independent testers all over the world. You company contracts with the umbrella company to get your tests executed. You only pay for defects found.

This method has the advantage that you do not need to set up the outsourced testers. A company has already done that for you. You also only have to pay for results you verify. A test cycle can run you as little as $5k. I just wonder what quality of tests you can get for such a small price. It could be worth a try.

Risk Based Testing

How do you prioritize testing? You can't do everything. So you should spend your time on tests that have the most bang for your buck. Risk based testing attempts to solve this problem.

There are two dimensions to risk. One is the technical dimension. This is where the developers use new technology. Another dimension is the business one. Where in the apps is there a high cost to the operations if there is failure?

You can use the two risk dimensions to prioritize testing. The areas that involve both a technical and business risk need the most testing resources. Other areas that have either one or the other dimensions of risk should be tested next. The low risk areas can be saved for last or skipped entirely. The result is that you spent your time wisely.

Silverlight Testing

How do you test a Silverlight application? That was the topic of a blog post I read recently. One way is to construct a Silverlight test harness. This is where you modify the application that is being tested. You put in hooks to work with your app tester. Let’s discuss this idea.

Currently Silverlight 4 is in beta. The latest release version is Silverlight 3. You can code up a test application. A good programming language for the test app is C#. This test app can send messages to the app you are testing. These messages interrogate the application for its current state.

By default a Silverlight app will take up the whole screen. So you need to make sure your test app can stay on top, or share the screen real estate with the app being testing. The use of Local Messaging is a simple way to communicate with the app being tested.

Simplicity is the main idea here. You do have to modify the app being tested. This feels a little more like unit testing to me. However it is a light version of it. There are alternatives to testing Silverlight apps. Perhaps I shall delve into those other techniques in a future post

Beauty in Testing

My favorite test magazine had a guest editor for last month's issue. He picked an excerpt from his book on beautiful testing for one of the articles. I guess when you are the boss, you can give yourself a competitive advantage. Let's see what this guy had to say about the beauty in testing.

You got to know who you are testing for. You must know your stakeholders. Otherwise you cannot please them. This can be anybody from developers to managers to support staff to users. Many of these people could have unrealistic expectations of your work. Get used to it.

It can be hard to impress the stakeholders. They want more than you finding a lot of bugs. Developers may want you to do the opposite. Metrics can help you quantify your success. I just want testers to find problems early so we can fix them on the cheap.

Outsourced Testing

It is becoming popular to outsource testing. How do you make sure it turns out well? You might need to appoint a third party adviser to oversee the activity. You should also define what your goals are. Do you want a little extra help to test? You alternatively may be looking for someone else to do all the testing. Or perhaps there is a skill lacking in your test organization. Define your objectives up front.

Choose outsourced help that has a process which matches your own. This results in less pain. To ensure you don't get ripped off, try to negotiate as fixed price contract for the outsourcing. Then the risk is put in the hands of the outsourcing company.

Exercise caution when you grow the outsourcing effort. Also watch out for expensive tool choices by the outsourcing company. They may not have your best interests at heart. More importantly, the costs of the tool might be prohibitive. Luckily our own project has a good sized test team. They also have some really good talent. So far they have only outsourced a little work to other employees in our own company. They have been short term engagements. But you never know. We might look outside for some help to test.

Testing Trends

What's new in the state of software testing today? That was the question posed to the editor of Software Test and Performance magazine. It is difficult to get the top three. There are many important topics in modern testing. What did the editor choose?

One is the creation of testing centers of excellence. Next is the use of test outsourcing. The third is delivering value to stakeholders using a technique called beautiful testing. These topics are the contents of the latest issue of the magazine.

I will be talking more about these topics in the future as they are the state of the test art.

Web 2.0 Testing

There is a new breed of applications that have been labeled Web 2.0. They require a different test strategy. Web 2.0 apps are created by the users themselves. For example, users can create mashups where they take different apps and integrate them into a new system. This complicates verification. What is a tester to do?

The first thing is that you need to use the software like a user for a while. You can also ensure that actions get logged to disk. That way you can mine the log. The log can be used to drive future tests. You can use it to set up regression tests.

You got to do all kinds of data inputs validation testing. Sure you can try some special characters as input. But you should also try a whole range of weird characters. Try to simulate a cross site scripting attack. Your app better be on lock down for this.

A good goal is to test small feature sets. You should also clone your production data to seed your test database. If necessary, strip out any private data from the set. Above all ensure you can roll back to a stable release at any given time. This is the web. Things happen fast.