Silverlight Testing

How do you test a Silverlight application? That was the topic of a blog post I read recently. One way is to construct a Silverlight test harness. This is where you modify the application that is being tested. You put in hooks to work with your app tester. Let’s discuss this idea.

Currently Silverlight 4 is in beta. The latest release version is Silverlight 3. You can code up a test application. A good programming language for the test app is C#. This test app can send messages to the app you are testing. These messages interrogate the application for its current state.

By default a Silverlight app will take up the whole screen. So you need to make sure your test app can stay on top, or share the screen real estate with the app being testing. The use of Local Messaging is a simple way to communicate with the app being tested.

Simplicity is the main idea here. You do have to modify the app being tested. This feels a little more like unit testing to me. However it is a light version of it. There are alternatives to testing Silverlight apps. Perhaps I shall delve into those other techniques in a future post

Beauty in Testing

My favorite test magazine had a guest editor for last month's issue. He picked an excerpt from his book on beautiful testing for one of the articles. I guess when you are the boss, you can give yourself a competitive advantage. Let's see what this guy had to say about the beauty in testing.

You got to know who you are testing for. You must know your stakeholders. Otherwise you cannot please them. This can be anybody from developers to managers to support staff to users. Many of these people could have unrealistic expectations of your work. Get used to it.

It can be hard to impress the stakeholders. They want more than you finding a lot of bugs. Developers may want you to do the opposite. Metrics can help you quantify your success. I just want testers to find problems early so we can fix them on the cheap.

Outsourced Testing

It is becoming popular to outsource testing. How do you make sure it turns out well? You might need to appoint a third party adviser to oversee the activity. You should also define what your goals are. Do you want a little extra help to test? You alternatively may be looking for someone else to do all the testing. Or perhaps there is a skill lacking in your test organization. Define your objectives up front.

Choose outsourced help that has a process which matches your own. This results in less pain. To ensure you don't get ripped off, try to negotiate as fixed price contract for the outsourcing. Then the risk is put in the hands of the outsourcing company.

Exercise caution when you grow the outsourcing effort. Also watch out for expensive tool choices by the outsourcing company. They may not have your best interests at heart. More importantly, the costs of the tool might be prohibitive. Luckily our own project has a good sized test team. They also have some really good talent. So far they have only outsourced a little work to other employees in our own company. They have been short term engagements. But you never know. We might look outside for some help to test.

Testing Trends

What's new in the state of software testing today? That was the question posed to the editor of Software Test and Performance magazine. It is difficult to get the top three. There are many important topics in modern testing. What did the editor choose?

One is the creation of testing centers of excellence. Next is the use of test outsourcing. The third is delivering value to stakeholders using a technique called beautiful testing. These topics are the contents of the latest issue of the magazine.

I will be talking more about these topics in the future as they are the state of the test art.

Web 2.0 Testing

There is a new breed of applications that have been labeled Web 2.0. They require a different test strategy. Web 2.0 apps are created by the users themselves. For example, users can create mashups where they take different apps and integrate them into a new system. This complicates verification. What is a tester to do?

The first thing is that you need to use the software like a user for a while. You can also ensure that actions get logged to disk. That way you can mine the log. The log can be used to drive future tests. You can use it to set up regression tests.

You got to do all kinds of data inputs validation testing. Sure you can try some special characters as input. But you should also try a whole range of weird characters. Try to simulate a cross site scripting attack. Your app better be on lock down for this.

A good goal is to test small feature sets. You should also clone your production data to seed your test database. If necessary, strip out any private data from the set. Above all ensure you can roll back to a stable release at any given time. This is the web. Things happen fast.

More on Fuzzing

Development magazines are paying more attention to fuzzing. Fuzz testing is subjecting an application to random data to see what breaks. This is applicable to web apps. It can be part of a comprehensive security test. Fuzz testing let’s you try inputs that stimulate paths a tester might not normally think about, or have time to manually perform.

There are free fuzz test generators out there. Some are Peach Fuzzer, MiniFuzz, and Spike. There are also some commercial products to assist with fuzzing as well. But let’s get back to why fuzzing is a good idea. Normally you test positive aspects of requirements. However you should also be testing for things not covered by any requirement. This is where fuzz testing can come in real handy.

I am a developer. Right now I am not employing fuzz testing yet. However I hope the independent test team on my project is hot on the technique.

Pairwise Testing

Testers often need to verify systems with many related variables. This can result in a huge amount of test cases. Often the combinations can be out of control. However a technique called Pairwise Testing can reduce the amount of test cases while still covering all scenarios. The hard part of this is generating the minimal set of cases. This is a good time to use a program to do the hard work.

One pairwise test case generator is Pairwise Independent Combinatorial Testing (PICT). This product is based on a tool that Microsoft themselves use. There is a recent new tool called QICT which accomplishes the same task. The beauty of QICT is that the source code is provided free of charge.

Pairwise testing is good for many test scenarios. Some good examples are testing databases and configuration data. I know that I often have an onslaught of test cases that I think need to be executed to provide full coverage. I might just have to use a tool such as PICT to make my job easier. This can potentially reduce the test time necessary, and improve my test throughput.