TestComplete by Smart Bear Software

Smart Bear Software has announced the newest version of their TestComplete package. This has a number of tools for testing. I think it aims to allow people with limited technical skills to become productive in testing. However there are some hooks in there for advanced users as well.

There is a Test Visualizer that let's you set up tests by clicking around in the user interface. There is also another tool that let's you type in free form text of the test. It then reads your text and generates test scripts. That seems pretty amazing.

The tool also has a Data-Driven Loop Wizard which let's you import data from a file to create tests. Finally for the seasoned testers, there is the JavaBridge. It allows you to use an API to grab data from within a Java application. That is sweet. If I get into more Java development, that feature alone could be worth the cost. Pricing starts at a grand for the Standard version.

The Many Faces of Quality

I read an insightful article on the different quality professions out there. These include quality assurance, quality control, verification, and validation. These terms are often confused and used imprecisely. So what exactly do they refer to.

Quality Assurance deals with processes. That does not necessarily mean testing. In fact, QA can involve review of a design or requirements document. This is in line with verification, which checks that development was conducted correctly.

Quality control deals with the product of software development. This is achieved through the execution of tests. It is closely related to validation, which is the execution of code.

Visual Studio Ultimate

There are a number of tools in the Visual Studio Ultimate Edition that help the test mission. These tools are included in the price of Microsoft's MSDN subscription. Some of these tools include Test Manager and Lab Manager.

Test Manager is meant for non-techies to developer test cases. This helps eliminate non-technical testers clicking buttons manually.

Lab Manager gives you an environment to conduct tests. It works with Intellitrace and Hyper-V. Unfortunately Lab Manager does not support Silverlight applications. This tools does require a lot of hardware to run on.

Unit Test Worth

Just read a rant that unit tests are all but worthless. Is this true? Let's start at the beginning. Unit tests are supposed to validate independent pieces. They attack at the function level. Code with many dependencies do not work well with unit tests.

Unit tests may help find some small errors. However isn't that what debugging is for? I am speaking hypothetically here. You are supposed to be able to refactor your code and run your unit tests to check whether it still works. Unfortunately unit tests may be too brittle to still work after refactoring.

Developers are optimists. Therefore their unit tests may just choose the easy cases. That is not much of a return on investment. However you should use a tool like Jester. It rewrites your Java code to tweak the logic a bit. If you Jester your code, and the unit tests still past, your tests are not doing much.

Acceptance tests are better predictors of whether your code works. These can be employed after a refactoring to make sure the code still works. Some people still swear by unit tests. They have stories of doing 100% coverage. Then the eventual deployment had no problems at all. I think the jury is still out on unit test effectiveness.

Crowdsourcing

You have heard of outsourcing to lower costs to testing. But have you heard of crowdsourcing? This is a new spin where a company has independent testers all over the world. You company contracts with the umbrella company to get your tests executed. You only pay for defects found.

This method has the advantage that you do not need to set up the outsourced testers. A company has already done that for you. You also only have to pay for results you verify. A test cycle can run you as little as $5k. I just wonder what quality of tests you can get for such a small price. It could be worth a try.

Risk Based Testing

How do you prioritize testing? You can't do everything. So you should spend your time on tests that have the most bang for your buck. Risk based testing attempts to solve this problem.

There are two dimensions to risk. One is the technical dimension. This is where the developers use new technology. Another dimension is the business one. Where in the apps is there a high cost to the operations if there is failure?

You can use the two risk dimensions to prioritize testing. The areas that involve both a technical and business risk need the most testing resources. Other areas that have either one or the other dimensions of risk should be tested next. The low risk areas can be saved for last or skipped entirely. The result is that you spent your time wisely.

Silverlight Testing

How do you test a Silverlight application? That was the topic of a blog post I read recently. One way is to construct a Silverlight test harness. This is where you modify the application that is being tested. You put in hooks to work with your app tester. Let’s discuss this idea.

Currently Silverlight 4 is in beta. The latest release version is Silverlight 3. You can code up a test application. A good programming language for the test app is C#. This test app can send messages to the app you are testing. These messages interrogate the application for its current state.

By default a Silverlight app will take up the whole screen. So you need to make sure your test app can stay on top, or share the screen real estate with the app being testing. The use of Local Messaging is a simple way to communicate with the app being tested.

Simplicity is the main idea here. You do have to modify the app being tested. This feels a little more like unit testing to me. However it is a light version of it. There are alternatives to testing Silverlight apps. Perhaps I shall delve into those other techniques in a future post

Beauty in Testing

My favorite test magazine had a guest editor for last month's issue. He picked an excerpt from his book on beautiful testing for one of the articles. I guess when you are the boss, you can give yourself a competitive advantage. Let's see what this guy had to say about the beauty in testing.

You got to know who you are testing for. You must know your stakeholders. Otherwise you cannot please them. This can be anybody from developers to managers to support staff to users. Many of these people could have unrealistic expectations of your work. Get used to it.

It can be hard to impress the stakeholders. They want more than you finding a lot of bugs. Developers may want you to do the opposite. Metrics can help you quantify your success. I just want testers to find problems early so we can fix them on the cheap.

Outsourced Testing

It is becoming popular to outsource testing. How do you make sure it turns out well? You might need to appoint a third party adviser to oversee the activity. You should also define what your goals are. Do you want a little extra help to test? You alternatively may be looking for someone else to do all the testing. Or perhaps there is a skill lacking in your test organization. Define your objectives up front.

Choose outsourced help that has a process which matches your own. This results in less pain. To ensure you don't get ripped off, try to negotiate as fixed price contract for the outsourcing. Then the risk is put in the hands of the outsourcing company.

Exercise caution when you grow the outsourcing effort. Also watch out for expensive tool choices by the outsourcing company. They may not have your best interests at heart. More importantly, the costs of the tool might be prohibitive. Luckily our own project has a good sized test team. They also have some really good talent. So far they have only outsourced a little work to other employees in our own company. They have been short term engagements. But you never know. We might look outside for some help to test.

Testing Trends

What's new in the state of software testing today? That was the question posed to the editor of Software Test and Performance magazine. It is difficult to get the top three. There are many important topics in modern testing. What did the editor choose?

One is the creation of testing centers of excellence. Next is the use of test outsourcing. The third is delivering value to stakeholders using a technique called beautiful testing. These topics are the contents of the latest issue of the magazine.

I will be talking more about these topics in the future as they are the state of the test art.

Web 2.0 Testing

There is a new breed of applications that have been labeled Web 2.0. They require a different test strategy. Web 2.0 apps are created by the users themselves. For example, users can create mashups where they take different apps and integrate them into a new system. This complicates verification. What is a tester to do?

The first thing is that you need to use the software like a user for a while. You can also ensure that actions get logged to disk. That way you can mine the log. The log can be used to drive future tests. You can use it to set up regression tests.

You got to do all kinds of data inputs validation testing. Sure you can try some special characters as input. But you should also try a whole range of weird characters. Try to simulate a cross site scripting attack. Your app better be on lock down for this.

A good goal is to test small feature sets. You should also clone your production data to seed your test database. If necessary, strip out any private data from the set. Above all ensure you can roll back to a stable release at any given time. This is the web. Things happen fast.