There is a Test Visualizer that let's you set up tests by clicking around in the user interface. There is also another tool that let's you type in free form text of the test. It then reads your text and generates test scripts. That seems pretty amazing.
The tool also has a Data-Driven Loop Wizard which let's you import data from a file to create tests. Finally for the seasoned testers, there is the JavaBridge. It allows you to use an API to grab data from within a Java application. That is sweet. If I get into more Java development, that feature alone could be worth the cost. Pricing starts at a grand for the Standard version.
Quality Assurance deals with processes. That does not necessarily mean testing. In fact, QA can involve review of a design or requirements document. This is in line with verification, which checks that development was conducted correctly.
Quality control deals with the product of software development. This is achieved through the execution of tests. It is closely related to validation, which is the execution of code.
Test Manager is meant for non-techies to developer test cases. This helps eliminate non-technical testers clicking buttons manually.
Lab Manager gives you an environment to conduct tests. It works with Intellitrace and Hyper-V. Unfortunately Lab Manager does not support Silverlight applications. This tools does require a lot of hardware to run on.
Unit tests may help find some small errors. However isn't that what debugging is for? I am speaking hypothetically here. You are supposed to be able to refactor your code and run your unit tests to check whether it still works. Unfortunately unit tests may be too brittle to still work after refactoring.
Developers are optimists. Therefore their unit tests may just choose the easy cases. That is not much of a return on investment. However you should use a tool like Jester. It rewrites your Java code to tweak the logic a bit. If you Jester your code, and the unit tests still past, your tests are not doing much.
Acceptance tests are better predictors of whether your code works. These can be employed after a refactoring to make sure the code still works. Some people still swear by unit tests. They have stories of doing 100% coverage. Then the eventual deployment had no problems at all. I think the jury is still out on unit test effectiveness.
This method has the advantage that you do not need to set up the outsourced testers. A company has already done that for you. You also only have to pay for results you verify. A test cycle can run you as little as $5k. I just wonder what quality of tests you can get for such a small price. It could be worth a try.
There are two dimensions to risk. One is the technical dimension. This is where the developers use new technology. Another dimension is the business one. Where in the apps is there a high cost to the operations if there is failure?
You can use the two risk dimensions to prioritize testing. The areas that involve both a technical and business risk need the most testing resources. Other areas that have either one or the other dimensions of risk should be tested next. The low risk areas can be saved for last or skipped entirely. The result is that you spent your time wisely.
Currently Silverlight 4 is in beta. The latest release version is Silverlight 3. You can code up a test application. A good programming language for the test app is C#. This test app can send messages to the app you are testing. These messages interrogate the application for its current state.
By default a Silverlight app will take up the whole screen. So you need to make sure your test app can stay on top, or share the screen real estate with the app being testing. The use of Local Messaging is a simple way to communicate with the app being tested.
Simplicity is the main idea here. You do have to modify the app being tested. This feels a little more like unit testing to me. However it is a light version of it. There are alternatives to testing Silverlight apps. Perhaps I shall delve into those other techniques in a future post
You got to know who you are testing for. You must know your stakeholders. Otherwise you cannot please them. This can be anybody from developers to managers to support staff to users. Many of these people could have unrealistic expectations of your work. Get used to it.
It can be hard to impress the stakeholders. They want more than you finding a lot of bugs. Developers may want you to do the opposite. Metrics can help you quantify your success. I just want testers to find problems early so we can fix them on the cheap.
Choose outsourced help that has a process which matches your own. This results in less pain. To ensure you don't get ripped off, try to negotiate as fixed price contract for the outsourcing. Then the risk is put in the hands of the outsourcing company.
Exercise caution when you grow the outsourcing effort. Also watch out for expensive tool choices by the outsourcing company. They may not have your best interests at heart. More importantly, the costs of the tool might be prohibitive. Luckily our own project has a good sized test team. They also have some really good talent. So far they have only outsourced a little work to other employees in our own company. They have been short term engagements. But you never know. We might look outside for some help to test.
One is the creation of testing centers of excellence. Next is the use of test outsourcing. The third is delivering value to stakeholders using a technique called beautiful testing. These topics are the contents of the latest issue of the magazine.
I will be talking more about these topics in the future as they are the state of the test art.
The first thing is that you need to use the software like a user for a while. You can also ensure that actions get logged to disk. That way you can mine the log. The log can be used to drive future tests. You can use it to set up regression tests.
You got to do all kinds of data inputs validation testing. Sure you can try some special characters as input. But you should also try a whole range of weird characters. Try to simulate a cross site scripting attack. Your app better be on lock down for this.
A good goal is to test small feature sets. You should also clone your production data to seed your test database. If necessary, strip out any private data from the set. Above all ensure you can roll back to a stable release at any given time. This is the web. Things happen fast.