Going the Distance

The customer encountered a problem while testing our application suite. We had just migrated to a new set of tools, and a new version of the database on the back end. It took a while for me to get around to researching the problem. Luckily I was able to duplicate the problem in a development environment. I tracked the source of the problem down to some bad test data. Then I checked the database where the customer performed their tests. They had the same problem there. The fix was to write a script to correct the data.

At first when I ran the script to correct the problem, it just bombed. It logged the problems it encountered. So I manually executed each of the statements in the script until I found what the problem was. That’s when I was done with the coding. It was time to test it out. Luckily I had a lot of data that was corrupt in my development environment. I ran the script and it fixed about 5000 records. I used a tool to view the corrected data. In general it looked pretty good. I was tempted to just ship the fix at this point since I had 9 other problem to research and resolve.

Instead I decided to take my unit tests a step further. I figured it would be easiest to write another script to verify all the records were updated correctly. It took about an hour to write this unit test script. That was not too much overhead considering the time it take to investigate and code a solution to the problem. I did find some exceptional scenarios in the script results when I churned the data with a script. I just checked all 5000 outputs records for correctness. At first I just output the number of problems the test script found. Then I actually wrote the problems out to a log for further analysis.

Here in development we always seem to be in a rush because we are understaffed and overworked. But here is the funny thing. If I sent out a script that did not fully work, I would waste a whole lot more time dealing with the repercussions. Why not spend a little more time and get it right the first time? I love to code. Writing a script to verify the unit test results is just a little more coding. So it is in fact fun work. Now just don’t tell the management staff that. Otherwise they will get the wrong idea.

Testing Database Code

Last night I read a blog entry entitled “How to Unit-Test Code That Interacts With a Database”. It stated that such testing is hard. But you still want to fully test the database interaction. The blog recommended that you connect to a separate database to do your unit testing. Each developer should ideally have their own database. In addition, the database should be reset to the state it started in prior to the unit test. The best database for testing is a small focused data set. Above all the unit test activity should not connect to the production database.

On our project we conduct unit testing on new code that we write, as well as on fixes that we produce. Our main test data set is a small but interesting combination of test cases for verification. In general most of our developers have their own schema for testing. This allows you to do whatever you want without impacting other developers. It also helps in that nobody else makes changes to the data in your database. In the past I have written elaborate unit test scripts which generate delete old test data, generate new test data, execute the tests, and report on the results. This is normally done when I write code that relies heavily on database interaction.

One of the best parts of the blog post I read was the large number of comments from readers. I will share some of them here for your benefit. Multiple people pointed out that the type of testing discussed here is integration testing rather than unit testing. Unit tests are concerned with logic or combination checks. Oracle users recommended the utPLSQL framework for Oracle. You can use the ut_setup and ut_teardown functions to take care of unit test data.

Some readers commented that you could treat the database changes as one big transaction, then roll it back after testing is done. This would definitely not work for my system. We commit transactions all over the place. We even have autonomous transactions which commit themselves without interacting with any outer transactions. You can do better than getting your own schema for testing. You can get a whole database installation of your own to conduct unit tests in. Express versions of Oracle and Microsoft SQL Server are free.

It appears that unit testing is hot in the blogosphere. This seems strange to me. But I am happy about it. Nobody seems to like unit testing where I work. Most developers seem to want to code, check it in, and ship it. I know I myself could use a more rigorous unit test habit.

Scripting

I read a stimulating blog post by Michiel de Mare entitled “Is Your Program Perfect”. The post was essentially a bunch of questions that made you consider whether there were any improvements you could make to a program to make it better. There was a section on testing that asked whether your program could be scripted from the command line. It also inquired whether your program exposes an API. I thought long and hard about these questions as I considered my projects own suite of applications. And I determined they were very weak in this area. The question now is how do we make changes to the programs to improve in these areas.

In the past I had played with WinRunner by Mercury Interactive. This tool pretty much allowed you to automate testing of any Windows application. The tests could be scripted and create easily by recording and playing back user actions. However this required you to purchase the expensive WinRunner tool. Furthermore this tool is going to be retired in a few years. It would be more economical and beneficial for our applications to provide better test and scripting support itself.

Currently one of our applications has an API that allows you to instantiate at least one of its screens programmatically. However it is a very cumbersome API. You have to create a data on the file system or in shared memory. It also requires clunky windows messages to initiate the actions. This support only exists in one of the five applications in our suite. And this support is for one very specialized part of the app. It would be great if a more comprehensive and simple API could be produce to control the application for test purposes. I envision this to be of great use during the unit test phase.

Obviously it will take a while to add the hooks to our applications. But we could start small and work our way up from there. Perhaps the best place to start would be to automate access to the new feature we are currently coding into the application. We should be very familiar with the design and implementation of the new code. And we already have time in the schedule to work on these pieces. During the unit test phase of these features we could add in the automated support and actually use it for unit testing.

All of this introspection was due to a small piece of Michiel’s blog post to question whether your software is at its best. Opening our apps up to automation may go a long way to simplifying our unit test strategy.

White Space

Our client has a system acceptance test team. Recently they approached me regarding a problem our load software had last year. They knew it had something to do with the input data containing spaces. And they were trying to determine the impact on the system, and whether they needed to do any additional testing of it this year. I gave this problem some thought and told the test team what I knew.

There is a certain field in one of our input files that is 4 characters in length. We are required to read this field, trim off any white spaces, and store the result in the database. This had worked properly for a number of years. A developer rewrote the load routine using a new tool. They did not correctly follow the requirements and trim the white space from the field. As a result, post load processing could not match the loaded value up with other values that had trimmed. Additionally the application that queries the data could not form the match either.

It took quite a bit to resolve this problem. We had to correct the loaded data for millions of records. Then we had to run the data back through post loads processing. Finally the original load software was corrected to deal properly with future files loaded. It was this last part that the test team wanted to verify. It was good that they are paying attention. Since we only load this particular file once a year, nobody other than the developer tested the fix to the loads code.

The test team was happy to hear the information I provided them. They were immediately running off identifying test cases and planning to generate some input file data to check out our code. We have an internal test team at our company. For some reason I doubt they also have this type of insight and premonition to verify that development follows through on the fixes we produce. Its always easier to deal with problems during test before the software goes out into a production environment.

Access Levels

Our application suite supports a number of access levels. Each user is assigned one of five access levels. Each level has a uniquely defined behavior in the application. In general, the higher the level, the more you can do. But higher levels do not always have the ability to do everything a lower level user can do.

There are a number of times when people believe the application is not behaving correctly. Frequently this is due to an ignorance of what each access level is allowed to do. Normally this happens with the newer developers or testers. It usually takes a senior developer to sort out the problem and trace it back to an access level issue.

I have been working with a new tester recently. He is testing out the application that I am responsible for. He has found some weird problems in my application. There is a lot of interaction between us since I often need to know exactly how to reproduce a problem, or what specific data the tester is using to validate the application.

The other day I asked the tester to reproduce the problem he had documented. He had to do a lot of work to reset the application access levels to the ones used when he encountered the problem. He told me that he was doing a lot of testing in the application at each of the access levels. This was recommended to him by a senior tester. This encouraged me. I have to admit that, while conducting my own unit tests, I did not test all the functionality at each of the access levels.

This emphasizes the differences between developers and testers conducting verification. A developer may be quick to declare success once one test passes. Or sometimes a developer will assume that some tests will pass based on the results of one other test. A tester may also have this opinion. But they will actually perform the tests to verify the hypothesis. However a developer may be quick to skip steps that seem repetitive and boring.

It is a good thing that we have an internal test team to keep development honest. Their absence would not be a total loss. Our customer has their own system acceptance test team. And since we are migrating to all new development tools this year, the users themselves have scheduled a functionality test of our upgraded application. I have the feeling that I am going to be researching a lot of trouble tickets this year. Since I like fixing bugs, this is a good thing.

Time to Market

I have a software development business on the side of my day job. Recently I have focused my business on writing small programs that would be of interest to hackers. The programs I develop are posted on my Black of Hat blog. Currently I am giving my programs away for free to try to increase readership of my new blog.

The interesting thing about these programs is that I have not been subjecting them to rigorous testing. Often times the goal is to get some software posted out on my blog quickly. So I run a couple quick unit tests, fix the obvious bugs, and warn my readers that the software may be in an early state.

My goal is to write many useful programs quickly. Of course I plan to improve and fix any bugs that pop up. But there is a definite lack of V&V being done. The question is whether formal test methods would be of any value in this fast paced environment. I don't get any direct penalty for releasing a program that crashes and burns. There may be some unhappy readers which turn away from my blog. However I am considering the faster time to market to be the prime objective.

Perhaps I could take my next program developed by my side business, apply more thorough testing to it, and try to measure the outcome subjectively. I am not against going through a formal test cycle for these programs. The objective is to gain the most amount of readers so that I can monetize my blog. Any way I can do that more effectively is worthwhile. Besides I feel as if I may be developing bad habits in my side business software development. It would be easier if I use the same methods in both my day and night jobs.

Test to Dev to Test

Over my career, I have met a number of software developers that have become testers. These testers often tell me that they got tired of working the long hours in development. They say that testing, while challenging, is less stressful and easier that writing code. And these are usually the testers that are creating scripts or writing code to test data.

I have also met developers who said they got trapped in the test environment. First it starts out with the need to write a high profile master test plan. Then who better to oversee the test execution than the author. Pretty soon you get known as somebody proficient who can run tests, and then you are pigeon holed into being a tester.

There is one other scenario which is not encouraging. Sometimes developers who just do not cut it as programmers get reassigned to the test team. It is almost as if this is a demotion. This seems strange as the top talent in the test world seem to make good money. Maybe it is just a psychological effect.

The reverse path appears to be difficult if not impossible. I have not met many developers who said they started out as a tester. There was one guy I knew who was pretty good with code. He wrote a lot of scripts to generate test data. But he was still officially assigned to the test team. Maybe there are such developers, but they don't want to talk about their test background. Still it seems a very rare move. It looks like a one way street between development and testing roles.