True Testing

A manager on my project wanted to know how some things worked in one of our applications. They needed to speak with a customer. So this information needed to be very accurate. I kind of knew how the application worked. However I did not know all the details for sure. We have a lot of documentation that describes the most intricate details of the applications. Some of this documentation is old. My instinct told me not to trust the documentation. I had no other alternative than to reverse engineer the application behavior through a series of tests.

I spent a lot of time conducting different scenarios to test. It was good that I had some idea on how the application worked. That way I could choose tests where I was not sure of the results. I could also infer a lot from just a few tests results. This was not an exhaustive search of all the possibilities. That would have taken far too long. It took about half a day to get the data I needed.

This technique may not work for everybody. You have to make sure you really get a firm understanding of what is going on behind the scenes. However I trust this more than consulting some documentation or even reviewing the source code. In my experience, it is best to trust nothing. I know that sounds a bit like the X Files. But it is the safest approach.

Now I can say with confidence how the application behaves for some certain behavior. If anybody wants to challenge me, I can ask them where they got their information. Is that just the way it is supposed to be? Did they hear it from somebody a long time ago? Did they read this in some document? I can top all of that. I just tested the actual application out. You can’t beat that kind of experience.

Good Test Data

You need reliable test data to conduct independent verification. This is more complicated than it seems for some systems. The Software Maintenance blog demonstrates that sometimes even developers have trouble creating test data.

Ideally you can run the system like it operates in a production environment. The closer to the real world you make it, the better the quality of your test data. However this is not always feasible. Sometimes you need to simulate the real world with generated test data. It is these circumstances where you really need to understand the system deeply. There is always the chance that you introduce differences in your data generation that skew test results.

One way to perform test data generation is to do the hard work once. Then you can save a copy of the data that you can use time and time again. There may be some circumstances where you will need to tweak the test data. However the heavy lifting will already be done. This is how we approach test data generation on my project. Some DBAs run processes each year to give us a new data set valid for that year. The results are passed to both the development and internal test teams. The work in generating this test data is not too high. The results are awesome.

Wasted Unit Tests

I read a blog post entitled “Can Unit Testing Be A Waste” on AgileSoftwareDevelopment. It was written by pbielicki. He recommended you write unit tests at a high level of abstraction. Writing tests for getters and setters is a total waste of time. The goal of agile development is to eliminate waste. You should add tests just in time to verify functionality. You should also use common sense while choosing what to test.

One of the most interesting parts of the blog post was the many comments it received. For example, there was agreement that you need a coverage tool to even know where you stand with unit testing. There was emphasis on the goal of unit testing being a quality product. Another goal is to satisfy the customer.
Here on my project we are supposed to do unit testing. For the most part there is some level of unit test. Previously we had implemented a system of peer review to ensure there was adequate documentation of the unit tests. The peer review also ensured there was enough coverage for the unit test cases. However we have recently stopped performing such peer reviews. This is unfortunate. And I think it is starting to show in the quality of the code we are producing.

I am certain we are not wasting time on unit testing. We try to determine the minimal set of tests that will ensure something we code actually works. Our team borders on the side of too little when developing unit tests. Like many developers, we do not particularly like unit testing. I do it because I like it even less when I ship out very buggy code. That only creates a whole new set of worse problems. My company works on a maintenance contract where we get penalized when there are a lot of bugs shipped out to production. So this is a dollar and cents type of issue.

My manager has said that we have a documented peer review plan. The real problem is that I have never seen it. Thus it is truly not a priority. Yeah if I banged on doors and made a big deal about it, somebody would finally find me a copy of the thing. However it would be worthless if nobody is following it. Ouch. Get ready for a rough ride guys.

The State of Unit Testing

Python and Ruby developers are big on unit testing. However there is not as much excitement with it in the Java world. Test Driven Development is getting to the point where it is becoming a religion. So if you are doing agile software development, you can expect to also be doing unit testing.

There has been a few recent books on JUnit. Aside from that, there is really not a lot new with JUnit. There are also other unit testing frameworks out there such as TestNG. None of these really help with enterprise systems though. They are very hard to test since they have a lot of dependencies. Unit testing is also very difficult with legacy applications.

Like TDD, agile in general is going mainstream. And unit testing is one of the most adopted practices in software. However unit testing seems to be starting to be on the decline. In general it is hard to afford the time required for unit testing. Most projects and developers do basic unit testing. But if an existing system works, it is difficult to justify extra cost to add unit tests.

The goal of unit testing is better code. There are free tools available for unit testing. So the tool cost is usually not an issue. Good unit tests, like good code, require a lot of time and skill. All developers on your team must unit test for you to get the maximum benefit. You also need to do a lot of work with unit tests when you system changes drastically.

On my own project, we are supposed to do unit testing on all new code and any fixes. Previously we were required to generate documentation on the unit test activities. This documentation was inspected at the time of peer review. Now we only have management direction to make sure we do unit testing and produce some tangible results. However there are no checks to ensure this is done. You can imagine the tendencies given a hectic software life cycle where we work.

For a while I was starting to slack a bit with my own unit tests. I would check a few things out, and decide to ship the software based on that. But I am always up to improving myself. One of the areas I determined needed help was my unit testing. I found myself writing a lot of scripts lately to fix problems. I have decided that for every script I write, I would also write at least one other script that performed a unit test. Many times I end up writing multiple unit test scripts to create data, run the tests, and clean up my development environment. Sometimes this uncovers critical bugs and saves the day.

Going the Distance

The customer encountered a problem while testing our application suite. We had just migrated to a new set of tools, and a new version of the database on the back end. It took a while for me to get around to researching the problem. Luckily I was able to duplicate the problem in a development environment. I tracked the source of the problem down to some bad test data. Then I checked the database where the customer performed their tests. They had the same problem there. The fix was to write a script to correct the data.

At first when I ran the script to correct the problem, it just bombed. It logged the problems it encountered. So I manually executed each of the statements in the script until I found what the problem was. That’s when I was done with the coding. It was time to test it out. Luckily I had a lot of data that was corrupt in my development environment. I ran the script and it fixed about 5000 records. I used a tool to view the corrected data. In general it looked pretty good. I was tempted to just ship the fix at this point since I had 9 other problem to research and resolve.

Instead I decided to take my unit tests a step further. I figured it would be easiest to write another script to verify all the records were updated correctly. It took about an hour to write this unit test script. That was not too much overhead considering the time it take to investigate and code a solution to the problem. I did find some exceptional scenarios in the script results when I churned the data with a script. I just checked all 5000 outputs records for correctness. At first I just output the number of problems the test script found. Then I actually wrote the problems out to a log for further analysis.

Here in development we always seem to be in a rush because we are understaffed and overworked. But here is the funny thing. If I sent out a script that did not fully work, I would waste a whole lot more time dealing with the repercussions. Why not spend a little more time and get it right the first time? I love to code. Writing a script to verify the unit test results is just a little more coding. So it is in fact fun work. Now just don’t tell the management staff that. Otherwise they will get the wrong idea.

Testing Database Code

Last night I read a blog entry entitled “How to Unit-Test Code That Interacts With a Database”. It stated that such testing is hard. But you still want to fully test the database interaction. The blog recommended that you connect to a separate database to do your unit testing. Each developer should ideally have their own database. In addition, the database should be reset to the state it started in prior to the unit test. The best database for testing is a small focused data set. Above all the unit test activity should not connect to the production database.

On our project we conduct unit testing on new code that we write, as well as on fixes that we produce. Our main test data set is a small but interesting combination of test cases for verification. In general most of our developers have their own schema for testing. This allows you to do whatever you want without impacting other developers. It also helps in that nobody else makes changes to the data in your database. In the past I have written elaborate unit test scripts which generate delete old test data, generate new test data, execute the tests, and report on the results. This is normally done when I write code that relies heavily on database interaction.

One of the best parts of the blog post I read was the large number of comments from readers. I will share some of them here for your benefit. Multiple people pointed out that the type of testing discussed here is integration testing rather than unit testing. Unit tests are concerned with logic or combination checks. Oracle users recommended the utPLSQL framework for Oracle. You can use the ut_setup and ut_teardown functions to take care of unit test data.

Some readers commented that you could treat the database changes as one big transaction, then roll it back after testing is done. This would definitely not work for my system. We commit transactions all over the place. We even have autonomous transactions which commit themselves without interacting with any outer transactions. You can do better than getting your own schema for testing. You can get a whole database installation of your own to conduct unit tests in. Express versions of Oracle and Microsoft SQL Server are free.

It appears that unit testing is hot in the blogosphere. This seems strange to me. But I am happy about it. Nobody seems to like unit testing where I work. Most developers seem to want to code, check it in, and ship it. I know I myself could use a more rigorous unit test habit.

Scripting

I read a stimulating blog post by Michiel de Mare entitled “Is Your Program Perfect”. The post was essentially a bunch of questions that made you consider whether there were any improvements you could make to a program to make it better. There was a section on testing that asked whether your program could be scripted from the command line. It also inquired whether your program exposes an API. I thought long and hard about these questions as I considered my projects own suite of applications. And I determined they were very weak in this area. The question now is how do we make changes to the programs to improve in these areas.

In the past I had played with WinRunner by Mercury Interactive. This tool pretty much allowed you to automate testing of any Windows application. The tests could be scripted and create easily by recording and playing back user actions. However this required you to purchase the expensive WinRunner tool. Furthermore this tool is going to be retired in a few years. It would be more economical and beneficial for our applications to provide better test and scripting support itself.

Currently one of our applications has an API that allows you to instantiate at least one of its screens programmatically. However it is a very cumbersome API. You have to create a data on the file system or in shared memory. It also requires clunky windows messages to initiate the actions. This support only exists in one of the five applications in our suite. And this support is for one very specialized part of the app. It would be great if a more comprehensive and simple API could be produce to control the application for test purposes. I envision this to be of great use during the unit test phase.

Obviously it will take a while to add the hooks to our applications. But we could start small and work our way up from there. Perhaps the best place to start would be to automate access to the new feature we are currently coding into the application. We should be very familiar with the design and implementation of the new code. And we already have time in the schedule to work on these pieces. During the unit test phase of these features we could add in the automated support and actually use it for unit testing.

All of this introspection was due to a small piece of Michiel’s blog post to question whether your software is at its best. Opening our apps up to automation may go a long way to simplifying our unit test strategy.