True Testing

A manager on my project wanted to know how some things worked in one of our applications. They needed to speak with a customer. So this information needed to be very accurate. I kind of knew how the application worked. However I did not know all the details for sure. We have a lot of documentation that describes the most intricate details of the applications. Some of this documentation is old. My instinct told me not to trust the documentation. I had no other alternative than to reverse engineer the application behavior through a series of tests.

I spent a lot of time conducting different scenarios to test. It was good that I had some idea on how the application worked. That way I could choose tests where I was not sure of the results. I could also infer a lot from just a few tests results. This was not an exhaustive search of all the possibilities. That would have taken far too long. It took about half a day to get the data I needed.

This technique may not work for everybody. You have to make sure you really get a firm understanding of what is going on behind the scenes. However I trust this more than consulting some documentation or even reviewing the source code. In my experience, it is best to trust nothing. I know that sounds a bit like the X Files. But it is the safest approach.

Now I can say with confidence how the application behaves for some certain behavior. If anybody wants to challenge me, I can ask them where they got their information. Is that just the way it is supposed to be? Did they hear it from somebody a long time ago? Did they read this in some document? I can top all of that. I just tested the actual application out. You can’t beat that kind of experience.

Good Test Data

You need reliable test data to conduct independent verification. This is more complicated than it seems for some systems. The Software Maintenance blog demonstrates that sometimes even developers have trouble creating test data.

Ideally you can run the system like it operates in a production environment. The closer to the real world you make it, the better the quality of your test data. However this is not always feasible. Sometimes you need to simulate the real world with generated test data. It is these circumstances where you really need to understand the system deeply. There is always the chance that you introduce differences in your data generation that skew test results.

One way to perform test data generation is to do the hard work once. Then you can save a copy of the data that you can use time and time again. There may be some circumstances where you will need to tweak the test data. However the heavy lifting will already be done. This is how we approach test data generation on my project. Some DBAs run processes each year to give us a new data set valid for that year. The results are passed to both the development and internal test teams. The work in generating this test data is not too high. The results are awesome.

Wasted Unit Tests

I read a blog post entitled “Can Unit Testing Be A Waste” on AgileSoftwareDevelopment. It was written by pbielicki. He recommended you write unit tests at a high level of abstraction. Writing tests for getters and setters is a total waste of time. The goal of agile development is to eliminate waste. You should add tests just in time to verify functionality. You should also use common sense while choosing what to test.

One of the most interesting parts of the blog post was the many comments it received. For example, there was agreement that you need a coverage tool to even know where you stand with unit testing. There was emphasis on the goal of unit testing being a quality product. Another goal is to satisfy the customer.
Here on my project we are supposed to do unit testing. For the most part there is some level of unit test. Previously we had implemented a system of peer review to ensure there was adequate documentation of the unit tests. The peer review also ensured there was enough coverage for the unit test cases. However we have recently stopped performing such peer reviews. This is unfortunate. And I think it is starting to show in the quality of the code we are producing.

I am certain we are not wasting time on unit testing. We try to determine the minimal set of tests that will ensure something we code actually works. Our team borders on the side of too little when developing unit tests. Like many developers, we do not particularly like unit testing. I do it because I like it even less when I ship out very buggy code. That only creates a whole new set of worse problems. My company works on a maintenance contract where we get penalized when there are a lot of bugs shipped out to production. So this is a dollar and cents type of issue.

My manager has said that we have a documented peer review plan. The real problem is that I have never seen it. Thus it is truly not a priority. Yeah if I banged on doors and made a big deal about it, somebody would finally find me a copy of the thing. However it would be worthless if nobody is following it. Ouch. Get ready for a rough ride guys.

The State of Unit Testing

Python and Ruby developers are big on unit testing. However there is not as much excitement with it in the Java world. Test Driven Development is getting to the point where it is becoming a religion. So if you are doing agile software development, you can expect to also be doing unit testing.

There has been a few recent books on JUnit. Aside from that, there is really not a lot new with JUnit. There are also other unit testing frameworks out there such as TestNG. None of these really help with enterprise systems though. They are very hard to test since they have a lot of dependencies. Unit testing is also very difficult with legacy applications.

Like TDD, agile in general is going mainstream. And unit testing is one of the most adopted practices in software. However unit testing seems to be starting to be on the decline. In general it is hard to afford the time required for unit testing. Most projects and developers do basic unit testing. But if an existing system works, it is difficult to justify extra cost to add unit tests.

The goal of unit testing is better code. There are free tools available for unit testing. So the tool cost is usually not an issue. Good unit tests, like good code, require a lot of time and skill. All developers on your team must unit test for you to get the maximum benefit. You also need to do a lot of work with unit tests when you system changes drastically.

On my own project, we are supposed to do unit testing on all new code and any fixes. Previously we were required to generate documentation on the unit test activities. This documentation was inspected at the time of peer review. Now we only have management direction to make sure we do unit testing and produce some tangible results. However there are no checks to ensure this is done. You can imagine the tendencies given a hectic software life cycle where we work.

For a while I was starting to slack a bit with my own unit tests. I would check a few things out, and decide to ship the software based on that. But I am always up to improving myself. One of the areas I determined needed help was my unit testing. I found myself writing a lot of scripts lately to fix problems. I have decided that for every script I write, I would also write at least one other script that performed a unit test. Many times I end up writing multiple unit test scripts to create data, run the tests, and clean up my development environment. Sometimes this uncovers critical bugs and saves the day.