True Testing

A manager on my project wanted to know how some things worked in one of our applications. They needed to speak with a customer. So this information needed to be very accurate. I kind of knew how the application worked. However I did not know all the details for sure. We have a lot of documentation that describes the most intricate details of the applications. Some of this documentation is old. My instinct told me not to trust the documentation. I had no other alternative than to reverse engineer the application behavior through a series of tests.

I spent a lot of time conducting different scenarios to test. It was good that I had some idea on how the application worked. That way I could choose tests where I was not sure of the results. I could also infer a lot from just a few tests results. This was not an exhaustive search of all the possibilities. That would have taken far too long. It took about half a day to get the data I needed.

This technique may not work for everybody. You have to make sure you really get a firm understanding of what is going on behind the scenes. However I trust this more than consulting some documentation or even reviewing the source code. In my experience, it is best to trust nothing. I know that sounds a bit like the X Files. But it is the safest approach.

Now I can say with confidence how the application behaves for some certain behavior. If anybody wants to challenge me, I can ask them where they got their information. Is that just the way it is supposed to be? Did they hear it from somebody a long time ago? Did they read this in some document? I can top all of that. I just tested the actual application out. You can’t beat that kind of experience.

Good Test Data

You need reliable test data to conduct independent verification. This is more complicated than it seems for some systems. The Software Maintenance blog demonstrates that sometimes even developers have trouble creating test data.

Ideally you can run the system like it operates in a production environment. The closer to the real world you make it, the better the quality of your test data. However this is not always feasible. Sometimes you need to simulate the real world with generated test data. It is these circumstances where you really need to understand the system deeply. There is always the chance that you introduce differences in your data generation that skew test results.

One way to perform test data generation is to do the hard work once. Then you can save a copy of the data that you can use time and time again. There may be some circumstances where you will need to tweak the test data. However the heavy lifting will already be done. This is how we approach test data generation on my project. Some DBAs run processes each year to give us a new data set valid for that year. The results are passed to both the development and internal test teams. The work in generating this test data is not too high. The results are awesome.

Wasted Unit Tests

I read a blog post entitled “Can Unit Testing Be A Waste” on AgileSoftwareDevelopment. It was written by pbielicki. He recommended you write unit tests at a high level of abstraction. Writing tests for getters and setters is a total waste of time. The goal of agile development is to eliminate waste. You should add tests just in time to verify functionality. You should also use common sense while choosing what to test.

One of the most interesting parts of the blog post was the many comments it received. For example, there was agreement that you need a coverage tool to even know where you stand with unit testing. There was emphasis on the goal of unit testing being a quality product. Another goal is to satisfy the customer.
Here on my project we are supposed to do unit testing. For the most part there is some level of unit test. Previously we had implemented a system of peer review to ensure there was adequate documentation of the unit tests. The peer review also ensured there was enough coverage for the unit test cases. However we have recently stopped performing such peer reviews. This is unfortunate. And I think it is starting to show in the quality of the code we are producing.

I am certain we are not wasting time on unit testing. We try to determine the minimal set of tests that will ensure something we code actually works. Our team borders on the side of too little when developing unit tests. Like many developers, we do not particularly like unit testing. I do it because I like it even less when I ship out very buggy code. That only creates a whole new set of worse problems. My company works on a maintenance contract where we get penalized when there are a lot of bugs shipped out to production. So this is a dollar and cents type of issue.

My manager has said that we have a documented peer review plan. The real problem is that I have never seen it. Thus it is truly not a priority. Yeah if I banged on doors and made a big deal about it, somebody would finally find me a copy of the thing. However it would be worthless if nobody is following it. Ouch. Get ready for a rough ride guys.

The State of Unit Testing

Python and Ruby developers are big on unit testing. However there is not as much excitement with it in the Java world. Test Driven Development is getting to the point where it is becoming a religion. So if you are doing agile software development, you can expect to also be doing unit testing.

There has been a few recent books on JUnit. Aside from that, there is really not a lot new with JUnit. There are also other unit testing frameworks out there such as TestNG. None of these really help with enterprise systems though. They are very hard to test since they have a lot of dependencies. Unit testing is also very difficult with legacy applications.

Like TDD, agile in general is going mainstream. And unit testing is one of the most adopted practices in software. However unit testing seems to be starting to be on the decline. In general it is hard to afford the time required for unit testing. Most projects and developers do basic unit testing. But if an existing system works, it is difficult to justify extra cost to add unit tests.

The goal of unit testing is better code. There are free tools available for unit testing. So the tool cost is usually not an issue. Good unit tests, like good code, require a lot of time and skill. All developers on your team must unit test for you to get the maximum benefit. You also need to do a lot of work with unit tests when you system changes drastically.

On my own project, we are supposed to do unit testing on all new code and any fixes. Previously we were required to generate documentation on the unit test activities. This documentation was inspected at the time of peer review. Now we only have management direction to make sure we do unit testing and produce some tangible results. However there are no checks to ensure this is done. You can imagine the tendencies given a hectic software life cycle where we work.

For a while I was starting to slack a bit with my own unit tests. I would check a few things out, and decide to ship the software based on that. But I am always up to improving myself. One of the areas I determined needed help was my unit testing. I found myself writing a lot of scripts lately to fix problems. I have decided that for every script I write, I would also write at least one other script that performed a unit test. Many times I end up writing multiple unit test scripts to create data, run the tests, and clean up my development environment. Sometimes this uncovers critical bugs and saves the day.

Going the Distance

The customer encountered a problem while testing our application suite. We had just migrated to a new set of tools, and a new version of the database on the back end. It took a while for me to get around to researching the problem. Luckily I was able to duplicate the problem in a development environment. I tracked the source of the problem down to some bad test data. Then I checked the database where the customer performed their tests. They had the same problem there. The fix was to write a script to correct the data.

At first when I ran the script to correct the problem, it just bombed. It logged the problems it encountered. So I manually executed each of the statements in the script until I found what the problem was. That’s when I was done with the coding. It was time to test it out. Luckily I had a lot of data that was corrupt in my development environment. I ran the script and it fixed about 5000 records. I used a tool to view the corrected data. In general it looked pretty good. I was tempted to just ship the fix at this point since I had 9 other problem to research and resolve.

Instead I decided to take my unit tests a step further. I figured it would be easiest to write another script to verify all the records were updated correctly. It took about an hour to write this unit test script. That was not too much overhead considering the time it take to investigate and code a solution to the problem. I did find some exceptional scenarios in the script results when I churned the data with a script. I just checked all 5000 outputs records for correctness. At first I just output the number of problems the test script found. Then I actually wrote the problems out to a log for further analysis.

Here in development we always seem to be in a rush because we are understaffed and overworked. But here is the funny thing. If I sent out a script that did not fully work, I would waste a whole lot more time dealing with the repercussions. Why not spend a little more time and get it right the first time? I love to code. Writing a script to verify the unit test results is just a little more coding. So it is in fact fun work. Now just don’t tell the management staff that. Otherwise they will get the wrong idea.

Testing Database Code

Last night I read a blog entry entitled “How to Unit-Test Code That Interacts With a Database”. It stated that such testing is hard. But you still want to fully test the database interaction. The blog recommended that you connect to a separate database to do your unit testing. Each developer should ideally have their own database. In addition, the database should be reset to the state it started in prior to the unit test. The best database for testing is a small focused data set. Above all the unit test activity should not connect to the production database.

On our project we conduct unit testing on new code that we write, as well as on fixes that we produce. Our main test data set is a small but interesting combination of test cases for verification. In general most of our developers have their own schema for testing. This allows you to do whatever you want without impacting other developers. It also helps in that nobody else makes changes to the data in your database. In the past I have written elaborate unit test scripts which generate delete old test data, generate new test data, execute the tests, and report on the results. This is normally done when I write code that relies heavily on database interaction.

One of the best parts of the blog post I read was the large number of comments from readers. I will share some of them here for your benefit. Multiple people pointed out that the type of testing discussed here is integration testing rather than unit testing. Unit tests are concerned with logic or combination checks. Oracle users recommended the utPLSQL framework for Oracle. You can use the ut_setup and ut_teardown functions to take care of unit test data.

Some readers commented that you could treat the database changes as one big transaction, then roll it back after testing is done. This would definitely not work for my system. We commit transactions all over the place. We even have autonomous transactions which commit themselves without interacting with any outer transactions. You can do better than getting your own schema for testing. You can get a whole database installation of your own to conduct unit tests in. Express versions of Oracle and Microsoft SQL Server are free.

It appears that unit testing is hot in the blogosphere. This seems strange to me. But I am happy about it. Nobody seems to like unit testing where I work. Most developers seem to want to code, check it in, and ship it. I know I myself could use a more rigorous unit test habit.

Scripting

I read a stimulating blog post by Michiel de Mare entitled “Is Your Program Perfect”. The post was essentially a bunch of questions that made you consider whether there were any improvements you could make to a program to make it better. There was a section on testing that asked whether your program could be scripted from the command line. It also inquired whether your program exposes an API. I thought long and hard about these questions as I considered my projects own suite of applications. And I determined they were very weak in this area. The question now is how do we make changes to the programs to improve in these areas.

In the past I had played with WinRunner by Mercury Interactive. This tool pretty much allowed you to automate testing of any Windows application. The tests could be scripted and create easily by recording and playing back user actions. However this required you to purchase the expensive WinRunner tool. Furthermore this tool is going to be retired in a few years. It would be more economical and beneficial for our applications to provide better test and scripting support itself.

Currently one of our applications has an API that allows you to instantiate at least one of its screens programmatically. However it is a very cumbersome API. You have to create a data on the file system or in shared memory. It also requires clunky windows messages to initiate the actions. This support only exists in one of the five applications in our suite. And this support is for one very specialized part of the app. It would be great if a more comprehensive and simple API could be produce to control the application for test purposes. I envision this to be of great use during the unit test phase.

Obviously it will take a while to add the hooks to our applications. But we could start small and work our way up from there. Perhaps the best place to start would be to automate access to the new feature we are currently coding into the application. We should be very familiar with the design and implementation of the new code. And we already have time in the schedule to work on these pieces. During the unit test phase of these features we could add in the automated support and actually use it for unit testing.

All of this introspection was due to a small piece of Michiel’s blog post to question whether your software is at its best. Opening our apps up to automation may go a long way to simplifying our unit test strategy.

White Space

Our client has a system acceptance test team. Recently they approached me regarding a problem our load software had last year. They knew it had something to do with the input data containing spaces. And they were trying to determine the impact on the system, and whether they needed to do any additional testing of it this year. I gave this problem some thought and told the test team what I knew.

There is a certain field in one of our input files that is 4 characters in length. We are required to read this field, trim off any white spaces, and store the result in the database. This had worked properly for a number of years. A developer rewrote the load routine using a new tool. They did not correctly follow the requirements and trim the white space from the field. As a result, post load processing could not match the loaded value up with other values that had trimmed. Additionally the application that queries the data could not form the match either.

It took quite a bit to resolve this problem. We had to correct the loaded data for millions of records. Then we had to run the data back through post loads processing. Finally the original load software was corrected to deal properly with future files loaded. It was this last part that the test team wanted to verify. It was good that they are paying attention. Since we only load this particular file once a year, nobody other than the developer tested the fix to the loads code.

The test team was happy to hear the information I provided them. They were immediately running off identifying test cases and planning to generate some input file data to check out our code. We have an internal test team at our company. For some reason I doubt they also have this type of insight and premonition to verify that development follows through on the fixes we produce. Its always easier to deal with problems during test before the software goes out into a production environment.

Access Levels

Our application suite supports a number of access levels. Each user is assigned one of five access levels. Each level has a uniquely defined behavior in the application. In general, the higher the level, the more you can do. But higher levels do not always have the ability to do everything a lower level user can do.

There are a number of times when people believe the application is not behaving correctly. Frequently this is due to an ignorance of what each access level is allowed to do. Normally this happens with the newer developers or testers. It usually takes a senior developer to sort out the problem and trace it back to an access level issue.

I have been working with a new tester recently. He is testing out the application that I am responsible for. He has found some weird problems in my application. There is a lot of interaction between us since I often need to know exactly how to reproduce a problem, or what specific data the tester is using to validate the application.

The other day I asked the tester to reproduce the problem he had documented. He had to do a lot of work to reset the application access levels to the ones used when he encountered the problem. He told me that he was doing a lot of testing in the application at each of the access levels. This was recommended to him by a senior tester. This encouraged me. I have to admit that, while conducting my own unit tests, I did not test all the functionality at each of the access levels.

This emphasizes the differences between developers and testers conducting verification. A developer may be quick to declare success once one test passes. Or sometimes a developer will assume that some tests will pass based on the results of one other test. A tester may also have this opinion. But they will actually perform the tests to verify the hypothesis. However a developer may be quick to skip steps that seem repetitive and boring.

It is a good thing that we have an internal test team to keep development honest. Their absence would not be a total loss. Our customer has their own system acceptance test team. And since we are migrating to all new development tools this year, the users themselves have scheduled a functionality test of our upgraded application. I have the feeling that I am going to be researching a lot of trouble tickets this year. Since I like fixing bugs, this is a good thing.

Time to Market

I have a software development business on the side of my day job. Recently I have focused my business on writing small programs that would be of interest to hackers. The programs I develop are posted on my Black of Hat blog. Currently I am giving my programs away for free to try to increase readership of my new blog.

The interesting thing about these programs is that I have not been subjecting them to rigorous testing. Often times the goal is to get some software posted out on my blog quickly. So I run a couple quick unit tests, fix the obvious bugs, and warn my readers that the software may be in an early state.

My goal is to write many useful programs quickly. Of course I plan to improve and fix any bugs that pop up. But there is a definite lack of V&V being done. The question is whether formal test methods would be of any value in this fast paced environment. I don't get any direct penalty for releasing a program that crashes and burns. There may be some unhappy readers which turn away from my blog. However I am considering the faster time to market to be the prime objective.

Perhaps I could take my next program developed by my side business, apply more thorough testing to it, and try to measure the outcome subjectively. I am not against going through a formal test cycle for these programs. The objective is to gain the most amount of readers so that I can monetize my blog. Any way I can do that more effectively is worthwhile. Besides I feel as if I may be developing bad habits in my side business software development. It would be easier if I use the same methods in both my day and night jobs.

Test to Dev to Test

Over my career, I have met a number of software developers that have become testers. These testers often tell me that they got tired of working the long hours in development. They say that testing, while challenging, is less stressful and easier that writing code. And these are usually the testers that are creating scripts or writing code to test data.

I have also met developers who said they got trapped in the test environment. First it starts out with the need to write a high profile master test plan. Then who better to oversee the test execution than the author. Pretty soon you get known as somebody proficient who can run tests, and then you are pigeon holed into being a tester.

There is one other scenario which is not encouraging. Sometimes developers who just do not cut it as programmers get reassigned to the test team. It is almost as if this is a demotion. This seems strange as the top talent in the test world seem to make good money. Maybe it is just a psychological effect.

The reverse path appears to be difficult if not impossible. I have not met many developers who said they started out as a tester. There was one guy I knew who was pretty good with code. He wrote a lot of scripts to generate test data. But he was still officially assigned to the test team. Maybe there are such developers, but they don't want to talk about their test background. Still it seems a very rare move. It looks like a one way street between development and testing roles.

Top Notch Testers

Recently we had some late new requirements that needed to get into the system. We rolled them out after our latest major release of the software. There was a hasty test cycle before shipping the software out the door. And guess what? There are a lot of problems with the code in the Production environment.

The good news is now that we are fixing the problem, our internal test team is better equipped and has more time to verify our changes. A recent set of changes has failed internal test a couple times. Seems I have made a number of invalid assumptions about the data in the system.

A lot of the good testing comes from the fact that the tester assigned to my code is actually an old developer. He used to sit next to me in my cubicle on a past project. Somehow he got moved to the test organization. And now he is taking test to a whole new level.

This reminds me of another tester from a couple years ago. Let's call her "SJ". She astounded me and my team when she first started verifying out fixes. SJ has a deep technical understanding about the changes we were making. Sometimes she even dug down and showed me where I had made mistakes in the code. When I asked SJ how she was able to figure this out, she said that she too was a developer in a past life. She wanted something different and was doing testing now.

Not all testers have to be developers at heart. But I have found that it certainly helps.

Dev Test

I am a software developer by trade. Most projects I have worked on were mainly software maintenance. However I have worked on quite a few new development projects. It seems that I often write more code to do unit tests that I do to implement the software. This is normally for code that is mission critical, where errors can be very costly for everyone involved.

So I have started this blog to research the verification and validation field. Specifically I want to focus on the testing of computer software. Sometimes normal developers consider test professionals to be a step or two below a coder. However I have worked on projects where the quality assurance team were the domain experts and knew what the users wanted the software to do. I am sure those that test at big commercial companies like Microsoft are no slouches.

On a related note, I recently went to an interview for a software position. Got interviewed by a number of people on the project. The most intense interview was with the lead of the test team. This guy was a former developer and the one that had been on the project the longest. I aced a couple of tough testing questions from him. But I got some others that I had trouble working my way through. Perhaps those questions will serve as a starting point for the discussions here:

1. What is a load test?
2. What is a stress test?