Development Diligence

The customer informed us that there were some special situations where they wanted their data formatted in non-standard ways. I was tasked with making that happen. Had to change some column data types to enable this. My grand plan was to have one function do the formatting, and all the places that needed to do the custom formatting would just call this one helper function.

I had a lot of work to implement these requirements. Making the changes to load future data was not that hard. But I also had to go in and correct data previously loaded. That was a massive data correction script. I rushed through to get all the work done on time. I thought I did a pretty good job. Then I got to our internal test phase.

The tester assigned to verify my work had a lot of questions. I had to walk her through all the different scenarios, showing her where the source and target data was located. She did ask a lot of good questions. One of the last questions she asked was how to correlate the target to source data records. When I looked closer to answer the question, I found some problems.

What I had done was look how the old data was originally loaded into the system. I copied the technique, inserting my custom formatting function in the mix. The problem was that the original load code had a bug in it. It had since been fixed in recent code. But I was working from some bad code. The bug crept into my data correction script. Luckily the tester caught it by asking a lot of questions to understand exactly what was going on.

Now I have some work to do to implement the fix.

Automated Testing Depth

We are working on replacing an old and expensive ad hoc reporting system. The trouble is that there is a new database that holds all the information. The reports team tried to give the customer access to the current database. The data model was so different than the old one, the customer rejected that proposal.


So my team is coming in to create a bunch of tables and views to replicate the old schema. It is quite a chore as some of the data is tricky to produce. There is a dedicated test team making sure everything turns out okay. One of the testers has been tasked with automating the testing of our solution.


The tester is a Java programmer. She has written a program that will query our new database, match up the records with the old database, and alert us if there is any difference. That sounds like the best way to guarantee we get the values correct.


The big team lead stepped in to start peer reviewing our work on this system. He asked how we could be sure we were getting the values correct. I told him our Java programmer tester is checking it out. When pressed for details, the tester said she only compares 1 record per table. And if there are any NULL values involved, all bets are off. Oops. Need some more due diligence here.

Automated out of a Job

I read an incredible story today. Not sure if it true or not. Still very interesting. A guy got hired in the quality assurance department. He was a programmer at heart. So he decided to automate his job. He was able to spend about a week to get a set of programs to do all the testing he was assigned.


Here is the strange part. After totally automating his job, he sat back and start to chill. He came to work. But he played video games and worked out at the gym. He kept to himself and was not noticed for six years. His programs kept doing the verification that was required for a whopping six years.


Eventually his boss found out and fired him. Personally I applaud his automation skills. Maybe if he was better at being on the down low, he might have had a job for another six years or more. The guy was frugal and saved up a lot of cash. Now he is thinking about learning some newer technologies and going back to programming.

Test Data Gen Techniques

A developer came onto the project and made some sweeping changing to the software. Then he left when funding problems arose. I was left to deal with the mess. My goal was to test the changes to see if anything got broke. I found myself lacking data to run all the regression tests.

I dreaded creating data from scratch. There are so many fields to each of the records. They have varying formats and rules. Plus the data is interrelated. Ouch. There was no time for this. Then I read a somewhat related article on generating data if you already have some.

The idea is to take some existing data, distort it, creating new data sets. Specifically I saw this applied to image data. You stretch an image in a non-uniform fashion. The result is a new image that can be used for test. The key is that you cannot use random stretching across the image. The streching of points must be related to the stretching of points in the local area.

There is a whole science behind image distortion. I did not dig deep. The idea let me implement something similar in my own development database. I first identified a really good record. Then I wrote some routines to replicate that data, distorting the fields that could not be the same. The result was a quick way to generate tons of data.

I am now a happy tester. The icing on the cake was that the changes made by the long gone developer all seem to work just like the original code. I am going to use the distortion technique to cook up a batch of test data for my own new changes to the system.

Test Report Formatting

We have a standard format for our unit test reports. There is a section at the top that identifies all kinds of information about what we are testing. Then there is a section in the middle for our test steps. Finally there is a section at the bottom where we list our test results. Aside from these major areas on the document, we are pretty free in how we write our documentation.

The most important part for me is the steps. That is the meat of the test documentation. I often ponder what the best way to document your test steps is. Our testing team usually has generic steps that try to show the intent of what they are testing. I like to document very specific steps that anyone could follow if they read the document.

The problem with my technique is that it is hard to see the big picture of what those steps are accomplishing. Sure I could read the steps, and remember back what was in my mind. However I think the document should be readable to anyone else as well. Perhaps a combination of detailed steps with occsaional commentary to explain the big picture are in order.

Marked as Passed

Our development team has delegated the first line of cutomer support to our internal testing team. The test team is to try to replicate problems reported by our customer. This cuts down on developers being distracted from their primary role. It is best if the internal testers can help the customers figure out what's wrong.

Recently the customers reported a very strange behavior in some of our new software. A tester got assigned the task to replicate the problem. This tester could not even get the applicaiton launched. She reached out to a developer for help. The developer produced a recent build for the tester to use. Then tester then was able to launch the app.

This tester proceeded to then try the steps documented by the customer trouble ticket. There was no trouble for the tester when she tried following the steps. She made the conclusion that they developer must have fixed the customer problem. The tester reported that the problem was fixed and verified. Management took this diagnosis as gospel. Promises were made.

Disaster ensued from this mix up. The customer thought they were getting a fix. They received no such correction. Imagine development's surprise when this thing came back to us. Development was sure they never resolved the problem that the customer raised. This was tracked down and the conclusion was that the tester was never able to recreate the problem voiced by the customer. This is much different than testing and passing a fix.

Breaking Tests

I've got a relatively mature set of code that generates unit test data for a complex program we have. This code has been developed over the past 5 to 6 years. It creates about 15 postive test cases for different scenarios in the program. It also can also generate the same amount of negative test cases.

This year we added three more test cases. So the test code was modified to produce three more positive test cases worth of data. Things were looking good. I had a long standing action item to create data for a whole separate path that the program sometimes takes. This resulted in a new slew of tests.

At first I had some trouble generating the test data for the new scenarios. But I dug down and determined what I needed to change to get everything set up. I pretty much based the data creation on the existing routines to produce data for the existing scenarios. I was happy when the task was complete.

I decided to do a round of regression tests to make sure the original test scenarios still worked. Ooops. Looks like I broke half of them. Good thing I have an easy way to run the whole suite of tests for the original scenario. But now I got to big back in to ensure the new and old unit test scenarios are correctly served.