Time for TDD

Test Driven Development (TDD) is gaining more acceptance in programming circles. But at least one question still remains. When should you apply TDD? In other words, when do you develop the tests? A simple answer is to create the tests first. That way you have tested code from day one. However this may not always be the right way to go.

I read a blog entry stressing that TDD is beneficial for maintenance work. You should have mature code complete with self tests. However the very first class you write for a new system may not need tests right away. Testing may in fact get in the way of rapid progress. I myself am divided on this very topic.

In a perfect world, you would have understood all the requirements and performed a full analysis and design on the project. At that point you are ready to add TDD before you write the first line of code. The real world is never perfect. So this may not be the way to go. Sometimes you need to write some code to figure things out. I would not call this production code. This may be throw away code. And that code does not need TDD applied to it.

We do not use TDD in general on my project. I have tried it. For small tasks it seems to work well. But when everything is crazy and we are terribly behind schedule, the TDD goes out the window and we hack together solutions. Sure it is not the best way to go. But it is life. There is probably a time and place for TDD. That time may not be before you write any code.

Cost of TDD

I read a blog about a developer that tried Test Driven Development (TDD) for a day. He found that a useful way to generate the tests is to debug your code. When you find errors, you can write a test that checks for that bug. This would be perfect for my project. It is a maintenance project. We get trouble tickets all the tine that we resolve. It would not take much more effort to integrate the tests for the bugs we fix into a automated test script. We already do the unit testing for each fix anyway.

Getting back to the subject of TDD, the developer who tried it and wrote about it did not like it. He said the TDD technique led to very little actual debugging. He did gain some new respect for the method in general. However he would still not use it for everyday programming purposes. While the effort did seem to result in higher quality code, it took much longer to develop code that way.

When can you use TDD? I know we can’t use it on our project. We hardly have the time to race through coding and hack together some solutions on my project. If TDD causes the development cycle to double in time, we would never get anything done at my job. Here is the funny thing about this.

Perhaps we really are the best candidates for a TDD approach. The problem is that it would be a hard sell. One of the main things that drives our schedule is customer needs. And our particular customer wants more functionality in less time. They don’t seem to care about our methodology. Yes they do not like bugs. However they will tolerate bugs if it means they get a lot of features right away.

Production Volume

I read this blog about how you should test using volume comparable to production. Big hard disks cost little these days right? Wrong. Well yes some el cheapo hard disk can be found for a bargain. However that disk will never match the characteristics of our production server box.

Now I don't know all the details. But we have a massive box to run our production system. There is probably something like 128 processors on the darn thing. And there is more RAM in there than you could find on a mainframe.

Don't get me started about the super speed disk array we got powering our database. But I guess you get the point. You cannot compare your little bitty workstation with a production box. To do some is futile. I guess the only pearl of wisdom from the blog post I read was that you should not do tests if you only have a couple records in your tests tables. Point taken.

Oracle Application Suite

You would think a tool called Oracle Application Suite would be focused on some type of database work. Specifically it sounds like it does something with the Oracle Database. However I just read an article that tells how this tool can be used for testing.

The Oracle Application Suite is actually a part of Oracle Enterprise Manager. Now I have heard of this product. However I have seen DBAs use this to manage Oracle databases. But I digress. Let's get back to how Oracle Application Suite can be used to test.

There are three things that Oracle Application Suite does to assist testing. It helps load testing by simulating a lot of users logging on to the database. It also performs functional testing by automating test scripts. Finally it helps with all kinds of chores related to web application testing.

Now like most Oracle tools, I hope this one is free. If so I plan on checking it out. On a final note, I understand this tool integrates with other products. I guess I will have to play with it to find out how that works.

Fuzzing

I read an article in Software Test and Performance magazine about the topic “Fuzzing”. This was the first time I had heard this term. However recently I heard some of my team’s tester talking about the issues involved with fuzzing. So I thought I would write about this as well.

Fuzzing is a negative software testing method. That means you do not test for whether the system works correctly with correct inputs. Instead fuzzing concentrates on things such as inputting malformed data to an application. It also involves attempts to defeat system security. This used to be a hacking technique. Now it has been formalized as a testing methodology.

Requirements themselves can be written in a positive and negative form. That is, requirements usually dictate what the software must do. However some requirements also define the desired system behavior by describing also what the system shall not do.

A way to perform fuzzing is to start with a test case. Then you can modify the test data and rerun the test. The more general realm of testing is robustness testing. This is where you test how the system behaves when there are invalid inputs. One such form of robustness testing is fuzzing. The main purpose of fuzzing is to identify security problems in the system.

When I was with some of our team’s testers, I heard them talking about their negative testing. I did not drop the “fuzzing” buzzword with them. However it was good to see that they are addressing modern topics in the test world.

Changing Test Data

Our customer reported a bug in production. There were many theories as to why this problem was happening. I told people that these were all interesting theories. But these theories, like any opinions, were really not worth much. When I got some time, I dug into the problem. I traced the problem back to the line of code that caused the problem. In the end I determined some configuration data was missing. That resulted in the good code causing a bad transaction. I unit tested my fix by hacking together some data to meet the specific scenario where the error occurs.

Then the fix went to our internal test team. A tester called me up and asked me about the problem. I explained the problem from top to bottom. You would think that would be sufficient for a tester to run with the fix and test it out. However I got back a question that I seem to get a lot from our internal test team. How do I test that? In essence they wanted to know how you set up the data conditions that result in replication of the error.

Now I am sympathetic somewhat to the testers. They do not have the source code in front of them. The documentation for our system is merely average. So it might be unreasonable to assume they had all the tools they needed to figure things out. Therefore I outlined a set of steps that could be taken to replicate this problem.

Development had previously produced a tool which generated input files for our loads component. You could use the tool to generate input files which contain the data the make the problem happen. I recommended that the testers generate up the input files with the tool, run them through the loads program, and duplicate the problem.

I guess this was too much work. The testers tried to simulate all that work by taking data previously mocked and loaded, and performing updates on that data. This is not a bad decision. But you have to know what you are doing. Unfortunately they were unable to replicate the problem. My boss told me to get in there and help them out to get this fix shipped. Essentially I had to reverse engineer our loading code. Then I kept giving them all the data they needed to change manually in order to duplicate the problem.

This is not a good scene. I have things to do. If you want to add value as a tester, you got to do the hard work. Then you will come out smarter and be able to wing it one day. But not now guys.

Testing is Hard Work

Yesterday the test manager on my project came to visit me. She was planning the tests for some new changes I am working on. She asked me to explain some of the changes as they did not make much sense to her.

I tried to get into the details with her. She took some notes. Then she said she would get some people from her team to dig into it. Today I got a message from her. She said her team could not make heads or tails of the requirements, and asked me to help them.

Well I shot off an informative email on the current state of the application, and what the customer wanted instead. Then I got a response on how they could test that. I gave the test team a little more information. Then I got another response asking how they could set up data to do the tests.

I did not want to just provide them with my own unit test plan. Then they would not be adding any value to the work. In an ideal world, they would be able to figure out how to test according to the documented requirements. Unfortunately the requirements team does not know much more than the testers. So I continued to try to give more ideas until they felt like they had enough to go on. How do we get to that ideal world where there is independent validation of my work?