Development Diligence

The customer informed us that there were some special situations where they wanted their data formatted in non-standard ways. I was tasked with making that happen. Had to change some column data types to enable this. My grand plan was to have one function do the formatting, and all the places that needed to do the custom formatting would just call this one helper function.

I had a lot of work to implement these requirements. Making the changes to load future data was not that hard. But I also had to go in and correct data previously loaded. That was a massive data correction script. I rushed through to get all the work done on time. I thought I did a pretty good job. Then I got to our internal test phase.

The tester assigned to verify my work had a lot of questions. I had to walk her through all the different scenarios, showing her where the source and target data was located. She did ask a lot of good questions. One of the last questions she asked was how to correlate the target to source data records. When I looked closer to answer the question, I found some problems.

What I had done was look how the old data was originally loaded into the system. I copied the technique, inserting my custom formatting function in the mix. The problem was that the original load code had a bug in it. It had since been fixed in recent code. But I was working from some bad code. The bug crept into my data correction script. Luckily the tester caught it by asking a lot of questions to understand exactly what was going on.

Now I have some work to do to implement the fix.

Automated Testing Depth

We are working on replacing an old and expensive ad hoc reporting system. The trouble is that there is a new database that holds all the information. The reports team tried to give the customer access to the current database. The data model was so different than the old one, the customer rejected that proposal.


So my team is coming in to create a bunch of tables and views to replicate the old schema. It is quite a chore as some of the data is tricky to produce. There is a dedicated test team making sure everything turns out okay. One of the testers has been tasked with automating the testing of our solution.


The tester is a Java programmer. She has written a program that will query our new database, match up the records with the old database, and alert us if there is any difference. That sounds like the best way to guarantee we get the values correct.


The big team lead stepped in to start peer reviewing our work on this system. He asked how we could be sure we were getting the values correct. I told him our Java programmer tester is checking it out. When pressed for details, the tester said she only compares 1 record per table. And if there are any NULL values involved, all bets are off. Oops. Need some more due diligence here.

Automated out of a Job

I read an incredible story today. Not sure if it true or not. Still very interesting. A guy got hired in the quality assurance department. He was a programmer at heart. So he decided to automate his job. He was able to spend about a week to get a set of programs to do all the testing he was assigned.


Here is the strange part. After totally automating his job, he sat back and start to chill. He came to work. But he played video games and worked out at the gym. He kept to himself and was not noticed for six years. His programs kept doing the verification that was required for a whopping six years.


Eventually his boss found out and fired him. Personally I applaud his automation skills. Maybe if he was better at being on the down low, he might have had a job for another six years or more. The guy was frugal and saved up a lot of cash. Now he is thinking about learning some newer technologies and going back to programming.

Test Data Gen Techniques

A developer came onto the project and made some sweeping changing to the software. Then he left when funding problems arose. I was left to deal with the mess. My goal was to test the changes to see if anything got broke. I found myself lacking data to run all the regression tests.

I dreaded creating data from scratch. There are so many fields to each of the records. They have varying formats and rules. Plus the data is interrelated. Ouch. There was no time for this. Then I read a somewhat related article on generating data if you already have some.

The idea is to take some existing data, distort it, creating new data sets. Specifically I saw this applied to image data. You stretch an image in a non-uniform fashion. The result is a new image that can be used for test. The key is that you cannot use random stretching across the image. The streching of points must be related to the stretching of points in the local area.

There is a whole science behind image distortion. I did not dig deep. The idea let me implement something similar in my own development database. I first identified a really good record. Then I wrote some routines to replicate that data, distorting the fields that could not be the same. The result was a quick way to generate tons of data.

I am now a happy tester. The icing on the cake was that the changes made by the long gone developer all seem to work just like the original code. I am going to use the distortion technique to cook up a batch of test data for my own new changes to the system.

Test Report Formatting

We have a standard format for our unit test reports. There is a section at the top that identifies all kinds of information about what we are testing. Then there is a section in the middle for our test steps. Finally there is a section at the bottom where we list our test results. Aside from these major areas on the document, we are pretty free in how we write our documentation.

The most important part for me is the steps. That is the meat of the test documentation. I often ponder what the best way to document your test steps is. Our testing team usually has generic steps that try to show the intent of what they are testing. I like to document very specific steps that anyone could follow if they read the document.

The problem with my technique is that it is hard to see the big picture of what those steps are accomplishing. Sure I could read the steps, and remember back what was in my mind. However I think the document should be readable to anyone else as well. Perhaps a combination of detailed steps with occsaional commentary to explain the big picture are in order.

Marked as Passed

Our development team has delegated the first line of cutomer support to our internal testing team. The test team is to try to replicate problems reported by our customer. This cuts down on developers being distracted from their primary role. It is best if the internal testers can help the customers figure out what's wrong.

Recently the customers reported a very strange behavior in some of our new software. A tester got assigned the task to replicate the problem. This tester could not even get the applicaiton launched. She reached out to a developer for help. The developer produced a recent build for the tester to use. Then tester then was able to launch the app.

This tester proceeded to then try the steps documented by the customer trouble ticket. There was no trouble for the tester when she tried following the steps. She made the conclusion that they developer must have fixed the customer problem. The tester reported that the problem was fixed and verified. Management took this diagnosis as gospel. Promises were made.

Disaster ensued from this mix up. The customer thought they were getting a fix. They received no such correction. Imagine development's surprise when this thing came back to us. Development was sure they never resolved the problem that the customer raised. This was tracked down and the conclusion was that the tester was never able to recreate the problem voiced by the customer. This is much different than testing and passing a fix.

Breaking Tests

I've got a relatively mature set of code that generates unit test data for a complex program we have. This code has been developed over the past 5 to 6 years. It creates about 15 postive test cases for different scenarios in the program. It also can also generate the same amount of negative test cases.

This year we added three more test cases. So the test code was modified to produce three more positive test cases worth of data. Things were looking good. I had a long standing action item to create data for a whole separate path that the program sometimes takes. This resulted in a new slew of tests.

At first I had some trouble generating the test data for the new scenarios. But I dug down and determined what I needed to change to get everything set up. I pretty much based the data creation on the existing routines to produce data for the existing scenarios. I was happy when the task was complete.

I decided to do a round of regression tests to make sure the original test scenarios still worked. Ooops. Looks like I broke half of them. Good thing I have an easy way to run the whole suite of tests for the original scenario. But now I got to big back in to ensure the new and old unit test scenarios are correctly served.

Software Testing Class

I saw a posting for a free Software Testing course by Udacity. It starts up in a few weeks. I am busy this summer trying to relearn the Java programming language. However I always wanted to check out some online learning that does not suck. It seems that online is the future.

This course covers things such as code coverage (no pun intended). It also seems to spend a lot of time on random testing. I just finished up some unit testing. So this topic is very relevant to my job at work. The course is taught by John Regehr and Sean Bennett. To tell the truth, I don't know these guys. But they must be okay if they are approved to teach at Udacity.

I am not exactly sure how Udacity makes money. The courses seem free. Perhaps it is advertising funded? It looks like they don't require you to even buy a book. The whole school seems to be based on the Python programming language, which I have no experience in. I really want to get involved with some online learned that represents the education model to come.

Test Data Generation

We have a big need to generate test data to verify all kinds of functionality in our system. Personally I like to use Oracle PL/SQL to do the job. This is very practical. The data needs to be in our Oracle 11g database. What better way than to use some code compiled into the database?

Now PL/SQL does have some more advanced programming constructs such as collections. However you can't beat high level languages such as C++ or Java to have advanced features. I have been experimenting with writing some utility apps in Java.

The next step will be to start writing some Java apps to generate test data. I know how to use JDBC to do simple database queries and DML. Let's see where this takes me.

Automation Saves the Day

We have a new developer working on fixing some bugs. She got assigned a tough part of the system. The actual code is not too difficult. The testing is a real chore. The processes need a lot of requirements to be met before any of the data is processed. I have gone through this pain in the past while trying to test out large changes to this subsystem.

Luckily I performed a lot of tests and set up a data generation system. This system creates just the right data to test out all parts of the subsystem. I found this data generation to have come in handy when the test team could not figure out how to test any of this stuff.

The new developer was running a little behind schedule. She was finding, like all would, that it was next to impossible to test anything in this subsystem. I pointed her to the testing data generator and her problems were solved. Now she is teaching some testers how to use this tool to conduct their own test data sets. This tool is going a long way to making peoples' lives easier. Mine included.

Independent Verification

Our project has an independent testing team. Their goal is to verify that our software meets the requirements and has minimal bugs. The lead of this team recently left the company. Her replacement seems like she has management potential.

We just got through a big release of our software. It was to be delivered on a Friday. I was taking it easy that Friday since it had been a long week. Things were supposed to go out easily, with a formality of checking our production install.

One of the testers noticed some errors in the application. It turns out the changes for the release were not in order. The thing that was disturbing was that the software changes had passed test. Now it looked like the pass was achieved in error. Now the delivery was late and we had an emergency on our hands.

They needed some testing to get done. The guy who was supposed to have completed the testing had already left the company. Darn. The existing testers had little clue as to how they might test the system. What did we do? The developers were brought in to rerun the testing. Sounds like a massive fail to me.

Shopping Cart Validation

Recently I had the task to ensure a mock shopping cart app on the web was working correctly. I decided to play with the darn thing to gain insight on all the things I needed to check. At first glance all seemed well. I added items to my cart. The total amount at the bottom seemed to be incrementing correctly. Then I found some disturbing behavior.

When I removed an item from my cart, it did not always accurately decrease my total price. It did work sometimes. But it did not work most of the time. There was also some weird errors when I removed the last item in the cart. I wanted to go further. I wanted to assist the developer in figuring out what was going wrong.

Initially the incorrect total price seemed random. Then I started calculating the amount it would decrement. Although it was not the amount of the item I removed from the cart, it would decrement by the sales price of some other items in the store. I found that the error would be predictable when I removed items from certain positions in my cart list. Those were the details that let the developer hone in on the exact problem. The cart was implemented in JavaScript. Multiple variables were used in the code. However some of them were supposed to be the same variable. The real lesson from this is that a tester can shine some great light into the nature of bugs. This can help get the bugs resolved faster.

TestComplete by Smart Bear Software

Smart Bear Software has announced the newest version of their TestComplete package. This has a number of tools for testing. I think it aims to allow people with limited technical skills to become productive in testing. However there are some hooks in there for advanced users as well.

There is a Test Visualizer that let's you set up tests by clicking around in the user interface. There is also another tool that let's you type in free form text of the test. It then reads your text and generates test scripts. That seems pretty amazing.

The tool also has a Data-Driven Loop Wizard which let's you import data from a file to create tests. Finally for the seasoned testers, there is the JavaBridge. It allows you to use an API to grab data from within a Java application. That is sweet. If I get into more Java development, that feature alone could be worth the cost. Pricing starts at a grand for the Standard version.

The Many Faces of Quality

I read an insightful article on the different quality professions out there. These include quality assurance, quality control, verification, and validation. These terms are often confused and used imprecisely. So what exactly do they refer to.

Quality Assurance deals with processes. That does not necessarily mean testing. In fact, QA can involve review of a design or requirements document. This is in line with verification, which checks that development was conducted correctly.

Quality control deals with the product of software development. This is achieved through the execution of tests. It is closely related to validation, which is the execution of code.

Visual Studio Ultimate

There are a number of tools in the Visual Studio Ultimate Edition that help the test mission. These tools are included in the price of Microsoft's MSDN subscription. Some of these tools include Test Manager and Lab Manager.

Test Manager is meant for non-techies to developer test cases. This helps eliminate non-technical testers clicking buttons manually.

Lab Manager gives you an environment to conduct tests. It works with Intellitrace and Hyper-V. Unfortunately Lab Manager does not support Silverlight applications. This tools does require a lot of hardware to run on.

Unit Test Worth

Just read a rant that unit tests are all but worthless. Is this true? Let's start at the beginning. Unit tests are supposed to validate independent pieces. They attack at the function level. Code with many dependencies do not work well with unit tests.

Unit tests may help find some small errors. However isn't that what debugging is for? I am speaking hypothetically here. You are supposed to be able to refactor your code and run your unit tests to check whether it still works. Unfortunately unit tests may be too brittle to still work after refactoring.

Developers are optimists. Therefore their unit tests may just choose the easy cases. That is not much of a return on investment. However you should use a tool like Jester. It rewrites your Java code to tweak the logic a bit. If you Jester your code, and the unit tests still past, your tests are not doing much.

Acceptance tests are better predictors of whether your code works. These can be employed after a refactoring to make sure the code still works. Some people still swear by unit tests. They have stories of doing 100% coverage. Then the eventual deployment had no problems at all. I think the jury is still out on unit test effectiveness.

Crowdsourcing

You have heard of outsourcing to lower costs to testing. But have you heard of crowdsourcing? This is a new spin where a company has independent testers all over the world. You company contracts with the umbrella company to get your tests executed. You only pay for defects found.

This method has the advantage that you do not need to set up the outsourced testers. A company has already done that for you. You also only have to pay for results you verify. A test cycle can run you as little as $5k. I just wonder what quality of tests you can get for such a small price. It could be worth a try.

Risk Based Testing

How do you prioritize testing? You can't do everything. So you should spend your time on tests that have the most bang for your buck. Risk based testing attempts to solve this problem.

There are two dimensions to risk. One is the technical dimension. This is where the developers use new technology. Another dimension is the business one. Where in the apps is there a high cost to the operations if there is failure?

You can use the two risk dimensions to prioritize testing. The areas that involve both a technical and business risk need the most testing resources. Other areas that have either one or the other dimensions of risk should be tested next. The low risk areas can be saved for last or skipped entirely. The result is that you spent your time wisely.

Silverlight Testing

How do you test a Silverlight application? That was the topic of a blog post I read recently. One way is to construct a Silverlight test harness. This is where you modify the application that is being tested. You put in hooks to work with your app tester. Let’s discuss this idea.

Currently Silverlight 4 is in beta. The latest release version is Silverlight 3. You can code up a test application. A good programming language for the test app is C#. This test app can send messages to the app you are testing. These messages interrogate the application for its current state.

By default a Silverlight app will take up the whole screen. So you need to make sure your test app can stay on top, or share the screen real estate with the app being testing. The use of Local Messaging is a simple way to communicate with the app being tested.

Simplicity is the main idea here. You do have to modify the app being tested. This feels a little more like unit testing to me. However it is a light version of it. There are alternatives to testing Silverlight apps. Perhaps I shall delve into those other techniques in a future post

Beauty in Testing

My favorite test magazine had a guest editor for last month's issue. He picked an excerpt from his book on beautiful testing for one of the articles. I guess when you are the boss, you can give yourself a competitive advantage. Let's see what this guy had to say about the beauty in testing.

You got to know who you are testing for. You must know your stakeholders. Otherwise you cannot please them. This can be anybody from developers to managers to support staff to users. Many of these people could have unrealistic expectations of your work. Get used to it.

It can be hard to impress the stakeholders. They want more than you finding a lot of bugs. Developers may want you to do the opposite. Metrics can help you quantify your success. I just want testers to find problems early so we can fix them on the cheap.