Verification and Validation

Test Data Generation

2012-01-24T23:01:00.004-05:00

We have a big need to generate test data to verify all kinds of functionality in our system. Personally I like to use Oracle PL/SQL to do the job. This is very practical. The data needs to be in our Oracle 11g database. What better way than to use some code compiled into the database?

Now PL/SQL does have some more advanced programming constructs such as collections. However you can't beat high level languages such as C++ or Java to have advanced features. I have been experimenting with writing some utility apps in Java.

The next step will be to start writing some Java apps to generate test data. I know how to use JDBC to do simple database queries and DML. Let's see where this takes me.

Automation Saves the Day

2011-10-13T15:41:00.002-04:00

We have a new developer working on fixing some bugs. She got assigned a tough part of the system. The actual code is not too difficult. The testing is a real chore. The processes need a lot of requirements to be met before any of the data is processed. I have gone through this pain in the past while trying to test out large changes to this subsystem.

Luckily I performed a lot of tests and set up a data generation system. This system creates just the right data to test out all parts of the subsystem. I found this data generation to have come in handy when the test team could not figure out how to test any of this stuff.

The new developer was running a little behind schedule. She was finding, like all would, that it was next to impossible to test anything in this subsystem. I pointed her to the testing data generator and her problems were solved. Now she is teaching some testers how to use this tool to conduct their own test data sets. This tool is going a long way to making peoples' lives easier. Mine included.

Independent Verification

2011-08-30T10:29:00.003-04:00

Our project has an independent testing team. Their goal is to verify that our software meets the requirements and has minimal bugs. The lead of this team recently left the company. Her replacement seems like she has management potential.

We just got through a big release of our software. It was to be delivered on a Friday. I was taking it easy that Friday since it had been a long week. Things were supposed to go out easily, with a formality of checking our production install.

One of the testers noticed some errors in the application. It turns out the changes for the release were not in order. The thing that was disturbing was that the software changes had passed test. Now it looked like the pass was achieved in error. Now the delivery was late and we had an emergency on our hands.

They needed some testing to get done. The guy who was supposed to have completed the testing had already left the company. Darn. The existing testers had little clue as to how they might test the system. What did we do? The developers were brought in to rerun the testing. Sounds like a massive fail to me.

Shopping Cart Validation

2011-03-05T01:13:00.002-05:00

Recently I had the task to ensure a mock shopping cart app on the web was working correctly. I decided to play with the darn thing to gain insight on all the things I needed to check. At first glance all seemed well. I added items to my cart. The total amount at the bottom seemed to be incrementing correctly. Then I found some disturbing behavior.

When I removed an item from my cart, it did not always accurately decrease my total price. It did work sometimes. But it did not work most of the time. There was also some weird errors when I removed the last item in the cart. I wanted to go further. I wanted to assist the developer in figuring out what was going wrong.

Initially the incorrect total price seemed random. Then I started calculating the amount it would decrement. Although it was not the amount of the item I removed from the cart, it would decrement by the sales price of some other items in the store. I found that the error would be predictable when I removed items from certain positions in my cart list. Those were the details that let the developer hone in on the exact problem. The cart was implemented in JavaScript. Multiple variables were used in the code. However some of them were supposed to be the same variable. The real lesson from this is that a tester can shine some great light into the nature of bugs. This can help get the bugs resolved faster.

TestComplete by Smart Bear Software

2010-08-20T11:23:00.004-04:00

Smart Bear Software has announced the newest version of their TestComplete package. This has a number of tools for testing. I think it aims to allow people with limited technical skills to become productive in testing. However there are some hooks in there for advanced users as well.

There is a Test Visualizer that let's you set up tests by clicking around in the user interface. There is also another tool that let's you type in free form text of the test. It then reads your text and generates test scripts. That seems pretty amazing.

The tool also has a Data-Driven Loop Wizard which let's you import data from a file to create tests. Finally for the seasoned testers, there is the JavaBridge. It allows you to use an API to grab data from within a Java application. That is sweet. If I get into more Java development, that feature alone could be worth the cost. Pricing starts at a grand for the Standard version.

The Many Faces of Quality

2010-08-19T13:25:00.002-04:00

I read an insightful article on the different quality professions out there. These include quality assurance, quality control, verification, and validation. These terms are often confused and used imprecisely. So what exactly do they refer to.

Quality Assurance deals with processes. That does not necessarily mean testing. In fact, QA can involve review of a design or requirements document. This is in line with verification, which checks that development was conducted correctly.

Quality control deals with the product of software development. This is achieved through the execution of tests. It is closely related to validation, which is the execution of code.

Visual Studio Ultimate

2010-08-18T13:18:00.004-04:00

There are a number of tools in the Visual Studio Ultimate Edition that help the test mission. These tools are included in the price of Microsoft's MSDN subscription. Some of these tools include Test Manager and Lab Manager.

Test Manager is meant for non-techies to developer test cases. This helps eliminate non-technical testers clicking buttons manually.

Lab Manager gives you an environment to conduct tests. It works with Intellitrace and Hyper-V. Unfortunately Lab Manager does not support Silverlight applications. This tools does require a lot of hardware to run on.

Unit Test Worth

2010-05-25T17:03:00.003-04:00

Just read a rant that unit tests are all but worthless. Is this true? Let's start at the beginning. Unit tests are supposed to validate independent pieces. They attack at the function level. Code with many dependencies do not work well with unit tests.

Unit tests may help find some small errors. However isn't that what debugging is for? I am speaking hypothetically here. You are supposed to be able to refactor your code and run your unit tests to check whether it still works. Unfortunately unit tests may be too brittle to still work after refactoring.

Developers are optimists. Therefore their unit tests may just choose the easy cases. That is not much of a return on investment. However you should use a tool like Jester. It rewrites your Java code to tweak the logic a bit. If you Jester your code, and the unit tests still past, your tests are not doing much.

Acceptance tests are better predictors of whether your code works. These can be employed after a refactoring to make sure the code still works. Some people still swear by unit tests. They have stories of doing 100% coverage. Then the eventual deployment had no problems at all. I think the jury is still out on unit test effectiveness.

Crowdsourcing

2010-04-05T20:51:00.003-04:00

You have heard of outsourcing to lower costs to testing. But have you heard of crowdsourcing? This is a new spin where a company has independent testers all over the world. You company contracts with the umbrella company to get your tests executed. You only pay for defects found.

This method has the advantage that you do not need to set up the outsourced testers. A company has already done that for you. You also only have to pay for results you verify. A test cycle can run you as little as $5k. I just wonder what quality of tests you can get for such a small price. It could be worth a try.

Risk Based Testing

2010-04-04T22:57:00.002-04:00

How do you prioritize testing? You can't do everything. So you should spend your time on tests that have the most bang for your buck. Risk based testing attempts to solve this problem.

There are two dimensions to risk. One is the technical dimension. This is where the developers use new technology. Another dimension is the business one. Where in the apps is there a high cost to the operations if there is failure?

You can use the two risk dimensions to prioritize testing. The areas that involve both a technical and business risk need the most testing resources. Other areas that have either one or the other dimensions of risk should be tested next. The low risk areas can be saved for last or skipped entirely. The result is that you spent your time wisely.

Silverlight Testing

2010-03-04T16:19:00.002-05:00

How do you test a Silverlight application? That was the topic of a blog post I read recently. One way is to construct a Silverlight test harness. This is where you modify the application that is being tested. You put in hooks to work with your app tester. Let’s discuss this idea.

Currently Silverlight 4 is in beta. The latest release version is Silverlight 3. You can code up a test application. A good programming language for the test app is C#. This test app can send messages to the app you are testing. These messages interrogate the application for its current state.

By default a Silverlight app will take up the whole screen. So you need to make sure your test app can stay on top, or share the screen real estate with the app being testing. The use of Local Messaging is a simple way to communicate with the app being tested.

Simplicity is the main idea here. You do have to modify the app being tested. This feels a little more like unit testing to me. However it is a light version of it. There are alternatives to testing Silverlight apps. Perhaps I shall delve into those other techniques in a future post

Beauty in Testing

2010-02-12T23:44:00.002-05:00

My favorite test magazine had a guest editor for last month's issue. He picked an excerpt from his book on beautiful testing for one of the articles. I guess when you are the boss, you can give yourself a competitive advantage. Let's see what this guy had to say about the beauty in testing.

You got to know who you are testing for. You must know your stakeholders. Otherwise you cannot please them. This can be anybody from developers to managers to support staff to users. Many of these people could have unrealistic expectations of your work. Get used to it.

It can be hard to impress the stakeholders. They want more than you finding a lot of bugs. Developers may want you to do the opposite. Metrics can help you quantify your success. I just want testers to find problems early so we can fix them on the cheap.

Outsourced Testing

2010-02-10T19:58:00.002-05:00

It is becoming popular to outsource testing. How do you make sure it turns out well? You might need to appoint a third party adviser to oversee the activity. You should also define what your goals are. Do you want a little extra help to test? You alternatively may be looking for someone else to do all the testing. Or perhaps there is a skill lacking in your test organization. Define your objectives up front.

Choose outsourced help that has a process which matches your own. This results in less pain. To ensure you don't get ripped off, try to negotiate as fixed price contract for the outsourcing. Then the risk is put in the hands of the outsourcing company.

Exercise caution when you grow the outsourcing effort. Also watch out for expensive tool choices by the outsourcing company. They may not have your best interests at heart. More importantly, the costs of the tool might be prohibitive. Luckily our own project has a good sized test team. They also have some really good talent. So far they have only outsourced a little work to other employees in our own company. They have been short term engagements. But you never know. We might look outside for some help to test.

Testing Trends

2010-02-09T18:41:00.003-05:00

What's new in the state of software testing today? That was the question posed to the editor of Software Test and Performance magazine. It is difficult to get the top three. There are many important topics in modern testing. What did the editor choose?

One is the creation of testing centers of excellence. Next is the use of test outsourcing. The third is delivering value to stakeholders using a technique called beautiful testing. These topics are the contents of the latest issue of the magazine.

I will be talking more about these topics in the future as they are the state of the test art.

Web 2.0 Testing

2010-01-21T00:43:00.001-05:00

There is a new breed of applications that have been labeled Web 2.0. They require a different test strategy. Web 2.0 apps are created by the users themselves. For example, users can create mashups where they take different apps and integrate them into a new system. This complicates verification. What is a tester to do?

The first thing is that you need to use the software like a user for a while. You can also ensure that actions get logged to disk. That way you can mine the log. The log can be used to drive future tests. You can use it to set up regression tests.

You got to do all kinds of data inputs validation testing. Sure you can try some special characters as input. But you should also try a whole range of weird characters. Try to simulate a cross site scripting attack. Your app better be on lock down for this.

A good goal is to test small feature sets. You should also clone your production data to seed your test database. If necessary, strip out any private data from the set. Above all ensure you can roll back to a stable release at any given time. This is the web. Things happen fast.

More on Fuzzing

2009-12-07T19:07:00.002-05:00

Development magazines are paying more attention to fuzzing. Fuzz testing is subjecting an application to random data to see what breaks. This is applicable to web apps. It can be part of a comprehensive security test. Fuzz testing let’s you try inputs that stimulate paths a tester might not normally think about, or have time to manually perform.

There are free fuzz test generators out there. Some are Peach Fuzzer, MiniFuzz, and Spike. There are also some commercial products to assist with fuzzing as well. But let’s get back to why fuzzing is a good idea. Normally you test positive aspects of requirements. However you should also be testing for things not covered by any requirement. This is where fuzz testing can come in real handy.

I am a developer. Right now I am not employing fuzz testing yet. However I hope the independent test team on my project is hot on the technique.

Pairwise Testing

2009-12-02T22:18:00.001-05:00

Testers often need to verify systems with many related variables. This can result in a huge amount of test cases. Often the combinations can be out of control. However a technique called Pairwise Testing can reduce the amount of test cases while still covering all scenarios. The hard part of this is generating the minimal set of cases. This is a good time to use a program to do the hard work.

One pairwise test case generator is Pairwise Independent Combinatorial Testing (PICT). This product is based on a tool that Microsoft themselves use. There is a recent new tool called QICT which accomplishes the same task. The beauty of QICT is that the source code is provided free of charge.

Pairwise testing is good for many test scenarios. Some good examples are testing databases and configuration data. I know that I often have an onslaught of test cases that I think need to be executed to provide full coverage. I might just have to use a tool such as PICT to make my job easier. This can potentially reduce the test time necessary, and improve my test throughput.

.NET Runtimes

2009-12-01T17:54:00.001-05:00

Microsoft is changing the way that new versions of the .NET framework impact existing applications. Here are the goals that Microsoft has. Installing new versions of the .NET framework should not break existing applications that are installed. The new version should also be backward compatible. Finally applications should run against the version of the .NET framework they were released under.

These new Microsoft goals are realized in the .NET framework version 4. This is a statement that must be tested thoroughly. Previously the new .NET frameworks would be used for any existing installed application. However now Microsoft claims the latest version will not break existing apps. I can envision the need of virtual machines to test .NET framework upgrades, and the impacts on applications which depend on prior versions of the framework.

Microsoft claims that the .NET framework version 4 is the most backward compatible version yet. My recommendation is to trust, but verify.

What Matters Most

2009-11-25T00:37:00.003-05:00

I read a whole issue of a testing magazine tonight. There was one article that stood out in my mind. It was an interview with a guy that had started as a junior tester in his company, and worked his way up to CEO. They asked him what is key for a tester to succeed. He had some very specific guidance.

A tester must know what the end user needs. The tester must also be aware of what the user wants as well. You got to know the business. You get this by getting involved with operations. Some other important traits include strong analytical skills.

This advice matches what I see on my own project. We have a bunch of GUI applications. There are some programs that run on the server that do loading and computations. Add on a couple back end programs that do communications and you have our system. There is nothing special with this layout. However the business knowledge is king. If you can somehow figure out what the heck the users are trying to accomplish, the system makes a lot of sense. Otherwise you will be totally lost.

The Trouble with ASP.NET

2009-11-20T02:37:00.002-05:00

It is great if you can automate testing of software with unit tests. This is not too easy with ASP.NET from Microsoft. ASP.NET is a web development framework. There are multiple issues making matters difficult.

The first setback is that testing web sites requires transmission of HTTP requests, simulating browser clicks. The second trouble is the output. It is all HTML which is messy.

How do a lot of testers deal with web site verification? They end up clicking around looking for problems. Ouch. Microsoft's answer to the woe is ASP.NET MVC. So far I have no experience with the technology. However Microsoft is pitching it as the way to simplify the testing of web apps. We shall see.

Debugging Wrap Up

2009-10-13T01:21:00.003-04:00

Two more trustworthy rules for debugging are to keep an audit trail and to only declare a bug fixed if you fixed it. The first rule is self explanatory. Just make sure you use a consistent format for all your logging. That will help later when you try to filter all the audit data.

If you think you have a fix, remove it and see if the problem comes back. If not, then you probably have not fixed the problem. You have to be sure of yourself here.

Here are some last ideas. Try and get a diagram of the whole system you are debugging. Feel free to ask for help. When you do get help, you should request that assistants just report the symptoms. You don't need theories. Good luck with your debugging. I have made a career out of this activity myself.

Details are King

2009-10-10T00:02:00.004-04:00

Here are more tips to debug a problem. Keep testing older versions of the software until the problem goes away. Now you have narrowed down the build when the problem was first introduced. It should be easier to determine what changed at that time.

There are many instances where the smallest detail is the key to the problem. Let me tell you a story. My customer's acceptance team keep making the application crash. They said it happened at random. That never happened to me or any internal testers on my project. So I enlisted the help of the people experiencing the trouble.

I asked the testers to notice everything that happened before and after the problem. What was the weather like outside? Were the lights on? Did you sit close to the desk? Was there anybody sitting next to you? What time did it happen? You know. I needed to know all the facts, even if they seemed insignificant. Guess what? This helped us crack the case.

One tester found that when she swiveled her chair to the right, the application sometimes broke. Also when she dragged her spiral notebook from left to right the problem always happened. This keep attention to details helped her identify the cause of the problem. Each time she was doing these movements, she was pressing the space key on the keyboard. Sure enough that was the cause of the application aborting. Amazing.

One Thing at a Time

2009-10-08T19:53:00.003-04:00

Here is a guideline for debugging. Only change one variable at a time. I am not talking about programming variables. I am talking about one factor in the state of the system.

This can be taken a bit further as well. Make a change and check the result. If the bug was not fixed, undo the change and make exactly one more change. Repeat. The converse is to fix just one thing at a time as well. Then you can go on to other bugs.

Now let's talk a little about tools. Don't throw them away even if they seem like one time tools. You should actually build the tools into the initial design of the system. Make it easy to inspect the internal state of the system during runs.

Next time I will talk about audit trails and logging in general.

Say No To Guessing

2009-10-07T22:18:00.002-04:00

Here is a pseudo rule for debugging: Don't make guesses. What you should do instead is to compare runs that exhibit the problem with those that do not. That way you can narrow down the exact effects which contribute to the problem. This leads us to the Divide and Conquer rule I wrote about previously.

Theories about what the problem might be are not worth much. In fact, the can be counter productive. You will most likely waste time tyring to check whether your theory is correct or not. You end up right where you started, except it is much later in the day.

Now this is not to say that you should never harbor a guess at the source of a bug. You should just limit the amount of guesses you make. Once you get good at debugging, you can use few guesses to get to the source of the problem.

Stay tuned for a commentary on the use of tools for debugging. I will leave you with another debugging gem. Change but one variable each time you run a test. This will help focus your cause and effect research.

Hints for Debugging

2009-10-07T00:20:00.003-04:00

Here are some more hints and tips to debug problems in systems. Let's start with some more general rules for debugging:

Duplicate the problem
Do not think too much
Narrow down and focus

Ok let's now get back to the tips and tricks. You got to really know the tools you use. That includes knowing the limitations of the tool set.

Don't guess. Act like a detective and look at clues. Once you think you have a solution, run the system with the same conditions that exhibited the problem and demonstrate that the specific problem was indeed fixed.

Record each step required to make the problem occur. Do not fake the problem. Actually make it happen like it does in real use. You might need to employ automation to speed up the process of creating the bug. That is fine.

I will continue next time with some more general rules for debugging. You know I will come back to tools. And I will examine cause and effect.

Understanding the System

2009-10-04T09:55:00.002-04:00

You must understand the system before you can begin debugging it. Here is a novel thought. Read the instructions. This means you should go over the specification for the system. You can also read source code comments. Review the design documentation to gain understand.

Now you must understand one other thing which reading the documentation. You need to take what you read with a grain of salt. Don't trust everything you read.

On the other hand, you really need to know the contents of the user manual. Read it many times until you know it well. You must have a firm grasp on the fundamentals of the system. When I come back, we will get to the second rules of debugging - Make It Fail.

Debugging

2009-10-03T20:41:00.002-04:00

Halfway though reading a book on debugging, I realized that I had read it before. It was a good book. So I completed it for a second time. There were a lot of good stories and examples.

The book was written in 2002. However the rules it presented were timeless. The author was actually a hardware/electronics dude. But he nailed the software debugging process down well.

The goal of the advice in the book is to help you find out what is wrong with a system quickly. If you fix your commercial software first, you get a competitive advantage. That translates to more money for your company. So you had better listen up.

When you figure problems out fast, you get to go home quicker too. I know this all too well. If I do not solve problems, I am there to midnight, and I have to return on Saturday morning.

It is difficult to find debuggers who follow the rules in this book. That means if you do learn and use these rules, you will be valuable. Let's first define some terms. Troubleshooting is trying to find out what is broken. Debugging is finding out why. This book is about debugging.

Let's get on with it. The first rule is "understand the system". There are eight more rules. I will be sharing some of these in future posts. Until them, happy hunting.

Acceptance Test Limitations

2009-09-26T12:28:00.002-04:00

Our customer has a pretty mature organization. They have their own acceptance test group. They are all employees of the customer organization. Their goal is to test everything we do to ensure the stuff with work in Production.

A while back we delivered a major release. The customer has found all kinds of problems that we missed upon delivery. Luckily we were able to fix most of those issues. However there was a significant piece we were not able to deliver on time. We have been trying to deliver the results piecemeal. However the acceptance team is discovering this effort is producing a lot of bugs. Back to the drawing board.

Recently I read an article about the limits of acceptance tests. They cannot conduct a full regression test. You should not depend too heavily on tools for the tests. The testing only adds value to the effort. It needs management support. And the testers need to learn many skills for the project to benefit. I can honestly say that our customer organization has not let these limits hold back their testing. The acceptance test team really does add significant value to the project. My hat is off to them.

Stolen Art of Unit Testing

2009-09-23T19:31:00.002-04:00

I just read a blog post by the author of the book The Art of Unit Testing. He said a PDF version of his book was being shared illegally. The guy even posted a link to one such illegal download. So I downloaded it. I read the first chapter. That's as far as I got. I won't be buying the book.

Even though I took a pass, there were some things to learn from Chapter 1. Your project may fail even if you have unit tests. If fact the unit tests can be a liability. It does not matter whether you employ Test Driven Development. You might still fail. I believe that.

The author does encourage you to use a unit test framework. The examples in his book use NUnit. He also states that unit tests should run quickly. They should be able to finish in a few minutes. He must be talking about some specific unit tests, not the tests for a big system. You should also set up you tests so they do not require manually configuration before running.

I will give the guy one thing. The PDF version of his book was pretty good for a PDF. Most PDFs I view are boring black and white things. He used color appropriately. Since he was getting ripped off by people passing around free copies of his book, I will give out a link to buy The Art of Unit Testing on Amazon. Good luck Roy.

Automated Acceptance Tests

2009-07-28T20:41:00.002-04:00

Our customer is a savvy one. They have a full time team that conducts acceptance tests of our software. This team keeps on top of new requirements and the overall business. They make sure they are in the loop at all times during the decision making process. In the end, this team makes the final call on whether our software is ready for prime time.

Here is something new I have read that might help our customer's acceptance test team. That is automated acceptance tests. No that will not put them out of a job. It may make their lives easier. You write up your test cases in some English like language. Most of the time you focus on positive testing. That is, you test the normal case. The benefit it that you can run the tests against every new release.

Currently my customer does acceptance tests manually. That requires them to load up on resources during the software delivery period. It also means that the testers are forever generating huge volumes of interesting test data. That is a good step. However the tedious manual execution of the tests makes me dread the job. I must confess that they are successful in finding where our software breaks down or does not meet the business need. Automated acceptance testing might take their game to the next level. Do I hear a promotion in some acceptance tester's future?

Test Automation

2009-07-27T23:48:00.002-04:00

A common theme I read a lot about is the automation of testing. At the root this is wise move. Automate your testing. Then you can code up some changes, and retesting quickly by kicking off the tests. The big three places where automation can be used are unit, regression, and system acceptance tests.

On our own project, most of the test phases are manual. Once in a while I will automate a unit test if I have the time. However we normally have no time in development. Everything is rush rush rush.

Let's step back a minute and define our testing. Unit tests is where small units of code are tested. Normally you write your test cases in the same language that the product is implemented in. You try to cover all the lines of code in your tests. There are popular frameworks which assist in this endeavor.

Regression tests are supposed to cover all the functionality in your product. This is difficult to achieve. There is not really a framework you can pick up to help with this. That is because the products themselves are very diverse. The regression tests themselves can check different characteristics of the system. This includes things like performance.

This one is getting long. So I will pick up with System Acceptance tests next time.

Time for TDD

2009-06-24T21:31:00.002-04:00

Test Driven Development (TDD) is gaining more acceptance in programming circles. But at least one question still remains. When should you apply TDD? In other words, when do you develop the tests? A simple answer is to create the tests first. That way you have tested code from day one. However this may not always be the right way to go.

I read a blog entry stressing that TDD is beneficial for maintenance work. You should have mature code complete with self tests. However the very first class you write for a new system may not need tests right away. Testing may in fact get in the way of rapid progress. I myself am divided on this very topic.

In a perfect world, you would have understood all the requirements and performed a full analysis and design on the project. At that point you are ready to add TDD before you write the first line of code. The real world is never perfect. So this may not be the way to go. Sometimes you need to write some code to figure things out. I would not call this production code. This may be throw away code. And that code does not need TDD applied to it.

We do not use TDD in general on my project. I have tried it. For small tasks it seems to work well. But when everything is crazy and we are terribly behind schedule, the TDD goes out the window and we hack together solutions. Sure it is not the best way to go. But it is life. There is probably a time and place for TDD. That time may not be before you write any code.

Cost of TDD

2009-06-11T18:25:00.002-04:00

I read a blog about a developer that tried Test Driven Development (TDD) for a day. He found that a useful way to generate the tests is to debug your code. When you find errors, you can write a test that checks for that bug. This would be perfect for my project. It is a maintenance project. We get trouble tickets all the tine that we resolve. It would not take much more effort to integrate the tests for the bugs we fix into a automated test script. We already do the unit testing for each fix anyway.

Getting back to the subject of TDD, the developer who tried it and wrote about it did not like it. He said the TDD technique led to very little actual debugging. He did gain some new respect for the method in general. However he would still not use it for everyday programming purposes. While the effort did seem to result in higher quality code, it took much longer to develop code that way.

When can you use TDD? I know we can’t use it on our project. We hardly have the time to race through coding and hack together some solutions on my project. If TDD causes the development cycle to double in time, we would never get anything done at my job. Here is the funny thing about this.

Perhaps we really are the best candidates for a TDD approach. The problem is that it would be a hard sell. One of the main things that drives our schedule is customer needs. And our particular customer wants more functionality in less time. They don’t seem to care about our methodology. Yes they do not like bugs. However they will tolerate bugs if it means they get a lot of features right away.

Production Volume

2009-06-02T22:41:00.003-04:00

I read this blog about how you should test using volume comparable to production. Big hard disks cost little these days right? Wrong. Well yes some el cheapo hard disk can be found for a bargain. However that disk will never match the characteristics of our production server box.

Now I don't know all the details. But we have a massive box to run our production system. There is probably something like 128 processors on the darn thing. And there is more RAM in there than you could find on a mainframe.

Don't get me started about the super speed disk array we got powering our database. But I guess you get the point. You cannot compare your little bitty workstation with a production box. To do some is futile. I guess the only pearl of wisdom from the blog post I read was that you should not do tests if you only have a couple records in your tests tables. Point taken.

Oracle Application Suite

2009-05-03T15:24:00.002-04:00

You would think a tool called Oracle Application Suite would be focused on some type of database work. Specifically it sounds like it does something with the Oracle Database. However I just read an article that tells how this tool can be used for testing.

The Oracle Application Suite is actually a part of Oracle Enterprise Manager. Now I have heard of this product. However I have seen DBAs use this to manage Oracle databases. But I digress. Let's get back to how Oracle Application Suite can be used to test.

There are three things that Oracle Application Suite does to assist testing. It helps load testing by simulating a lot of users logging on to the database. It also performs functional testing by automating test scripts. Finally it helps with all kinds of chores related to web application testing.

Now like most Oracle tools, I hope this one is free. If so I plan on checking it out. On a final note, I understand this tool integrates with other products. I guess I will have to play with it to find out how that works.

Fuzzing

2009-04-29T17:37:00.002-04:00

I read an article in Software Test and Performance magazine about the topic “Fuzzing”. This was the first time I had heard this term. However recently I heard some of my team’s tester talking about the issues involved with fuzzing. So I thought I would write about this as well.

Fuzzing is a negative software testing method. That means you do not test for whether the system works correctly with correct inputs. Instead fuzzing concentrates on things such as inputting malformed data to an application. It also involves attempts to defeat system security. This used to be a hacking technique. Now it has been formalized as a testing methodology.

Requirements themselves can be written in a positive and negative form. That is, requirements usually dictate what the software must do. However some requirements also define the desired system behavior by describing also what the system shall not do.

A way to perform fuzzing is to start with a test case. Then you can modify the test data and rerun the test. The more general realm of testing is robustness testing. This is where you test how the system behaves when there are invalid inputs. One such form of robustness testing is fuzzing. The main purpose of fuzzing is to identify security problems in the system.

When I was with some of our team’s testers, I heard them talking about their negative testing. I did not drop the “fuzzing” buzzword with them. However it was good to see that they are addressing modern topics in the test world.

Changing Test Data

2009-03-23T21:43:00.002-04:00

Our customer reported a bug in production. There were many theories as to why this problem was happening. I told people that these were all interesting theories. But these theories, like any opinions, were really not worth much. When I got some time, I dug into the problem. I traced the problem back to the line of code that caused the problem. In the end I determined some configuration data was missing. That resulted in the good code causing a bad transaction. I unit tested my fix by hacking together some data to meet the specific scenario where the error occurs.

Then the fix went to our internal test team. A tester called me up and asked me about the problem. I explained the problem from top to bottom. You would think that would be sufficient for a tester to run with the fix and test it out. However I got back a question that I seem to get a lot from our internal test team. How do I test that? In essence they wanted to know how you set up the data conditions that result in replication of the error.

Now I am sympathetic somewhat to the testers. They do not have the source code in front of them. The documentation for our system is merely average. So it might be unreasonable to assume they had all the tools they needed to figure things out. Therefore I outlined a set of steps that could be taken to replicate this problem.

Development had previously produced a tool which generated input files for our loads component. You could use the tool to generate input files which contain the data the make the problem happen. I recommended that the testers generate up the input files with the tool, run them through the loads program, and duplicate the problem.

I guess this was too much work. The testers tried to simulate all that work by taking data previously mocked and loaded, and performing updates on that data. This is not a bad decision. But you have to know what you are doing. Unfortunately they were unable to replicate the problem. My boss told me to get in there and help them out to get this fix shipped. Essentially I had to reverse engineer our loading code. Then I kept giving them all the data they needed to change manually in order to duplicate the problem.

This is not a good scene. I have things to do. If you want to add value as a tester, you got to do the hard work. Then you will come out smarter and be able to wing it one day. But not now guys.

Testing is Hard Work

2009-03-17T17:10:00.004-04:00

Yesterday the test manager on my project came to visit me. She was planning the tests for some new changes I am working on. She asked me to explain some of the changes as they did not make much sense to her.

I tried to get into the details with her. She took some notes. Then she said she would get some people from her team to dig into it. Today I got a message from her. She said her team could not make heads or tails of the requirements, and asked me to help them.

Well I shot off an informative email on the current state of the application, and what the customer wanted instead. Then I got a response on how they could test that. I gave the test team a little more information. Then I got another response asking how they could set up data to do the tests.

I did not want to just provide them with my own unit test plan. Then they would not be adding any value to the work. In an ideal world, they would be able to figure out how to test according to the documented requirements. Unfortunately the requirements team does not know much more than the testers. So I continued to try to give more ideas until they felt like they had enough to go on. How do we get to that ideal world where there is independent validation of my work?

Application Lifecycle Management

2009-01-18T17:01:00.001-05:00

Recently I subscribed to Software Testing and Performance magazine. I just got my first issue. There were 30 pages in the issue. Within these pages, there were 3 large ads for conferences. I started to get suspicious that this magazine might be a front for advertising. Then I found a couple articles with technical content. One of these was the ST and Pedia column.

This particular column boasted that it was “translating the jargon of testing into plain English”. This month’s column focused on describing parts of the ALM. There was one main problem with this. It did not define what ALM stood for. Perhaps every tester knows this acronym by now. I think I will email the many testers I know and see if that is true.

Regardless, I did not know what ALM stood for. That made the whole column a damper. It was not quite clear how the different parts were related. Come on guys. You spell out SOA (Service Oriented Architecture). SOA is more common than ALM. Perhaps this was just an oversight. You should always write for an audience that does not know all the buzz words and acronyms. I will try to take this lesson to heart and spell out acronyms I use myself in my blog.

The whole point of receiving a print magazine like Software Testing and Performance is that I like to read it in places where I do not have a computer in front of me. Once I got to the computer, I was able to head over to Wikipedia for guidance. There I found that ALM stands for Application Lifecycle Management. But at this point, I no longer needed or wanted to read the magazine. I was in Wikipedia for my research.

I don’t know how else to combat this problem. Maybe I should write to the editor of the magazine. It did not look like the author’s of the column in question had listed their email addresses near the column. That is not too good. So far this magazine has shown me it has a lot of advertisements, and seems to be written for a specific audience which know more about testing than me.

True Testing

2008-09-17T18:38:00.002-04:00

A manager on my project wanted to know how some things worked in one of our applications. They needed to speak with a customer. So this information needed to be very accurate. I kind of knew how the application worked. However I did not know all the details for sure. We have a lot of documentation that describes the most intricate details of the applications. Some of this documentation is old. My instinct told me not to trust the documentation. I had no other alternative than to reverse engineer the application behavior through a series of tests.

I spent a lot of time conducting different scenarios to test. It was good that I had some idea on how the application worked. That way I could choose tests where I was not sure of the results. I could also infer a lot from just a few tests results. This was not an exhaustive search of all the possibilities. That would have taken far too long. It took about half a day to get the data I needed.

This technique may not work for everybody. You have to make sure you really get a firm understanding of what is going on behind the scenes. However I trust this more than consulting some documentation or even reviewing the source code. In my experience, it is best to trust nothing. I know that sounds a bit like the X Files. But it is the safest approach.

Now I can say with confidence how the application behaves for some certain behavior. If anybody wants to challenge me, I can ask them where they got their information. Is that just the way it is supposed to be? Did they hear it from somebody a long time ago? Did they read this in some document? I can top all of that. I just tested the actual application out. You can’t beat that kind of experience.

Good Test Data

2008-09-13T22:38:00.002-04:00

You need reliable test data to conduct independent verification. This is more complicated than it seems for some systems. The Software Maintenance blog demonstrates that sometimes even developers have trouble creating test data.

Ideally you can run the system like it operates in a production environment. The closer to the real world you make it, the better the quality of your test data. However this is not always feasible. Sometimes you need to simulate the real world with generated test data. It is these circumstances where you really need to understand the system deeply. There is always the chance that you introduce differences in your data generation that skew test results.

One way to perform test data generation is to do the hard work once. Then you can save a copy of the data that you can use time and time again. There may be some circumstances where you will need to tweak the test data. However the heavy lifting will already be done. This is how we approach test data generation on my project. Some DBAs run processes each year to give us a new data set valid for that year. The results are passed to both the development and internal test teams. The work in generating this test data is not too high. The results are awesome.

Wasted Unit Tests

2008-09-04T19:05:00.001-04:00

I read a blog post entitled “Can Unit Testing Be A Waste” on AgileSoftwareDevelopment. It was written by pbielicki. He recommended you write unit tests at a high level of abstraction. Writing tests for getters and setters is a total waste of time. The goal of agile development is to eliminate waste. You should add tests just in time to verify functionality. You should also use common sense while choosing what to test.

One of the most interesting parts of the blog post was the many comments it received. For example, there was agreement that you need a coverage tool to even know where you stand with unit testing. There was emphasis on the goal of unit testing being a quality product. Another goal is to satisfy the customer.
Here on my project we are supposed to do unit testing. For the most part there is some level of unit test. Previously we had implemented a system of peer review to ensure there was adequate documentation of the unit tests. The peer review also ensured there was enough coverage for the unit test cases. However we have recently stopped performing such peer reviews. This is unfortunate. And I think it is starting to show in the quality of the code we are producing.

I am certain we are not wasting time on unit testing. We try to determine the minimal set of tests that will ensure something we code actually works. Our team borders on the side of too little when developing unit tests. Like many developers, we do not particularly like unit testing. I do it because I like it even less when I ship out very buggy code. That only creates a whole new set of worse problems. My company works on a maintenance contract where we get penalized when there are a lot of bugs shipped out to production. So this is a dollar and cents type of issue.

My manager has said that we have a documented peer review plan. The real problem is that I have never seen it. Thus it is truly not a priority. Yeah if I banged on doors and made a big deal about it, somebody would finally find me a copy of the thing. However it would be worthless if nobody is following it. Ouch. Get ready for a rough ride guys.

The State of Unit Testing

2008-09-02T19:36:00.002-04:00

Python and Ruby developers are big on unit testing. However there is not as much excitement with it in the Java world. Test Driven Development is getting to the point where it is becoming a religion. So if you are doing agile software development, you can expect to also be doing unit testing.

There has been a few recent books on JUnit. Aside from that, there is really not a lot new with JUnit. There are also other unit testing frameworks out there such as TestNG. None of these really help with enterprise systems though. They are very hard to test since they have a lot of dependencies. Unit testing is also very difficult with legacy applications.

Like TDD, agile in general is going mainstream. And unit testing is one of the most adopted practices in software. However unit testing seems to be starting to be on the decline. In general it is hard to afford the time required for unit testing. Most projects and developers do basic unit testing. But if an existing system works, it is difficult to justify extra cost to add unit tests.

The goal of unit testing is better code. There are free tools available for unit testing. So the tool cost is usually not an issue. Good unit tests, like good code, require a lot of time and skill. All developers on your team must unit test for you to get the maximum benefit. You also need to do a lot of work with unit tests when you system changes drastically.

On my own project, we are supposed to do unit testing on all new code and any fixes. Previously we were required to generate documentation on the unit test activities. This documentation was inspected at the time of peer review. Now we only have management direction to make sure we do unit testing and produce some tangible results. However there are no checks to ensure this is done. You can imagine the tendencies given a hectic software life cycle where we work.

For a while I was starting to slack a bit with my own unit tests. I would check a few things out, and decide to ship the software based on that. But I am always up to improving myself. One of the areas I determined needed help was my unit testing. I found myself writing a lot of scripts lately to fix problems. I have decided that for every script I write, I would also write at least one other script that performed a unit test. Many times I end up writing multiple unit test scripts to create data, run the tests, and clean up my development environment. Sometimes this uncovers critical bugs and saves the day.

Going the Distance

2008-08-27T18:13:00.001-04:00

The customer encountered a problem while testing our application suite. We had just migrated to a new set of tools, and a new version of the database on the back end. It took a while for me to get around to researching the problem. Luckily I was able to duplicate the problem in a development environment. I tracked the source of the problem down to some bad test data. Then I checked the database where the customer performed their tests. They had the same problem there. The fix was to write a script to correct the data.

At first when I ran the script to correct the problem, it just bombed. It logged the problems it encountered. So I manually executed each of the statements in the script until I found what the problem was. That’s when I was done with the coding. It was time to test it out. Luckily I had a lot of data that was corrupt in my development environment. I ran the script and it fixed about 5000 records. I used a tool to view the corrected data. In general it looked pretty good. I was tempted to just ship the fix at this point since I had 9 other problem to research and resolve.

Instead I decided to take my unit tests a step further. I figured it would be easiest to write another script to verify all the records were updated correctly. It took about an hour to write this unit test script. That was not too much overhead considering the time it take to investigate and code a solution to the problem. I did find some exceptional scenarios in the script results when I churned the data with a script. I just checked all 5000 outputs records for correctness. At first I just output the number of problems the test script found. Then I actually wrote the problems out to a log for further analysis.

Here in development we always seem to be in a rush because we are understaffed and overworked. But here is the funny thing. If I sent out a script that did not fully work, I would waste a whole lot more time dealing with the repercussions. Why not spend a little more time and get it right the first time? I love to code. Writing a script to verify the unit test results is just a little more coding. So it is in fact fun work. Now just don’t tell the management staff that. Otherwise they will get the wrong idea.

Testing Database Code

2008-08-21T18:00:00.001-04:00

Last night I read a blog entry entitled “How to Unit-Test Code That Interacts With a Database”. It stated that such testing is hard. But you still want to fully test the database interaction. The blog recommended that you connect to a separate database to do your unit testing. Each developer should ideally have their own database. In addition, the database should be reset to the state it started in prior to the unit test. The best database for testing is a small focused data set. Above all the unit test activity should not connect to the production database.

On our project we conduct unit testing on new code that we write, as well as on fixes that we produce. Our main test data set is a small but interesting combination of test cases for verification. In general most of our developers have their own schema for testing. This allows you to do whatever you want without impacting other developers. It also helps in that nobody else makes changes to the data in your database. In the past I have written elaborate unit test scripts which generate delete old test data, generate new test data, execute the tests, and report on the results. This is normally done when I write code that relies heavily on database interaction.

One of the best parts of the blog post I read was the large number of comments from readers. I will share some of them here for your benefit. Multiple people pointed out that the type of testing discussed here is integration testing rather than unit testing. Unit tests are concerned with logic or combination checks. Oracle users recommended the utPLSQL framework for Oracle. You can use the ut_setup and ut_teardown functions to take care of unit test data.

Some readers commented that you could treat the database changes as one big transaction, then roll it back after testing is done. This would definitely not work for my system. We commit transactions all over the place. We even have autonomous transactions which commit themselves without interacting with any outer transactions. You can do better than getting your own schema for testing. You can get a whole database installation of your own to conduct unit tests in. Express versions of Oracle and Microsoft SQL Server are free.

It appears that unit testing is hot in the blogosphere. This seems strange to me. But I am happy about it. Nobody seems to like unit testing where I work. Most developers seem to want to code, check it in, and ship it. I know I myself could use a more rigorous unit test habit.

Scripting

2008-07-25T22:59:00.001-04:00

I read a stimulating blog post by Michiel de Mare entitled “Is Your Program Perfect”. The post was essentially a bunch of questions that made you consider whether there were any improvements you could make to a program to make it better. There was a section on testing that asked whether your program could be scripted from the command line. It also inquired whether your program exposes an API. I thought long and hard about these questions as I considered my projects own suite of applications. And I determined they were very weak in this area. The question now is how do we make changes to the programs to improve in these areas.

In the past I had played with WinRunner by Mercury Interactive. This tool pretty much allowed you to automate testing of any Windows application. The tests could be scripted and create easily by recording and playing back user actions. However this required you to purchase the expensive WinRunner tool. Furthermore this tool is going to be retired in a few years. It would be more economical and beneficial for our applications to provide better test and scripting support itself.

Currently one of our applications has an API that allows you to instantiate at least one of its screens programmatically. However it is a very cumbersome API. You have to create a data on the file system or in shared memory. It also requires clunky windows messages to initiate the actions. This support only exists in one of the five applications in our suite. And this support is for one very specialized part of the app. It would be great if a more comprehensive and simple API could be produce to control the application for test purposes. I envision this to be of great use during the unit test phase.

Obviously it will take a while to add the hooks to our applications. But we could start small and work our way up from there. Perhaps the best place to start would be to automate access to the new feature we are currently coding into the application. We should be very familiar with the design and implementation of the new code. And we already have time in the schedule to work on these pieces. During the unit test phase of these features we could add in the automated support and actually use it for unit testing.

All of this introspection was due to a small piece of Michiel’s blog post to question whether your software is at its best. Opening our apps up to automation may go a long way to simplifying our unit test strategy.

White Space

2008-07-18T17:34:00.000-04:00

Our client has a system acceptance test team. Recently they approached me regarding a problem our load software had last year. They knew it had something to do with the input data containing spaces. And they were trying to determine the impact on the system, and whether they needed to do any additional testing of it this year. I gave this problem some thought and told the test team what I knew.

There is a certain field in one of our input files that is 4 characters in length. We are required to read this field, trim off any white spaces, and store the result in the database. This had worked properly for a number of years. A developer rewrote the load routine using a new tool. They did not correctly follow the requirements and trim the white space from the field. As a result, post load processing could not match the loaded value up with other values that had trimmed. Additionally the application that queries the data could not form the match either.

It took quite a bit to resolve this problem. We had to correct the loaded data for millions of records. Then we had to run the data back through post loads processing. Finally the original load software was corrected to deal properly with future files loaded. It was this last part that the test team wanted to verify. It was good that they are paying attention. Since we only load this particular file once a year, nobody other than the developer tested the fix to the loads code.

The test team was happy to hear the information I provided them. They were immediately running off identifying test cases and planning to generate some input file data to check out our code. We have an internal test team at our company. For some reason I doubt they also have this type of insight and premonition to verify that development follows through on the fixes we produce. Its always easier to deal with problems during test before the software goes out into a production environment.

Access Levels

2008-07-08T06:55:00.000-04:00

Our application suite supports a number of access levels. Each user is assigned one of five access levels. Each level has a uniquely defined behavior in the application. In general, the higher the level, the more you can do. But higher levels do not always have the ability to do everything a lower level user can do.

There are a number of times when people believe the application is not behaving correctly. Frequently this is due to an ignorance of what each access level is allowed to do. Normally this happens with the newer developers or testers. It usually takes a senior developer to sort out the problem and trace it back to an access level issue.

I have been working with a new tester recently. He is testing out the application that I am responsible for. He has found some weird problems in my application. There is a lot of interaction between us since I often need to know exactly how to reproduce a problem, or what specific data the tester is using to validate the application.

The other day I asked the tester to reproduce the problem he had documented. He had to do a lot of work to reset the application access levels to the ones used when he encountered the problem. He told me that he was doing a lot of testing in the application at each of the access levels. This was recommended to him by a senior tester. This encouraged me. I have to admit that, while conducting my own unit tests, I did not test all the functionality at each of the access levels.

This emphasizes the differences between developers and testers conducting verification. A developer may be quick to declare success once one test passes. Or sometimes a developer will assume that some tests will pass based on the results of one other test. A tester may also have this opinion. But they will actually perform the tests to verify the hypothesis. However a developer may be quick to skip steps that seem repetitive and boring.

It is a good thing that we have an internal test team to keep development honest. Their absence would not be a total loss. Our customer has their own system acceptance test team. And since we are migrating to all new development tools this year, the users themselves have scheduled a functionality test of our upgraded application. I have the feeling that I am going to be researching a lot of trouble tickets this year. Since I like fixing bugs, this is a good thing.

Time to Market

2008-03-22T19:52:00.002-04:00

I have a software development business on the side of my day job. Recently I have focused my business on writing small programs that would be of interest to hackers. The programs I develop are posted on my Black of Hat blog. Currently I am giving my programs away for free to try to increase readership of my new blog.

The interesting thing about these programs is that I have not been subjecting them to rigorous testing. Often times the goal is to get some software posted out on my blog quickly. So I run a couple quick unit tests, fix the obvious bugs, and warn my readers that the software may be in an early state.

My goal is to write many useful programs quickly. Of course I plan to improve and fix any bugs that pop up. But there is a definite lack of V&V being done. The question is whether formal test methods would be of any value in this fast paced environment. I don't get any direct penalty for releasing a program that crashes and burns. There may be some unhappy readers which turn away from my blog. However I am considering the faster time to market to be the prime objective.

Perhaps I could take my next program developed by my side business, apply more thorough testing to it, and try to measure the outcome subjectively. I am not against going through a formal test cycle for these programs. The objective is to gain the most amount of readers so that I can monetize my blog. Any way I can do that more effectively is worthwhile. Besides I feel as if I may be developing bad habits in my side business software development. It would be easier if I use the same methods in both my day and night jobs.

Test to Dev to Test

2008-03-08T21:23:00.002-05:00

Over my career, I have met a number of software developers that have become testers. These testers often tell me that they got tired of working the long hours in development. They say that testing, while challenging, is less stressful and easier that writing code. And these are usually the testers that are creating scripts or writing code to test data.

I have also met developers who said they got trapped in the test environment. First it starts out with the need to write a high profile master test plan. Then who better to oversee the test execution than the author. Pretty soon you get known as somebody proficient who can run tests, and then you are pigeon holed into being a tester.

There is one other scenario which is not encouraging. Sometimes developers who just do not cut it as programmers get reassigned to the test team. It is almost as if this is a demotion. This seems strange as the top talent in the test world seem to make good money. Maybe it is just a psychological effect.

The reverse path appears to be difficult if not impossible. I have not met many developers who said they started out as a tester. There was one guy I knew who was pretty good with code. He wrote a lot of scripts to generate test data. But he was still officially assigned to the test team. Maybe there are such developers, but they don't want to talk about their test background. Still it seems a very rare move. It looks like a one way street between development and testing roles.

Top Notch Testers

2008-03-05T18:31:00.002-05:00

Recently we had some late new requirements that needed to get into the system. We rolled them out after our latest major release of the software. There was a hasty test cycle before shipping the software out the door. And guess what? There are a lot of problems with the code in the Production environment.

The good news is now that we are fixing the problem, our internal test team is better equipped and has more time to verify our changes. A recent set of changes has failed internal test a couple times. Seems I have made a number of invalid assumptions about the data in the system.

A lot of the good testing comes from the fact that the tester assigned to my code is actually an old developer. He used to sit next to me in my cubicle on a past project. Somehow he got moved to the test organization. And now he is taking test to a whole new level.

This reminds me of another tester from a couple years ago. Let's call her "SJ". She astounded me and my team when she first started verifying out fixes. SJ has a deep technical understanding about the changes we were making. Sometimes she even dug down and showed me where I had made mistakes in the code. When I asked SJ how she was able to figure this out, she said that she too was a developer in a past life. She wanted something different and was doing testing now.

Not all testers have to be developers at heart. But I have found that it certainly helps.

Dev Test

2008-01-19T01:10:00.000-05:00

I am a software developer by trade. Most projects I have worked on were mainly software maintenance. However I have worked on quite a few new development projects. It seems that I often write more code to do unit tests that I do to implement the software. This is normally for code that is mission critical, where errors can be very costly for everyone involved.

So I have started this blog to research the verification and validation field. Specifically I want to focus on the testing of computer software. Sometimes normal developers consider test professionals to be a step or two below a coder. However I have worked on projects where the quality assurance team were the domain experts and knew what the users wanted the software to do. I am sure those that test at big commercial companies like Microsoft are no slouches.

On a related note, I recently went to an interview for a software position. Got interviewed by a number of people on the project. The most intense interview was with the lead of the test team. This guy was a former developer and the one that had been on the project the longest. I aced a couple of tough testing questions from him. But I got some others that I had trouble working my way through. Perhaps those questions will serve as a starting point for the discussions here:

1. What is a load test?
2. What is a stress test?