inamerrata

One of the curious things about testing software is that as far as increasing your confidence in the code is concerned, it’s better to have a test suite that finds bugs, rather than one that doesn’t.

The problem is that for any piece of software that ever works, you can find an infinite number of test cases that pass, no matter how buggy it actually is. So if those are the only test cases you come up with, your testing methodology hasn’t actually achieved anything you hadn’t already managed to do when you were writing the code in the first place.

It’s possible that you did such a good job writing the code that there aren’t any bugs to be found, or that they’re so obscure that a test suite would have to be amazingly thorough (or lucky) to find them anyway. If so, congratulations! If you can do this reliably — or even if you can merely reliably tell when you’ve done this good a job at coding — there’s no point writing any tests for your code.

For people who don’t have the time or inclination to prove their code correct as they’re writing it, though, testing is still worthwhile. But given testing can’t ever prove the absence of bugs, when should you stop? The first point at which you should stop is when you find a bug. At that point, you should go back to coding, fix the bug, and start testing again; ideally reusing all your old tests because you’ve carefully scripted them.

But “rinse, repeat” algorithms only terminate when there are suds and water involved, and those aren’t terribly useful when we’re trying to write reliable software. Basically, your focus when testing your code is to find some way to break it. If you can’t do that, you’re not trying hard enough. There are plenty of techniques to help you break code: looking for special cases, looking for code paths that are obscure enough the last developer to commit changes might not have thought to check them, looking at edge cases, looking for discontinuities, all sorts of things. Unless you’re really the best programmer on the planet, there’s always some bug you’ll be able to find.

So, when should you stop? First, you shouldn’t stop before you’ve found (and fixed) at least some bugs. Users always pound on a program harder than the developer, so if you don’t find some bugs in testing, you can be certain to be getting reports hours after the first users start playing with it. Once you’ve found (and fixed!) a few bugs, though, you’ll find your test cases are getting more and more numerous, and more and more obscure. Once they start getting to the point where you find it pretty implausible that the cases will come up in real life, continue for another round, then stop.

Then, when a few days later, when you start getting people trying those implausible cases and finding bugs, start again.