You are currently looking at an older section of the wincent.dev website.
Please check the new version of the site at https://wincent.dev/ for updated content.

wincent knowledge base

« Lightweight issue tracking | Main | Avoiding protocol-related warnings »

February 13, 2006

Unit testing guidelines

Lately I've been spending a lot of time working on my unit testing framework, WOTest. In doing so I've had cause to think about unit testing "best practice", or at least what works best for me. In this article I summarize the guidelines that I've come up with that help me decide when, how and where to write unit tests:

  1. Write unit tests at all levels of your implementation
  2. Test your assumptions
  3. Base your unit tests on your documentation
  4. Base your unit tests on your code
  5. Base your unit tests on your expected results
  6. Write unit tests for your bugs

1. Write unit tests at all levels of your implementation

If you have a low-level method for adding numbers which is used by a high-level method for calculating window geometry then you should write tests for the low-level method and tests for the high-level method. In this way it will be easier to isolate faults. If a flaw in the low-level method causes the high-level method to fail you'll see failures in the tests for both methods. If the low-level method passes all the tests but the high-level method is broken you'll know where to direct your attention (to the high-level method).

In practice this means writing tests for every method of every class. It may sound like a lot of work but you'll appreciate the thoroughness of your tests if you later have to find a bug in a huge framework. Don't skip writing tests for simple methods that don't seem to need it; even the simplest-looking code can have bugs lurking in it.

One side consequence of the decision to write units tests for every method of every class is that it makes sense to group your unit tests using the same class/method hierarchy: that is, all tests for a given class will be grouped together into a test class, and all tests for a given method will be grouped into a method in that test class. You can of course add other, higher-level tests to the test class: these higher-level tests don't just test that a single method works as expected; rather they test more complex interactions between methods, classes (and sometimes mocks). I tend not to worry too much about keeping my unit tests totally isolated from one another (ideally you would test each class in total isolation from others, using mock objects as stand-ins); in practice keeping the tests "fairly isolated" is enough.

The highest level tests of all are sometimes called "acceptance tests". This basically means that you test that the program does what the user expects it to. Unit tests are lower-level, "atomic" tests that test the components (units) of the program. Acceptance tests are used to check the behavior of the program at the level at which the user interacts with it. They're called "acceptance" tests because they indicate whether the program will be acceptable to the user or not; that is, if it does what the user wants it to do. Unit tests are for programmers; acceptance tests are for users. In practice you write both kinds of tests using the same kinds of tools.

2. Test your assumptions

Sometimes you write code that takes advantage of some undocumented aspect of the Cocoa frameworks or the Objective-C runtime. I'm not talking here about using private or undocumented APIs; I'm referring to those places where the official documentation sometimes leaves things unsaid and taken for granted.

In those cases you end up making assumptions about the operation and characteristics of the context in which your program runs: assumptions about the operating system, the APIs, the environment.

Don't let these assumptions go untested. Write unit tests whenever your code rests upon an assumption you've made about something that isn't affirmed with 100% clarity in the official documentation. Use unit tests to eliminate potential ambiguity or uncertainty about the context in which your code is running.

3. Base your unit tests on your documentation

For me writing a working program consists of three simultaneous activities:

  1. Writing code-level documentation
  2. Writing unit tests
  3. Writing code

Note that writing code is only one of the three parts of making a working program. You write the code so that the machine has a set of instructions to follow. You write the unit tests to ensure that the code works as you think it does and to give you added confidence when it comes to working quickly and making big changes. You write the documentation — even for closed-source code that nobody else will ever see — for two reasons: firstly, because it will be helpful to you if you ever have to revisit that same code months or years later; secondly, because it forces you to think about coming up with better designs that are more programmer-friendly (if you have trouble writing documentation for something it is a sure sign that you could have implemented things in a better, simpler way).

So you write unit tests based on that documentation. For example, if your documentation says, "raises an exception when passed nil", you write a test that confirms that an exception is raised when you pass nil. If you documentation says, "returns NO if the receiver does not recognize the passed selector", you write a test that passes an unrecognized selector and checks to make sure that NO is returned. These unit tests confirm that your code conforms to the documentation.

4. Base your unit tests on your code

You'll then want to go though you code and look for every instance where you make an assertion, where an error can occur, or an exception can be thrown. You write unit tests for each of these places where the code can go off the rails. Basically you want to make sure that it goes off the rails when you expect it to. Use tests to ensure that assertions are correctly raised, that errors are reported, and that exceptions are thrown when appropriate.

5. Base your unit tests on your expected results

This is probably the most obvious basis for writing unit tests. You write tests that confirm that your methods return the expected results. Let's say you have a method that adds two numbers. You'll want to write tests that confirm that your method correctly returns that "2 plus 2 equals 4". There are three types of test to write here:

  1. Success conditions
  2. Failure conditions
  3. Boundary conditions

In the first type of test you expect success. You check that "2 plus 2" does indeed equal "4".

In the second type of test you intentionally provoke a failure. For example, you provoke an overflow and check that the result is invalid.

In the third type of test you pick the borderline between success and failure — that is, conditions that approach as close as possible to "failure" without actually getting there — and make sure that the actual behavior is as expected. For example, in code that loops through items in an array you will want to ensure that the code works properly with empty arrays (0 items) or single-item arrays. With code that works with strings you'll want to check that nil strings and empty strings are handled. With code that operates with buffers you'll want to check that things work correctly when the buffer is one-byte away from full, totally full, or not big enough (the failure case).

What this generally means is that I almost never write a WO_TEST_TRUE test without pairing it with an inverse WO_TEST_FALSE case. Basically the pair of tests says, "I expect this to work but this to fail". Just because a method works when I expect it to work doesn't mean that it's correct; it must also not work when I expect it to not work. And by testing for boundary conditions as well as straightforward success and failure cases I can have more confidence that the method will work in all cases where I believe it should work and fail in all cases where I believe it should fail.

6. Write unit tests for your bugs

Each time you find a bug you should write a test that exposes it. In other words, you write a test that fails because of the bug, but which would pass if the bug didn't exist. You then fix the bug and the test passes. If the bug ever comes back your test will catch it straight away.

Posted by wincent at February 13, 2006 02:38 AM