Guide to Testing Plans

This guide assumes you can already write a test case — if not, start with the Guide to Test Cases, which covers what an example or test looks like and how to come up with good ones.

Here the question is different. For a single function, you ask whether a test is right. For a program of any size, you ask whether the tests you have are enough: whether they cover the situations the program will actually meet, and how you would know if they didn't. That question is answered with a testing plan.

Why write a testing plan

The guidelines for coming up with test cases generate ideas faster than you can write test code. The plan is where you record them: a short, high-level list of the situations your tests need to cover, written in English, before any of them exist as code.

Note: A testing plan is not code. It is an overview of what you intend to check. One line of the plan may turn into several tests, and a line you cannot yet write a test for still belongs in the plan — that's a gap you now know about.

Writing the plan first pays off in three ways:

• You find gaps before you invest in code. Rearranging a list is cheap; discovering after you've written twenty tests that none of them touch empty input is not.
• You can share it. A TA, a partner, or a reader can look at your plan and tell you what you've missed, without reading a line of your code. When something goes wrong, a written plan also separates the code is wrong from the plan was wrong — two very different problems.
• You have something to audit against. As you write tests, check them off against the plan. What's left over at the end is your honest account of what is untested.

An example plan

Take a problem in the shape of the ones we work with in this course — the same one the Guide to Planning works through. We have a table of students and a table of advisors, where every student is assigned an advisor based strictly on the semester they are in:

The task: write advisor-for, which takes a student's name and returns that student's advisor. Before writing it, work down the questions from those same guidelines and write what each one turns up:

Three things to notice about that plan. First, it was written without knowing anything about how advisor-for will work — that's what makes it possible to write before the code. Second, several rows say decide: the plan surfaced questions the problem statement never answered, and each one is a decision to make deliberately rather than discover by accident. Third, the last column names the guideline that produced the row, which is how you can tell whether you worked through all of them or stopped after the easy ones.

Planning at the level of the whole program

Now suppose the next part of the problem is a computation that adds an advisor column to the whole student table. That is a different granularity, so it gets its own entries — and they are about the shape of the result rather than about one answer:

• The new table has the same number of rows as the old one, and the same columns plus one.
• Every row's advisor is the one advisor-for gives for that row's student. (A test of one part against another, which is why the two parts are worth planning together.)
• No existing column's values changed.
• An empty student table produces an empty result, not an error.

Notice that these do not name a single expected value anywhere. Properties like "same number of rows" are often easier to state, and harder to get accidentally right, than a hand-computed answer for a six-row table. The Guide to Test Cases says more about tests of this shape.

Kinds of tests

Almost everything in a plan is about behavioral tests: tests of the relationship between inputs and outputs. It helps to name the flavors they come in, because a plan that is short on one of them usually looks complete until you have the names to check it against.

• Expected things happen — the right outputs are produced for valid inputs.
• Expected errors are caught and reported — inputs may be perfectly valid but still call for an error message, as above.
• Unexpected things do not happen — the input is valid, but the system either shouldn't accept it or shouldn't return certain kinds of results on it.
• Invalid inputs are rejected — badly formatted data, missing data, the wrong type of data. This is the flavor that is out of scope here unless a problem asks for it, and that becomes central in a course on building systems for other people to use.

Performance tests

A different question is how much time or space the program uses. Such tests run on larger inputs than behavioral ones, and they are read differently:

• They are run several times on the same data, so that an accidental spike caused by something else happening on the computer isn't mistaken for a property of the program.
• They look at how the time changes as the data grows. With twice as much data, does the program take twice as long? Four times as long? The trend is the answer, not any single measurement.

Operating systems provide ways to track the time and space used to run a program.

Test granularity

The second thing to check a plan against is level. As programs get bigger than a single assignment's worth of functions, we test at more than one level:

• Unit tests check individual functions.
• Integration tests check that several parts work together.
• System tests check the whole program through its interface.

A good set of tests includes all three levels, and at each level you can ask both the behavioral and (if the data are large enough) the performance questions above. Most of what you write in this course is unit tests, with system-level tests appearing on the larger projects.

Judging the plan

Three ways to judge a set of tests, and so the plan behind it:

• Accuracy — do the expected answers reflect the behavior that is actually wanted? If not, the tests are buggy, and you may write buggy code to match them.
• Thoroughness — do the tests cover the range of situations you expect the program to meet? This is what the test-case guidelines are for.
• Code coverage — do the tests exercise all parts of the code?
  • both branches of every conditional
  • loops or recursion that run 0, 1, and many times
  • every function called at least once
  • every error that the code can report

Code coverage is tied to your particular implementation; the first two are about the problem, which is why they can be assessed before the code exists.

Keeping the plan alive

A plan is worth keeping after the code is written. When you find a bug, add the situation that produced it to the plan and then write the test — the plan now records why that test exists. When you change what the program does, the plan is where you notice which tests are now wrong. And when you ask an agent to propose tests, the plan is what you hand it: an agent working from your plan covers your list of situations, while an agent working from your implementation mostly confirms that the code does what it already does.

Testing practices (with and without agents)

• Keep the plan after the code is written, and audit the thoroughness of your tests against it.
• Anytime you find a bug, add a test that would have caught it. These are regression tests, and they are why a bug found once stays fixed.
• Don't trust an agent to design your test suite on its own. Agents, by construction, pick up on the most common scenarios. Good testing is largely about the uncommon ones.
• When you have an agent propose tests, do it at least once with the agent seeing the implementation — to target code and function coverage — and once with it seeing only your testing plan and design documents, so that its tests are not shaped by the code they are supposed to be checking.