This guide assumes you can already write a test case. If not, start with the Guide to Test Cases, which covers what an example or test looks like and how to come up with good ones.
When we are talking about a single test case, we care about whether it is accurate relative to the desired behavior. When we are writing a collection of tests, we care about whether they (together) are sufficient to cover the scenarios that we care about. We could gauge the quality of a test suite by reading all of the tests. It is more effective, however, to have a prose-level description of what needs to be tested, to check that that description is accurate, then to check whether the tests satisfy and exhaust that description. That description is called a testing plan.
What does a testing plan look like?
A testing plan is meant to be readable by humans. We therefore want to break it down into meaningful sections that align with how a human might understand or audit your work.
We have discussed different kinds of tests: boundary, core, validity/constraints, unexpected inputs, and feature interactions. While we could organize a testing plan around these categories, humans are more likely to think though a system in terms of its features or key tasks.
Consider a gradebook-management program. It might have features for updating a single grade, for importing grades for the entire class on a single assignment, for looking up a single student's grade on an assignment, and for generating a summary of students' averages across assignments. For each of these features, we might have multiple kinds of tests. Conceptually, we might organize our tests as follows:
Testing Validity Constraints:
- Make sure grades are greater than or equal to 0
- Make sure no grades are empty; every cell should contain a number (float) or None
Single-Feature Tests:
| Feature | Boundary | Core | Unexpected Input | Maintain Constraints |
|---|---|---|---|---|
| Update Grade | ... | ... | ... | ... |
| Import Grades | ... | ... | ... | ... |
| Lookup Grades | ... | ... | Looking up grade on assignment that isn't in the gradebook yields an error | ... |
| Summarize Averages | ... | ... | ... | ... |
Feature-Interactions:
- If a grade is updated for a student, looking up the grade for that student retrieves the updated grade
- If grades are imported for a new assignment, looking up a student's grade on that assignment retrieves the imported grade
- Updating the individual grade for a student results in an update to the average for that student
- Uploading grades for a new assignment results in updates to the averages for all students
As we can see from the single test we put in the table, the table format can get cramped; we showed it here because the table is a good way to think about this conceptually. As a result, it makes more sense for your testing plan to represent each row of the table separately, perhaps as follows:
Feature: Lookup Grade
Boundary: ...
Core: ...
Unexpected Input:
- Looking up grade on assignment that isn't in the gradebook yields an error
Constraint Maintenance: ...
Isn't this duplicating the constraints?
We have separately talked about maintaining a list of constraints on our data; here, we seem to be duplicating those constraints in the first part of the testing plan. It would certainly be reasonable to have constraints maintained in a separate document. We listed them here simply to reinforce that constraints, too, must be tested. If you have a separate constraints document, you can omit the constraints section here, unless there are subtleties about the constraints that you want to make sure get tested.
You should still include tests on each feature that constraints are maintained as functions update your data.
Assessing a Testing Plan
The quality of a testing plan lies in how well it covers the various features and behaviors of a program, as well as how well it anticipates the data that might be provided (some invalid, some valid but unusual, some usual, etc).
Both other humans and agents can be helpful in checking over testing plans.
- Humans are likely to understand the usage context of a program in ways that agents might miss; humans can also gauge whether the statements in the testing plan make sense to other humans.
- Agents are good at proposing potential gaps in your testing plan, or in identifying statements that aren't clear. Don't blindly trust the agent, however. Agents can sometimes get hung up on irrelevant details, propose data that can't arise in your context, or assume features of your program that don't exist. Ask an agent for ideas for improving your plan, but then think through its suggestions. You, the human, should remain in control of the testing plan and what goes in it.
Other Ways to use Agents
You can start a fresh session with an agent (fresh so that it isn't biased by your ideas), ask it for a testing plan for a problem, then compare your plan and the one the agent came up with. This can be a good way to find gaps either in your plan or in how you described its requirements. You can ask the agent to produce a comparison of the two plans so you can review the differences and decide what is relevant for your problem.
You could also give a fresh agent session the problem statement, ask it to generate individual test cases, then ask it to identify how its test align with the testing plan. You might uncover a situation that isn't covered by your plan, but that an agent identified based on solutions to similar problems in its training data.
From Testing Plans to Specific Tests
Once you have a testing plan that you are confident in, you can think about specific test cases that would exercise each statement or you can ask an agent to help you with that. Agents can do a good job creating tests from a testing plan. If you do ask an agent to develop tests from a plan, ask the agent to comment or label each test (or block of tests) with the part of the testing plan that it supports.
Be warned that agents sometimes generate trivial or meaningless tests, such as testing whether an existing library in the programming language is working (that's not part of what you're writing, so it doesn't belong in your testing plan). Always review what agents suggest about tests and testing.
Keeping the plan up-to-date
Testing plans should be kept up to date as you correct and extend a program. Sometimes, a bug arises because the testing plan failed to describe how part of a feature should behave. Sometimes, the plan covered an issue, but the actual tests missed an important example of input for testing that issue. Anytime you find an error in your program, update the testing plan and the tests as needed so they would catch the bug you just found. This is called regression testing; it's a key practice in real software development.