Guide to Test Cases

Examples and tests play a role throughout the life of a program.

• People write code to solve problems. Examples and tests help make sure the solution actually does what it needs to do.
• People need to read code after it has been written, to find out or remember what it does. Examples are a form of documentation.
• Programs get modified over time as features are added or analysis needs change. Tests help check that the additions don't break what already worked.

You may think the programs you write in this course are too small for any of this to matter. That is less true on later assignments and projects. More importantly, this course is teaching the practice of responsible programming, and that includes robust habits. If you leave here to help a professor elsewhere at Brown with their research, the accuracy and lifetime of your work could affect the validity of their results. These habits matter.

This guide is about writing individual examples and tests: what one looks like, how to come up with good ones, and what we look for when we grade them. Once you are writing programs big enough that the question becomes whether you have enough tests, the Guide to Testing Plans picks up from there.

Examples and tests

An example or a test has two critical parts: the expression you want to run and the answer you expect it to produce.

Note: Examples and tests are actual code that can be run, not notes in a comment. People ignore most notes in comments, and a comment cannot tell you that the code stopped agreeing with it. For examples and tests to help us build robust programs, they have to run against our code.

Examples (in where blocks)

All functions, including helper functions, require examples. The only exceptions are functions that return an Image and lam functions, which we will meet later in the course.

Examples should illustrate interesting variations, either in the inputs or in how the function behaves. You want some variation here — examples are both documentation and a way to think through the cases your code has to handle — but you don't have to cover everything.

Tests (in check blocks)

Write tests for more complicated functions, and whenever we tell you to put your tests in a separate file. Tests should cover the space of possible inputs fairly thoroughly. You can't test every possible input value, but you can test the different kinds of input that matter to your function. The guidelines in the next section are how you find those kinds.

check:
  double(10) is 20
  double(15) is 15 * 2   # the expected answer can be an expression
end

When to write them

Generally, write examples before you write the code. This forces you to be clear in your own mind about what you are trying to write.

Write tests before, during, and after you write the code. Tests capture more nuance, deeper examples, and finer detail than examples do, and they will occur to you at different moments:

• Before, from the design. These are tests at the level of the problem statement rather than of your code, so you can write them without having decided how the code will work. On a larger program, this is where a testing plan comes from.
• During, as you implement. If you notice a subtlety while writing part of the program, write a test that exercises it — or at least write yourself a note to.
• After, from the code. Once you know the details of the implementation, you can add tests for the fine-grained cases it turned out to have.

Coming up with examples and tests

Not all of these apply to every program. They are the questions to work through, and the ones we use when we look at your tests.

Cover edge cases

Edge cases are inputs at the boundaries of the space of inputs. A function that checks for "numbers between 4 and 8", or for "strings with at least 3 characters", has boundaries.

If the problem has boundaries, use values at each boundary and on either side of it. For a function that looks at values "greater than or equal to 5", have examples using 4.5, 5, and 6 — that is what catches a typo of > for >=.

Think about what the input represents

Consider two pieces of information you might represent with a string: names and passwords. Names may have spaces but probably not symbols like #; passwords rarely have spaces but often have symbols. When you come up with inputs, think about what the information means and what would be meaningful about it.

So for a program that processes passwords, the input might have digits, letters in both cases, and symbols; the symbols could be at the beginning, the middle, or the end; there is likely a minimum length and a practical maximum. All of those scenarios should show up in your examples and tests.

Think about the structure of the data

Some types have structure to their variation. A list can be empty or non-empty. A traffic light is red, yellow, or green. A table can have zero rows, one row, or many. Where your data has structural variation — as opposed to the representational variation of the previous point — have an example for each variation.

This is also the question to ask about your own code's structure: if the program iterates or recurs, do you have inputs that make it do so 0, 1, and many times? If it has a conditional, do you have inputs that take each branch?

Think about relationships within the data

When the data has multiple attributes — a table of records, say — the interesting cases are often about combinations rather than about single values. Have you checked rows that agree on one column and differ on another? If the data is meant to be sorted, have you considered duplicates, or gaps? If two tables are supposed to line up, what happens when they don't?

Think about what a real user would type

What variations might someone actually feed your program? If it takes a date, will everyone write dates the same way? Will someone try a leap year? A different calendar altogether?

Ignore out-of-scope inputs unless we say otherwise

If we tell you the program takes a positive integer, use positive integers in all of your examples and tests. You do not have to test 0, -1, or anything else outside the scope of the problem.

Why not? Error handling is a substantial topic on its own. We touch it only lightly here; you would see much more of it in later CS courses, where you have the tools to deal with such situations properly. For now the goal is programs that are robust on valid inputs.

If we ask for an error, test the error

There is a difference between a program being called on an invalid input and a program that has to report that it cannot produce an answer in some situation. When a problem statement tells you to report an error, have examples and tests that trigger it.

Different shapes of tests

• Give an expected answer for a given call — double(10) is 20. The expected answer can itself be an expression, as long as it is one you can check by eye: double(15) is 15 * 2.
• Check a property of the answer rather than the answer itself, using a function that recognizes it — "the result of doubling is even", or "the table that comes back has the same number of rows as the one that went in". These are useful when the exact answer is tedious to write out, and they are harder to get accidentally right.
• Write the program twice: once in the most obvious way, which is unlikely to be wrong, and once using the extra structure or cleverness that makes it fast. Then test that the two agree on the same inputs.
• For a whole program, a test may prescribe a sequence of actions and describe what should happen — the shape user testing usually takes.

How we grade examples and tests

We use the criteria in this guide: edge cases, variation in what the data represents, variation in the structure of the data, and error cases where the problem statement calls for them.

Sometimes we grade tests for correctness — do they accurately capture the problem statement — and for thoroughness — are they good enough at the criteria above to tell a working solution from a broken one. We will say more about this as it comes up.

The criteria above are about one test at a time. Judging a whole set of tests — is it thorough, does it exercise all of the code — is the subject of the Guide to Testing Plans.