Learning objectives
Last week you wrote tests for enroll, then audited code that an agent wrote. Kathi noticed three things while reviewing everyone's worksheets: (1) many of you are still learning how to identify and test boundary conditions, (2) many of you are better at testing for errors than in thinking of subtle non-error cases, and (3) many of you struggled more with Gemini than intended.
As a result, this week we're going to extend the problem to work on (or around) these issues. We're adding a waitlist feature and a drop function to the enrollment system, and Kathi has already asked an agent to generate the code. You will practice:
- categorizing different kinds of tests
- generating test scenarios in different categories
- describing property-based tests, and
- auditing generated code to propose what it might be doing wrong.
Setup
- Choose a partner. You will discuss everything together, but each of you submits your own worksheet.
- Open the starter file. It has the registrar's tables, and an empty enrollments table you can use to write out the states in your scenarios. You won't write code this week.
- Make a copy of the written answer worksheet, share it with your partner(s), and submit the link to it in the worksheet submission form (you can do this before filling anything in).
The problem, with two new features
Back to the registrar. The catalog and students tables are the same as last week. The enrollments table now has a third column, because courses have waitlists:
| course-id | roster | waitlist |
|---|---|---|
| "cs3" | [list: "cam"] | [list: "dee", "eli"] |
| "phys1" | [list: "dee"] | [list: ] |
Here are the registrar's rules. Read them closely: the differences from last week matter.
enroll(enrollments, student-id, course-id)
- If the student is already on the course's roster or waitlist, raise
"already-in". - If the student has not completed all of the course's prerequisites,
raise
"prereqs-missing". - If the course meets at the same time as a course the student is on
the roster of, raise
"time-conflict". Two meetings that are back-to-back (one ends exactly when the other starts) do not conflict. - Otherwise, if the course has room, add the student to the roster.
- Otherwise (the course is full), add the student to the end of its
waitlist. A full course no longer raises
"course-full".
drop(enrollments, student-id, course-id)
- If the student is on the roster, remove them. If that leaves a free seat, the first student on the waitlist who can still take the course (prerequisites met and no time conflict) moves from the waitlist to the roster. Waitlisted students who cannot take it stay in their position on the waitlist.
- If the student is on the waitlist, remove them from it.
- If the student is on neither, raise
"not-in-course".
Both functions return the new enrollments table.
Part 1Labeling test scenarios
A test plan is more than a pile of scenarios. Knowing why you are writing a scenario tells you what you are still missing. We will use six labels:
| Label | What the scenario does |
|---|---|
| core | A typical request that the problem statement covers and that works in an ordinary way. |
| boundary | A request sitting right at the edge of a number or an ordering in the data or inputs (e.g., just under, exactly at, or just over a capacity, a time, or the first/last position in a list. |
| feature interaction | Two or more features interacting at once; may not be clearly specified in the problem statement. |
| rejected input | The request is not allowed, and the problem statement says what error to raise. |
| violated constraint | The data is bad: the table you start from breaks one of the constraints about valid data looks like. The functions are allowed to assume this never happens, as long as part of our code checks data for validity before the program runs. |
| unhandled | The inputs are valid, but the problem statement said nothing about how to handle the case. Either raise an error or make a decision about how to handle the situation. |
The two easiest to mix up are rejected request and violated constraint. A rejected request is bad input: the table is fine, but what you asked for is not allowed, and the function is responsible for saying so. A violated constraint is bad data: what you started with was already wrong, and the function's behavior is not specified.
Label each scenario below with the single label that fits best. Be ready to defend your answer; some could be argued two ways.
- ana enrolls in cs1 and nobody is enrolled anywhere.
- cs1 holds 2 students. One seat is taken. ben enrolls.
- ben is on the roster of cs1 (Mon 9-10). He enrolls in econ1 (Mon 10-11).
- ben is on the roster of cs1. He drops econ1.
- The roster of cs1 lists three students.
- "zed" (not in the students table) enrolls in cs1.
- cs3 is full. Its waitlist is [list: "dee", "eli"]. cam drops cs3, but dee is now rostered in a course that overlaps cs3.
Our labels (open after you've decided)
1 core. 2 boundary (the last seat). 3 boundary (back-to-back meetings, so the edge of a time range). 4 rejected input (the rules name "not-in-course"). 5 violated constraint (capacity is 2; that is bad data, so nobody should be calling enroll on it). 6 unhandled (the rules never mention unknown students). 7 feature interaction (waitlist and drop and time conflicts all at once).
Open the situations you wrote last week. Label each one. Which labels did you use a lot, and which not at all? In your worksheet, write down any questions you have about the different kinds of tests.
Now write test scenarios for enroll and drop under the expanded problem description, as a list in your worksheet: for each, a description, the expected outcome, and a label. Your list must include:
- at least two boundary scenarios, and a note on what the boundary is;
- at least one feature interaction scenario that involves the waitlist;
- at least one rejected input;
- at least one unhandled scenario, with your decision about what ought to happen.
Hint: For every number or ordering in the rules, ask what happens just under it, exactly at it, and just over it. Where are the numbers in the rules? (There are more than you might think: seats, meeting times, spots in a waitlist.)
Checkpoint
Show a TA your labeled list.
Part 2A validity checker and properties
Step 1Constraints on Validity
Example tests check specific scenarios. Now we want something broader: a way to tell when any enrollment table is valid, and checks that say "calling enroll or drop will always result in a valid enrollment table". Earlier in the semester, we referred to constraints as describing assumptions that our functions want to make about the data. We're getting more rigorous about that now, preparing to use constraints as part of how we frame and audit code.
Here are four initial constraints:
- Only students enroll
- Courses exist only once in the system
- Courses don't over-enroll students
- No student is on the rosters of two courses whose meetings overlap.
Review these as worded for clarity and precision: if we gave you an set of our tables and asked you to check these, would you know what to look for, or are some details missing? Update the descriptions as needed, writing your analysis in your worksheet.
Write down additional constraints that you believe describe when an enrollment table is valid. Give your full set of proposed constraints to Gemini and ask it to evaluate them for clarity, precision, and completeness. Note anything that it reports you having missed.
Did you include a constraint like "the waitlist is non-empty only if the course is full"? Should we require this? Discuss it, reviewing the registrar's rules as needed. Put your analysis in your worksheet.
Checkpoint
Show a TA your constraints and your assessment of their clarify and precision.
Step 2Property-based tests
Constraints say what should be true of the data. Properties say what should be true after running a specific function. Here is a sample property about the enroll function:
- If calling
enroll(table, student, course)succeeeds without raising an error, thenstudentis on the roster or the waitlist ofcourse.
Develop a property about drop. You may find it easier to write a couple of smaller properties rather than one larger property. Put your property/ies in your worksheet.
Checkpoint
Have a TA check that your property is stated well
Part 3Auditing three agent-written versions
Now it's time to put your constraints and properties to work! Kathi used agents to develop multiple implementations of enroll and drop. Each one is a pair of functions written against the registrar's catalog and students tables (the same ones from earlier in the lab). The files contain only the functions, so you won't run them; you will read them, with the Gem's help. The question is whether they work.
Your job is to audit each of these versions: open them up, paste the code into this Gemini Gem (open it once per version), and use your constraints, properties, and test scenarios to determine whether the version works. Specifically, ask Gemini questions, such as "how or where does this code guarantee that only known students are taking courses?" When Gemini answers, look at its response, look at the code that it is highlighting; see whether the response feels right with what you know about programming so far.
In your worksheet, give a brief assessment of each version: do you think it works? If not, how precisely can you describe the error in the code? For example, instead of saying "an unknown student can enroll", describe why, when, or how (e.g., "an unknown student can enroll when the name used for the enrollment is capitalized differently than their name in the students table").
Also include the link to your Gemini transcript for each code version as you finish it.
You are not trying to fix the code. You are only trying to audit it.
Step 1Reflect
Answer in your worksheet: How are you finding the auditing process so far? Are the constraints and properties enough to find and describe the errors? Do you feel you need to read the code to identify the errors? What gives you confidence in your assessment during this process?
Getting help
In lab, flag down a TA. You do not have to be stuck to do it, and you do not have to have tried everything first — that is what they are in the room for. Ask early: a pair that spends forty minutes on a typo has lost the part of the lab that was worth doing.
Your partner counts too. Saying out loud what you think the code does is most of debugging, and it is faster than waiting.
After lab, short questions go on Ed, and anything longer is better brought to office hours.
Comments or concerns?
If you have comments or concerns, submit them on the anonymous feedback form. Sign in with your Brown Google account to open it; your email address is not recorded.