Lab 4: Reading, Testing, and Auditing with LLMs

Learning objectives

Today we step back from coding and instead work with LLMs a bit, learning how to read and debug programs. We'll also get some critique on testing plans.

This exercise is our first real exploration with coding with agents. In part 3, we'll ask an agent to help us generate a bit of code that technically you've seen all the pieces for, but that has more parts than you might be comfortable writing on your own right now. We're doing this to explore how useful our testing strategies and constraint statements are for auditing code that we didn't write. We're going to ask you to take good notes about how you are finding this activity in your worksheet; Kathi isn't sure how this will go --- it's part of the experimental aspect of 111E. It's also a skill we hope you will gain by the end of the course.

This lab will:

practice reading smaller list programs
have you write a testing plan, then chat with Gemini to revise and evaluate that plan;
give you practice auditing code you did not write, leveraging your testing plan and constraints

Setup

Choose a partner. You will discuss everything together, but each of you prompts Gemini on your own account and submits your own worksheet.
Make a copy of the written answer worksheet, share it with your partner(s), and submit the link to it in the worksheet submission form (you can do this before filling anything in).

Problem context: enrollment software

A college registrar is deciding whether to let an AI agent write the next version of the course-enrollment software. The office maintains two fixed tables of data: a catalog and information on what courses individual students have taken. Here are sample rows:

catalog (fragment)
id	capacity	credits	prereqs	day	start	stop
"cs1"	2	4	`[list: ]`	"Mon"	9	10
"cs3"	1	4	`[list: "cs1", "cs2"]`	"Wed"	9	10

}

The prereqs column lists the courses a student must have completed before taking the course. Times are hours on a 24-hour clock.

students (fragment)
id	completed
"ana"	`[list: ]`
"cam"	`[list: "cs1", "cs2"]`

}

The registration software is responsible for updating a third table containing the current enrollments. This table looks as follows:

enrollments
course-id	roster
"cs3"	`[list: "cam"]`
"phys1"	`[list: "dee"]`

As students enroll in courses, the roster values for those courses change accordingly.

Part 1Reading code that processes lists

Make a copy of this Pyret file. Starting on line 48, it contains four programs that use a combination of table and list operations. It also has some sample tables that you can use for testing. Each program performs some sort of constraints-like check on the registrar's tables.

List and table operations used here

Operation	Type	What it does
`filter`	`filter(keep :: (A -> Boolean), lst :: List<A>) -> List<A>`	The elements of `lst` for which `keep` returns true, in the same order.
`distinct`	`distinct(lst :: List<A>) -> List<A>`	`lst` with repeated elements removed, so each value appears once.
`member`	`member(lst :: List<A>, elt :: A) -> Boolean`	True when `elt` is in `lst`.
`remove`	`remove(lst :: List<A>, elt :: A) -> List<A>`	`lst` without any copies of `elt`.
`append`	`append(front :: List<A>, back :: List<A>) -> List<A>`	All of the elements of `front`, followed by all of the elements of `back`.
`length`	`lst.length() -> Number`	How many elements are in `lst`.
`all`	`all(check :: (A -> Boolean), lst :: List<A>) -> Boolean`	True when `check` returns true for every element of `lst` (and true for the empty list).
`get-column`	`tbl.get-column(name :: String) -> List`	The values in column `name` of table `tbl`, as a list, top row first.

Task 1

Identify a better name and add a doc string for each function that describes what it does.

An exercise like this is much easier if you work through concrete examples. Sketch out the tables and lists on paper. Show how they change at each step of the computation. Use that, combined with what you know about each operation, to figure out what the function does.

The TAs should see drawings on paper as they come around during this part, unless you are able to read off what these do in an instant.

Edit the file with your answers, then download it. You'll submit this file to Gradescope at the end of lab.

Part 2Develop a robust testing strategy

The main function in our registration system will be

enroll(enrollments, student-id, course-id)

If a student's attempt to enroll works, the function returns the updated enrollments table. If there is a problem, it raises an error such as "course full" or "prereqs missing"

Here is the registrar's stated rule for enrolling in courses:

A student can enroll in a course if they have met its prerequisites, there
is room, and the course doesn't meet at the same time as a course they're
already enrolled in. Enrolling adds the student to the course's roster.
Otherwise the enrollment is rejected: in that case, raise an error whose message is the
reason that applies, "already-in", "prereqs-missing",
"course-full", or "time-conflict".

Task 2

In your worksheet, list the situations you would want to check and what the outcome should be (e.g., "a student who enrolls in a course with seats gets added to the roster"). Aim for about 8 situations covering a variety of cases.

Task 3

If you and your partner developed your lists of situations independently, take a moment and compare notes. Are you thinking of the same situations?

Task 4

Start a session with this Gemini GEM. Paste in the situations you wrote in your worksheet. The GEM already knows the stated rule and the structure of the tables. Gemini will give you feedback on scenarios you might be missing or places where your descriptions could be more precise. Iterate with Gemini until it has no significant corrections to your scenarios. Paste the revised list back into your worksheet (leaving both your original and the revised one).

Before you move on to the next step, save your chat transcript as a google doc (shared with Kathi) and put the URL at the top of your worksheet.

Checkpoint

The TAs will lead a whiteboard discussion once most students have gotten to this point.

Task 5

What are you learning from this interaction with Gemini? Are there certain kinds of information that you missed originally? Imagine you were making notes to help you the next time you do a problem like this. What would you want to remind yourself about? Write your answers in your worksheet.

Task 6

Make a copy of the starter file. Ask Gemini to turn your test situations from Task 4 into one or more check blocks. Paste the result into the tests area of your starter file. Make sure your file runs (the tests will all fail because at the moment the enroll function just raises an error)

Checkpoint

Show your tests to a TA.

Part 3Auditing the agent's attempt

Now, we will ask Gemini to generate code for enroll based on the same description.

Task 7

Open this GEM and give Gemini the problem description from the top of Part 2 (the green box). Do NOT give Gemini your tests. Ask Gemini to produce the enroll function (we have given the GEM the table structure behind the scenes).

Skim the code you got back. Does it look as you might expect, with the sort of constructs that we've been using in class? If there are significant things that don't look familiar or that look wrong, consider whether you needed to add more information to your prompt to Gemini. Give Gemini more info if needed and try again. Repeat until you get a program with familiar elements at the high level.

Task 8

In your starter file, replace the placeholder enroll (between the two marked lines) with the code Gemini wrote. Paste only the function definitions: leave out any use context, include, or provide lines. Try to run the code. If it does not compile, see if you can fix the errors. Note in your worksheet what you changed.

You're welcome to report problems back to Gemini for fixing if you don't see what the issues are yourself. Our goal is simply to get a program that loads when you press the Run button (your tests do not need to pass at this point). If you are unable to get the code to run with a couple of small fixes, use the code from the dropdown below instead (which will let you continue with lab).

A pre-generated enroll function (if you need it)

# enroll(enrollments, student-id, course-id) -> Table
#
# Returns the updated enrollments table with student-id added to the roster of
# course-id. Raises an error whose message is the reason when the student
# cannot be enrolled.

fun enroll(enrollments, student-id, course-id):
  course = catalog.filter(lam(r): r["id"] == course-id end).row-n(0)
  student = students.filter(lam(r): r["id"] == student-id end).row-n(0)
  entry = enrollments.filter(lam(r): r["course-id"] == course-id end).row-n(0)
  roster = entry["roster"]

  # courses this student is currently enrolled in
  current-ids = enrollments.filter(lam(r): r["roster"].member(student-id) end)
    .get-column("course-id")
  current-courses = current-ids.map(
    lam(id): catalog.filter(lam(r): r["id"] == id end).row-n(0) end)

  # time conflicts: same day and the meetings touch or overlap
  conflicts = current-courses.filter(lam(other):
      (other["day"] == course["day"])
        and (other["start"] <= course["stop"])
        and (course["start"] <= other["stop"])
    end)

  # prerequisites: a prerequisite counts if the student has completed it or
  # is currently taking it
  met = course["prereqs"].filter(lam(p):
      student["completed"].member(p) or current-ids.member(p)
    end)

  if roster.member(student-id):
    raise("already-in")
  else if roster.length() >= course["capacity"]:
    raise("course-full")
  else if conflicts.length() > 0:
    raise("time-conflict")
  else if (course["prereqs"].length() > 0) and (met.length() == 0):
    raise("prereqs-missing")
  else:
    enrollments.filter(lam(r): not(r["course-id"] == course-id) end)
      .add-row(enrollments.row(course-id, link(student-id, roster)))
  end
end

Task 9

Now let's get your tests running. Focus on one test or check block at a time. Our next goal is to have a conversation with Gemini about the situation being tested, so we can figure out what the code is or isn't doing. The point isn't for you to read all the code. Instead, you want to get Gemini to explain how the computation works on the given input so you can look at what happens on the specific example in your test.

For example, you might ask "Does this code guarantee that a student can't be enrolled in the same course twice? Where does that happen?". Gemini should tell you which parts of the code are involved in this. Its description will read like a mini plan (e.g., "we extract the roster, then we check for membership"). Ask yourself whether the plan makes sense. If it doesn't, ask another question like "what if the student id was in all caps instead? Would it still work?". This is where you use your knowledge of the data and the constraints to interrogate Gemini about the code.

Through this process, Gemini may edit the code (that's fine), or ask you to clarify something about the data (also fine). Work with Gemini until (a) the test you started with passes, (b) the results look right when you experiment with the function with various inputs, and (c) you are confident that the plan Gemini is following in the code makes sense for that situation.

Checkpoint

Check in with the TAs at some point, to show them that you are making progess and aren't too stuck on what you should be doing.

Task 10

In your worksheet, document what you are experiencing while doing this. Are you unsure about how to turn a testing strategy item into a good question for Gemini? Relieved that Gemini can help you navigate the test? Frustrated by something? Excited that this works? This information is a big part of the experiment that we're running with this lab (and it will inform future assignments), so take a couple of minutes to fill this in carefully.

Keep repeating those two tasks to work through more errors in your program.

Checkpoint

At some point, the TAs will stop and go over some of this with a whiteboard, so you can talk through how to do this.

Task 11

If you get all of your tests working, try working on the constraints. Come up with constraints on the input table. Ask Gemini to explain to you how it maintains them when enrolling students. You could also work with Gemini to generate a checker for the constraints.

Task 12

Once you're down to 10 minutes left in lab, stop what you're working on and fill in the last reflection section in the worksheet. Kathi is curious about how you found this exercise overall, and whether this process is leaving you more confident or less confident in what Gemini produced.

Task 13

To hand in your work:

Paste your Gemini chat transcripts into the top of your worksheet: the conversation with the first GEM about your test situations, and every conversation with the second GEM about the enroll code.
Download your two Pyret files, the reading file and the starter file, and submit both to Gradescope.

Getting help

In lab, flag down a TA. You do not have to be stuck to do it, and you do not have to have tried everything first — that is what they are in the room for. Ask early: a pair that spends forty minutes on a typo has lost the part of the lab that was worth doing.

Your partner counts too. Saying out loud what you think the code does is most of debugging, and it is faster than waiting.

After lab, short questions go on Ed, and anything longer is better brought to office hours.

Comments or concerns?

If you have comments or concerns, submit them on the anonymous feedback form. Sign in with your Brown Google account to open it; your email address is not recorded.