On this page:
8.1 Reviewing Examples
8.2 The “Sweep”
8.3 Timeline
8.4 Partition of Labor
8.5 Copying the Work of Others
8.6 Being Professional

8 Peer Review

In industry, it is common to peer review code: that is, you present your code to other programmers, who critique it on design, correctness, structure, efficiency, and so on. In some organizations, peer-review is required before any code can be formally committed to the code base. These principles aren’t limited to industry: the benefits of peer-review are an argument for open source software, and for demands by security experts to make sources public. Finally, just about every programmer has had a friend come over and look at their code and provide feedback.

8.1 Reviewing Examples

“Code”, in the sense of the executable part of the program, isn’t the only artifact available to review. Many other parts, ranging from design documents to data files, are also worthy of inspection. In this class, we will emphasize the review of another important part of programs: its examples (test cases for the interfaces defined in the problem statement—as opposed to those that exercise implementation details).

There are many advantages to reviewing examples:
  • These examples can’t refer to implementation details, which may vary widely.

  • While people can reasonably disagree about implementation details, there ought to be much less difference of opinion about the expected behavior of the interfaces defined in the problem.

  • They are concrete. Therefore, they are often more accessible and easier to grasp.

  • They can usually be understood in isolation from each other. You can understand some without understanding them all, whereas different parts of source code are usually more tightly interlinked.

  • They might point to weaknesses in your own understanding of a problem!

This last point is especially important. The more effort you invest into reviewing, the more you might learn about the problem, which in turn might help you improve your own performance on the same assignment. This is very much part of our goal in asking you to perform reviews: you help others while helping yourself.

As a reviewer, you will typically be given three submissions to review. We have found this number to strike a good balance between effort and exposure to a diversity of thinking about the problem.

8.2 The “Sweep”

Over time, especially when one assignment builds on another, the set of examples can grow quite large. Reviewing these becomes onerous to other students. They might also miss problems that you would have liked to hear about.

We have successfully tested a simple alternative. Instead of submitting an entire test suite, we will let you submit only a sweep (a term inspired by archeology and geology). The sweep is a set of 5 to at most 10 examples that are meant to be representative of the problem. That is, the sweep asks you to put your “best foot forward” rather than just dumping every example you can think of and forcing your reviewers to sort them out. Be pithy, not prolix.

Naturally, when limited to a sweep, you may no longer be able to cover the entire space of inputs and outputs. You should instead focus on a handful of standard ones and the ones that illustrate interesting corner cases. Sometimes it helps to add a comment to explain why you think a particular example is interesting (or present in your sweep).

8.3 Timeline

Performing peer-review after an assignment has been submitted is a little like performing peer-review after a system has shipped: not only useless, but also frustrating when you do find a mistake or a way to make things better. Instead, therefore, we’re asking you to perform peer-review while an assignment is in progress.

To enable this, we want everyone to submit their examples within 48 hours of the assignment being published. Submissions will be assigned to reviewers as quickly as possible. Reviewers, in turn, must submit their reviews ideally within 24 hours. This way, both the reviewers and the authors will receive material with enough time to actually benefit from reading it.

8.4 Partition of Labor

Initially, we will have everyone serve as a reviewer and also receive reviews, to get you acquainted with the process.

After the first few assignments, to spread out the effort, on each problem we will divide the class into three groups:
  1. Reviewers, who will write reviews (but not receive any).

  2. Reviewees, who will receive reviews (but not write any).

  3. Others, who will neither receive nor write reviews.

Group assignment will be random, so over the course of the semester you will get to be in each of the different categories.

You will find out which group you’ve been assigned to after you submit your examples for review.

8.5 Copying the Work of Others

Once you’ve seen someone else’s excellent test case, you may find it hard to resist copying it. That’s okay—you’re welcome to! Just attribute it to peer-review. Also, if the submission reveals more than tests—in particular, solutions—ignore that part and alert us to it.

8.6 Being Professional

Be professional in your reviewing. Do we need to say more? Good.