My name as a barcode

How to Write Technical Paper Reviews

First version: 2009-02-21. Small updates to the “review”: 2009-02-28.

You may be a first-time conference PC member or journal reviewer. Or, you've been asked to write a sub-review for a paper by a formal reviewer. What constitutes a good review?

I'll tell you what a review consists of. It's the reviewer's job to be able to write a competent review, doing whatever is necessary—i.e., work backwards.

 
For instance, let's assume the “paper” is Rocky Raccoon.

I like to have three parts to a review:

1. Summarize the paper (1-3 paras). This sounds obvious, but it's critical. It's your way of telling the authors, “No matter what you may think you wrote, this is how it read to me”. That's very useful for the authors to know. Therefore, it's essential that this be in your own words: don't just copy the abstract. Also, don't be surprised to find this part, which seems easiest, is actually the hardest. I routinely find, as I sit down to write the summary, that I can't. It means I haven't really understood the paper. There is unfortunately only one solution: read it again.

 
The paper describes a young boy, Rocky, who loses his woman. Swearing revenge, Rocky checks into a saloon, from where he makes a dramatic entry into an adjacent hoe down. Unfortunately, the rival proves to be a quicker draw than Rocky, resulting in a gunshot injury. Rocky demonstrates courage when a doctor tries to help him, resorting instead to a Bible in his room at the saloon.

Note: the summary is just that, a summary, not an evaluation. Roughly, it means you take claims on face value. But not for long:

2. Critical evaluation (as long as necessary): Here's where you say what you think about the claims. Your response can run the gamut of abstraction from technical to philosophical. It's possible to disagree with technical details but like the overall direction; conversely, it's possible to like the technical work, but disagree with the direction (as being pointless or even wrong). You'd be surprised how often these scenarios arise. I try to stick to “cross-cutting” statements here, unless I'm making a very specific (but critical) technical comment (e.g., the algorithm in Fig 2, which is the central result of the paper, is wrong; this calls into question the entire effort). Remember that sometimes what's important is what isn't in the paper; this can often be far more important than what is.

Try to start by saying positive things, then the negatives. These can be of wildly different lengths, one part being a sentence or two and the other being several pages. It's okay (and expected) for every review to have both parts, because no paper is perfect. (Sometimes a paper is so close to excellent or rotten that it's easy to forget one part, but don't.)

It's sometimes helpful to have a brief “points in favor and against” section, consisting of just bullet points. This summarizes your critical evaluation, and helps others who read the review quickly get to the heart of how you feel (and therefore whether they agree with you or object to your opinion).

 
While the overall narrative structure is simple enough, the account has many unsatisfying elements. We are not given sufficient background about Rocky's history with firearms to determine whether his decision to burst in brandishing a gun was wise. It is also disturbing that Rocky chooses to waive medical advice. While the doctor sent to administer help is clearly incapable of doing so, we are left without enough information about Rocky's actual physical state to determine whether he is right to shrug off all medical help. Naturally, given the situation—competing for his woman—we expect Rocky's self-reporting to contain a great deal of swagger that hides the truth. Finally, the narrative element of the Bible is not described in enough detail to help this reviewer determine whether or not it can help in Rocky's revival. It is also difficult from the sparse description to determine exactly why the outcome was as it was. While the authors deserve praise for laying out all the events in a total order, we are not given enough detail about what happens at each step to be able to reproduce the outcome. Why did Rocky burst in not having already drawn? Did Daniel have prior warning? Was Rocky grinning because he was cocky, or was he expecting help from an accomplice who failed to materialize? Finally, at a higher level, the reviewer finds the account disturbing. Though nobody suffers mortal harm as a result of this incident, it is nevertheless disturbing that violence is considered a reasonable way of settling disputes. It is especially disconcerting that the document does not offer any commentary (much less condemnation) in this regard; indeed, it can be seen as glorifying such “solutions”.

3. Detailed comments: Now you focus on local details as much as necessary. Here it's fine to progress through the paper sequentially.

Keep in mind you're not being paid to proof-read. If you spot important typos, point them out. But you should not waste time pointing to every missing comma, etc. If there are a few, say there are a few and maybe give some examples (esp. if you spot a consistent error). If there are many, it's perfectly okay to complain that the authors should have been less sloppy and unprofessional. In extreme cases, I've asked for papers to be rejected because of presentation so poor I cannot trust the authors will fix it in the final version.

That said, it's important to know at what level to write a review. If the authors clearly don't know how to do research, or what conference to send a paper to, it's probably not worth providing lots of minute, low-level comments when what you need to do is break out the clue stick. At the other end, the paper may be excellent but also have lots of little flaws. If the paper has a very high likelihood of being accepted, then it may be worth a little time pointing out the small flaws, lest they persist.

 
Is “Magil” really a girl's name?
 
Why would a hotel room be located immediately adjacent to the site of a hoe down? Is this a budget hotel?

Now, for the process. I'll tell you what I do; you can use it as a starting point to figure out your own process.

My style of reviewing is to keep a buffer open as I'm reading the paper and make notes as I go along. The notes include questions and concerns (“It's about analyzing routers; I expect the central issue to be modeling dynamics”; or “They say they'll deal with interfaces to foreign functions; make sure they return to this before the end of the paper!”—you'd be surprised by how often people promise one thing up front and deliver something else by the end). Periodically, I will stop and take in the paper (which is when I ask myself questions like, “What is this really about?” and “To have solved the problem they claim, what would they have to have done to convince me?”), which is a good time to take notes. Then I take it all in again when I'm done.

I then try to write the summary, which forces me to re-read parts of the paper. Having finished the summary, now that I have it all in my head, I think hard about what I feel about the paper (the critical evaluation). I might have an opinion immediately, but sometimes I let the paper gel in my head for a day or two, and return to it a few more times (oh, it's about X; but wait, problem X requires addressing Y; did they?—ah, I see they did Z, which is sort of like Y; does this satisfy me?, etc). Then I write the critical evaluation.

At this point, I've taken care of many of the elements in my notes. Some questions have been answered and can disappear. Some notes may actually prove to be warnings: they promised to do X and never did, and if I felt X was important to fulfill the claims of the paper, that becomes a point of major criticism. And so on. I filter out these remarks from my notes.

What's left is essentially the detailed notes. I clean these up into proper prose, and bung them into the review.

It's okay for the review to help the author understand how I read it. For instance, I will sometimes say, “At this point in sec 2.1 I am expecting to find some mention of how you represent the graphs, and I find it distracting that you don't say anything”; if it shows up in sec 4, I will edit this remark to say, “I see you brought it up in sec 4, but that was two whole pages away; I'd have liked to at least get a forward pointer, if not a brief description, in 2.1 itself.” Good authors will appreciate such information. (You eventually learn which papers are written by authors who seem to care and which by ones who don't, and spend time on these kinds of remarks accordingly.)

Unless you intend to leave actors ambiguous, use the active voice, never the passive voice. But that brings up:

Don't get personal. This seems obvious, but it's actually a bit subtle. I find that as I'm writing notes, I often use the phrase “you” (like, “you're promising to ...” or “you should have said ...”). Write notes to yourself however you want, but try never to let this tone remain in your final review. The correct form is “the paper”; even “the authors” is best avoided unless necessary. My philosophy is that you're reviewing a paper, not a person (or people). Papers make mistakes; papers even give the impression of trying to deceive (hopefully accidentally). But we should always give the authors the benefit of doubt and assume they did not make these mistakes. Writing about “the paper” gives them a strong hint (who wrote it, after all?) without outright accusing them of anything.

Finally, if you're a sub-reviewer, unless you've done this before, don't spend too long before showing something to the reviewer. It took me years to learn how to write reviews and to find my voice, so you will probably need some practice and feedback, too. Send drafts so you can get feedback. It's okay to get an education out of the reviewer—after all, they're getting something out of you, too!