Measuring the Effectiveness of Error Messages Designed for Novice Programmers

Guillaume Marceau, Kathi Fisler, Shriram Krishnamurthi

ACM Technical Symposium on Computer Science Education, 2011

(Best Paper.)


Good error messages are critical for novice programmers. Recognizing this, the DrRacket programming environment provides a series of pedagogically-inspired language subsets with error messages customized to each subset. We apply human-factors research methods to explore the effectiveness of these messages. Unlike existing work in this area, we study messages at a finegrained level by analyzing the edits students make in response to various classes of errors. We present a rubric (which is not language specific) to evaluate student responses, apply it to a course-worth of student lab work, and describe what we have learned about using the rubric effectively. We also discuss some concrete observations on the effectiveness of these messages.


Compared to our Scheme 2010 paper, this one describes a narrower set of experiments, but a whole course of data in section 5. To read about a broader set of experiments, see the other paper.



These papers may differ in formatting from the versions that appear in print. They are made available only to support the rapid dissemination of results; the printed versions, not these, should be considered definitive. The copyrights belong to their respective owners.