Brown CS News

Multiple members of the Brown CS community have returned from the ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW) with two honors for their work. Brown CS alums Sarah Bawabe and Laura Wilson, students Tongyu Zhou and Ezra Marks, and Professor Jeff Huang’s paper (“The UX Factor: Using Comparative Peer Review to Evaluate Designs through User Preferences”) has received an Honorable Mention as well as an Impact Recognition Award.

CSCW is a prominent venue for research in the design and use of technologies that affect groups, organizations, communities, and networks. Bringing together top researchers and practitioners, CSCW explores the technical, social, material, and theoretical challenges of designing technology to support collaborative work and life activities. Both honors were chosen by a separate Awards Committee based on both the original reviews and committee members’ assessment of nominated papers. Impact Recognitions, new this year, are intended to recognize papers that contribute to a potentially significant impact in CSCW research, in practice, in design, in policy, or in the real world in substantive ways.

In their paper, the authors develop UX Factor, a peer evaluation platform that aims to characterize the behavior of peer reviews and the consistency of the ranking models used to aggregate these reviews. This system harnesses the power of pairwise comparisons to minimize bias and encourage context-driven analysis. By adopting UX Factor in a user interface course of 133 students and teaching assistants (TAs) across three different individual design projects over a semester, they found that the system was effective in eliciting high-quality feedback. Among their findings, the researchers note that raters had higher agreement than random preferences, and with at least 15 ratings per submission, a simple average of ratings produced rankings that were consistent to both the raw ratings and other more complex models. These rankings were robust to disagreeable raters and changing class sizes, demonstrating the potential of comparative peer review to match the quality of expert feedback at scale.

UX Factor has also been translated into Japanese for the College Creative Jam run by Adobe in Japan, where it was used as the main submission and feedback platform. [Note that the linked site above is mostly in Japanese.]

For more information, click the link that follows to contact Brown CS Communication and Outreach Specialist Jesse C. Polhemus.