Examining “Reproducibility in Computer Science”
1 What We Are Doing
2 Progress
3 How to Review
4 Purported Not Building; Disputed; Not Checked (14)
5 Purported Building; Disputed; Not Checked (6)
6 Conflicting Checks! (0)
7 Misclassified (3)
8 Purported Not Building But Found Building (34)
9 Purported Building But Found Not Building (0)
10 Purported Not Building; Confirmed (2)
11 Purported Building; Confirmed (1)
12 All Others Purported Not Building (63)
13 All Other Purported Building (108)
14 Threats to Validity

Examining “Reproducibility in Computer Science”

1 What We Are Doing

Welcome to repo-repe-repro: the repository to repeat an experiment in “reproducibility”!
A group led by Christian Collberg attempted to evaluate the buildability of artifacts from research papers. Our goal is to allow the community to review and reconstruct their findings. Note: We are not the original authors! If you have questions about the original study, please contact them, not us!
We are grateful to Collberg, et al. for initiating this discussion and making all their data available. This is a valuable service based on an enormous amount of manual labor. Even if we end up disagreeing with their findings, we remain deeply appreciative of their service to the community by highlighting these important issues.
We do disagree with Collberg, et al.’s use of the term “reproducibility”. Many people, including ourselves, associate it with an independent reconstruction of a work. This paper, for instance, spells out the difference between repeatability and reproducibility and provides interesting examples.

2 Progress

Purported Not Building; Disputed; Not Checked

 

6%

 

••••••

Purported Building; Disputed; Not Checked

 

2%

 

••

Conflicting Checks!

 

0%

 

Misclassified

 

1%

 

Purported Not Building But Found Building

 

14%

 

••••••••••••••

Purported Building But Found Not Building

 

0%

 

Purported Not Building; Confirmed

 

0%

 

Purported Building; Confirmed

 

0%

 

All Others Purported Not Building

 

27%

 

•••••••••••••••••••••••••••

All Other Purported Building

 

46%

 

••••••••••••••••••••••••••••••••••••••••••••••

3 How to Review

To participate, fork this repository on GitHub:
You will need to do this so that you can edit or add files to your copy of the repository and then share them with us. (Although you can clone the repository elsewhere, pull requests are easiest from GitHub; please help us keep our workload reasonable.)
  • To dispute a Collberg, et al. review—open to anyone—find the directory corresponding to the paper and create a file named dispute.txt. In it, explain why you are disputing the review.

  • To review an artifact—open to anyone without a COI with the paper—make sure you understand what Collberg, et al. did (see section 3 of their paper). In addition, feel free to also run the system, but remember that that is not their primary metric. Using the format given below, write your review in files as follows (if there’s already one, append, don’t replace):

    • If you believe the artifact passes, write your findings in cleared.txt.

    • If it fails, write your findings in problem.txt.

    • If the artifact has been misclassified by Collberg, et al. and should not even be part of the study, write your findings in misclass.txt. For instance: the paper is actually theoretical; the paper offers no code download at all; etc.

We need a standard for judgment. We believe a useful standard is PHOSITA. That is, when an artifact doesn’t build or run out of the box, you should make only reasonable (and minimal) changes. If with these it can pass muster, consider clearing it (but indicate precisely what you had to do). If with these it does not, reject it. In particular, even if you were successful, if you had to use ingenuity, then the artifact should be regarded as failing. If in doubt, feel free to discuss on social media.
When you are done, commit and send a pull request.
Please enter your review using the following format:

Time [in minutes]:

Platform [OS, libraries, etc.]:

Skill level (at least the following; tell us if you have a paper-specific skill):

- make, but that's about it

- I can work around issues like failed dependencies and unset environment variables

- I can build complex software like GCC and the Linux kernel

Sequence of steps to build:

SHA-1 of the downloaded files, dates of download, and URLs:

Detailed evaluation (including comments about running it):

4 Purported Not Building; Disputed; Not Checked (14)

The original study claimed that the paper did not build; someone has disputed this claim, but nobody has yet re-examined it.

CCS'12

 

Gilles Barthe, David Pointcheval, Santiago Zanella Beguelin

 

Verified security of redundancy-free encryption from Rabin and RSA

 

notes

 

dispute!


SOSP'11

 

Joshua B. Leners, Hao Wu, Wei-Lun Hung, Marcos Kawazoe Aguilera, Michael Walfish

 

Detecting failures in distributed systems with the Falcon spy network

 

notes

 

dispute!


PLDI'12

 

Daniel Perelman, Sumit Gulwani, Thomas Ball, Dan Grossman

 

Type-directed completion of partial expressions

 

notes

 

dispute!


OOPSLA'12

 

Vivek Kumar, Daniel Frampton, Stephen M. Blackburn, David Grove, Olivier Tardieu

 

Work-stealing without the baggage

 

notes

 

dispute!


OOPSLA'12

 

Robert Dyer, Hoan Nguyen, Hridesh Rajan, Tien Nguyen

 

Analyzing ultra-large-scale code corpus with boa

 

notes

 

dispute!


CCS'12

 

Richard Wartell, Vishwath Mohan, Kevin W. Hamlen, Zhiqiang Lin

 

Binary stirring: self-randomizing instruction addresses of legacy x86 binary code

 

notes

 

dispute!


PLDI'12

 

Danfeng Zhang, Aslan Askarov, Andrew C. Myers

 

Language-based control and mitigation of timing channels

 

notes

 

dispute!


VLDB'12

 

Changbin Liu, Lu Ren, Boon Thau Loo, Yun Mao, Prithwish Basu

 

Cologne: A Declarative Distributed Constraint Optimization Platform

 

notes

 

dispute!


OOPSLA'12

 

Munawar Hafiz, Paul Adamczyk

 

The nature of order: from security patterns to a pattern language

 

notes

 

dispute!


OOPSLA'12

 

Adam Betts, Nathan Chong, Alastair F. Donaldson, Shaz Qadeer, Paul Thomson

 

GPUVerify: a verifier for GPU kernels

 

notes

 

dispute!


VLDB'12

 

Yucheng Low, Joseph Gonzalez, Aapo Kyrola, Danny Bickson, Joseph M. Hellerstein

 

Distributed GraphLab: A Framework for Machine Learning in the Cloud

 

notes

 

dispute!


PLDI'12

 

Luke Cartey, Rune Lyngso, Oege de Moor

 

Synthesising graphics card programs from DSLs

 

notes

 

dispute!


PLDI'12

 

Michael Pradel, Thomas R. Gross

 

Fully automatic and precise detection of thread safety violations

 

notes

 

dispute!


OOPSLA'12

 

Zhongxian Gu, Earl T. Barr, Drew Schleck, Zhendong Su

 

Reusing debugging knowledge via trace-based bug search

 

notes

 

dispute!


5 Purported Building; Disputed; Not Checked (6)

The original study claimed that the paper did build; someone has disputed this claim, but nobody has yet re-examined it.

PLDI'12

 

Susmit Sarkar, Kayvan Memarian, Scott Owens, Mark Batty, Peter Sewell, Luc Maranget, Jade Alglave, Derek Williams

 

Synchronising C/C++ and POWER

 

notes

 

dispute!


VLDB'12

 

Jeffrey Jestes, Jeff M. Phillips, Mingwang Tang, Feifei Li

 

Ranking Large Temporal Data

 

notes

 

dispute!


OOPSLA'12

 

Zachary R. Anderson

 

Efficiently combining parallel software using fine-grained,language-level, hierarchical resource management policies

 

notes

 

dispute!


OOPSLA'12

 

Alon Mishne, Sharon Shoham, Eran Yahav

 

Typestate-based semantic code search over partial programs

 

notes

 

dispute!


OOPSLA'12

 

Guilherme Ottoni, Haiping Zhao, Iain Proctor, Minghui Yang, Xin Qi, Mark Williams, Qi Gao, Andrew Paroski, Scott MacVicar, Jason Evans, Stephen Tu

 

The HipHop compiler for PHP

 

notes

 

dispute!


VLDB'12

 

Andrew Pavlo, Evan P. C. Jones, Stanley B. Zdonik

 

On Predictive Modeling for Optimizing Transaction Execution in Parallel OLTP Systems

 

notes

 

dispute!


6 Conflicting Checks! (0)

The re-examination has produced conflicting findings.
Nothing here!

7 Misclassified (3)

On re-examination, this paper should not have been included in the original study at all.

CCS'12

 

Muhammad Jamshed, Jihyung Lee, Sangwoo Moon, Insu Yun, Deokjin Kim, Sungryoul Lee, Yung Yi, KyoungSoo Park

 

Kargus: a highly-scalable software-based intrusion detection system

 

notes

 




misclassified

PLDI'12

 

Boris Petrov, Martin T. Vechev, Manu Sridharan, Julian Dolby

 

Race detection for web applications

 

notes

 




misclassified

OOPSLA'12

 

Tomas Kalibera, Matthew Mole, Richard Jones, Jan Vitek

 

A black-box approach to understanding concurrency in DaCapo

 

notes

 

dispute!


misclassified

8 Purported Not Building But Found Building (34)

The original study claimed that the paper did not build, but the re-examination has found it does build.

TOCS'30

 

Michael Ferdman, Almutaz Adileh, Yusuf Onur Kocberber, Stavros Volos, Mohammad Alisafaee, Djordje Jevdjic, Cansu Kaynak, Adrian Daniel Popescu, Anastasia Ailamaki, Babak Falsafi

 

Quantifying the Mismatch between Emerging Scale-Out Applications and Modern Processors

 

notes

 


cleared?

VLDB'12

 

Michele Dallachiesa, Besmira Nushi, Katsiaryna Mirylenka, Themis Palpanas

 

Uncertain Time-Series Similarity: Return to the Basics

 

notes

 

dispute!
cleared?

TACO'9

 

Servesh Muralidharan, Stephen Golan, David Gregg

 

Compiler support for lightweight context switching

 

notes

 


cleared?

VLDB'12

 

Peter Bailis, Shivaram Venkataraman, Michael J. Franklin, Joseph M. Hellerstein, Ion Stoica

 

Probabilistically Bounded Staleness for Practical Partial Quorums

 

notes

 

dispute!
cleared?

TODS'37

 

Yinan Li, Jignesh M Patel, Allison Terrell

 

WHAM: A High-Throughput Sequence Alignment Method

 

notes

 


cleared?

PLDI'12

 

Bruno C. d. S. Oliveira, Tom Schrijvers, Wontae Choi, Wonchan Lee, Kwangkeun Yi

 

The implicit calculus: a new foundation for generic programming

 

notes

 


cleared?

SOSP'11

 

Diego Ongaro, Stephen M. Rumble, Ryan Stutsman, John K. Ousterhout, Mendel Rosenblum

 

Fast crash recovery in RAMCloud

 

notes

 


cleared?

CCS'12

 

Willem De Groef, Dominique Devriese, Nick Nikiforakis, Frank Piessens

 

FlowFox: a web browser with flexible and precise information flow control

 

notes

 

dispute!
cleared?

OSDI'12

 

Matthew J Renzelmann, Asim Kadav, Michael M Swift

 

SymDrive: testing drivers without devices

 

notes

 


cleared?

CCS'12

 

Joseph A. Akinyele, Matthew Green, Susan Hohenberger, Matthew W. Pagano

 

Machine-generated algorithms, proofs and software for the batch verification of digital signature schemes

 

notes

 

dispute!
cleared?

OSDI'12

 

Adam Belay, Andrea Bittau, Ali Mashtizadeh, David Terei, David Mazieres, Christos Kozyrakis

 

Dune: safe user-level access to privileged CPU features

 

notes

 


cleared?

OOPSLA'12

 

Seonghoon Kang, Sukyoung Ryu

 

Formal specification of a JavaScript module system

 

notes

 

dispute!
cleared?

PLDI'12

 

Sergey Grebenshchikov, Nuno P. Lopes, Corneliu Popeea, Andrey Rybalchenko

 

Synthesizing software verifiers from proof rules

 

notes

 

dispute!
cleared?

PLDI'12

 

Chun Chen

 

Polyhedra scanning revisited

 

notes

 

dispute!
cleared?

SIGMOD'12

 

Xixuan Feng, Arun Kumar, Benjamn Recht, Christopher Re

 

Towards a unified architecture for in-RDBMS analytics

 

notes

 


cleared?

SIGMOD'12

 

Junhao Gan, Jianlin Feng, Qiong Fang, Wilfred Ng

 

Locality-sensitive hashing scheme based on dynamic collision counting

 

notes

 

dispute!
cleared?

OOPSLA'12

 

Giorgio Ausiello, Camil Demetrescu, Irene Finocchi, Donatella Firmani

 

k-Calling context profiling

 

notes

 

dispute!
cleared?

VLDB'12

 

Venu Satuluri, Srinivasan Parthasarathy

 

Bayesian Locality Sensitive Hashing for Fast Similarity Search

 

notes

 


cleared?

OOPSLA'12

 

Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, Andrew McGregor

 

AutoMan: a platform for integrating human-based and digital computation

 

notes

 

dispute!
cleared?

TACO'9

 

Antoniu Pop, Albert Cohen

 

OpenStream: Expressiveness and data-flow compilation of OpenMP streaming programs

 

notes

 


cleared?

CCS'12

 

Andreas Holzer, Martin Franz, Stefan Katzenbeisser, Helmut Veith

 

Secure two-party computations in ANSI C

 

notes

 


cleared?

OOPSLA'12

 

Ravi Chugh, David Herman, Ranjit Jhala

 

Dependent types for JavaScript

 

notes

 

dispute!
cleared?

TISSEC'15

 

Yossi Gilad, Amir Herzberg

 

LOT: A Defense Against IP Spoofing and Flooding Attacks

 

notes

 

dispute!
cleared?

OSDI'12

 

David Isaac Wolinsky, Henry Corrigan-Gibbs, Bryan Ford, Aaron Johnson

 

Dissent in numbers: making strong anonymity scale

 

notes

 

dispute!
cleared?

OOPSLA'12

 

Sam Tobin-Hochstadt, David Van Horn

 

Higher-order symbolic execution via contracts

 

notes

 

dispute!
cleared?

PLDI'12

 

Emilio Coppa, Camil Demetrescu, Irene Finocchi

 

Input-sensitive profiling

 

notes

 


cleared?

SOSP'11

 

Raluca A. Popa, Catherine M. S. Redfield, Nickolai Zeldovich, Hari Balakrishnan

 

CryptDB: protecting confidentiality with encrypted query processing

 

notes

 


cleared?

VLDB'12

 

Felix Halim, Stratos Idreos, Panagiotis Karras, Roland H C Yap

 

Stochastic Database Cracking: Towards Robust Adaptive Indexing in Main-Memory Column-Stores

 

notes

 

dispute!
cleared?

OOPSLA'12

 

T. Stephen Strickland, Sam Tobin-Hochstadt, Robert Bruce Findler, Matthew Flatt

 

Chaperones and impersonators: run-time support for reasonable interposition

 

notes

 

dispute!
cleared?

PLDI'12

 

Aaron Turon

 

Reagents: expressing and composing fine-grained concurrency

 

notes

 


cleared?

OOPSLA'12

 

Pavel Parizek, Ondrej Lhotak

 

Predicate abstraction of Java programs with collections

 

notes

 


cleared?

OSDI'12

 

Xi Wang, Haogang Chen, Zhihao Jia, Nickolai Zeldovich, M Frans Kaashoek

 

Improving integer security for systems with KINT

 

notes

 


cleared?

OOPSLA'12

 

Laura Effinger-Dean, Brandon Lucia, Luis Ceze, Dan Grossman, Hans-Juergen Boehm

 

IFRit: interference-free regions for dynamic data-race detection

 

notes

 

dispute!
cleared?

PLDI'12

 

John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, Xuejun Yang

 

Test-case reduction for C compiler bugs

 

notes

 

dispute!
cleared?

9 Purported Building But Found Not Building (0)

The original study claimed that the paper did build, but the re-examination has found it does not build.
Nothing here!

10 Purported Not Building; Confirmed (2)

The original study claimed that the paper did not build, and the re-examination has confirmed this.

TOPLAS'34

 

Zhengqin luo, Tamara Rezk, Manuel Serrano, Gerard Boudol

 

Reasoning about Web Applications: An Operational Semantics for HOP

 

notes

 

dispute!

problem?

PLDI'12

 

Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, Shan Lu

 

Understanding and detecting real-world performance bugs

 

notes

 



problem?

11 Purported Building; Confirmed (1)

The original study claimed that the paper did build, and the re-examination has confirmed this.

OOPSLA'12

 

Yanhong A. Liu, Scott D. Stoller, Bo Lin, Michael Gorbovitski

 

From clarity to efficiency for distributed algorithms

 

notes

 

dispute!
cleared?

12 All Others Purported Not Building (63)

The original study claimed that the paper did not build, and nobody has initiated re-examination.

OOPSLA'12

 

Stefan Muller, Stephen Chong

 

Towards a practical secure concurrent language

 

notes

 




CCS'12

 

Mihhail Aizatulin, Andrew D. Gordon, Jan Jurjens

 

Computational verification of C protocol implementations by symbolic execution

 

notes

 




PLDI'12

 

Florian Benz, Andreas Hildebrandt, Sebastian Hack

 

A dynamic program analysis to find floating-point accuracy problems

 

notes

 




PLDI'12

 

Justin Holewinski, Ragavendar Ramamurthi, Mahesh Ravishankar, Naznin Fauzia, Louis-Noel Pouchet, Atanas Rountev, P. Sadayappan

 

Dynamic trace-based analysis of vectorization potential of applications

 

notes

 




TISSEC'15

 

Karthikeyan Bhargavan, Cedric Fournet, Ricardo Corin, Eugen Zalinescu

 

Verified Cryptographic Implementations for TLS

 

notes

 




OOPSLA'12

 

Michael Kling, Sasa Misailovic, Michael Carbin, Martin C. Rinard

 

Bolt: on-demand infinite loop escape in unmodified binaries

 

notes

 




TOCS'30

 

Adrian Schupbach, Andrew Baumann, Timothy Roscoe, Simon Peter

 

A Declarative Language Approach to Device Configuration

 

notes

 




VLDB'12

 

Cheng Sheng, Nan Zhang, Yufei Tao, Xin Jin

 

Optimal Algorithms for Crawling a Hidden Database in the Web

 

notes

 




ASPLOS'12

 

Joseph L. Greathouse, Hongyi Xin, Yixin Luo, Todd Austin

 

A case for unlimited watchpoints

 

notes

 




VLDB'12

 

Brian E. Ruttenberg, Ambuj K. Singh

 

Indexing the Earth Mover's Distance Using Normal Distributions

 

notes

 




VLDB'12

 

Lingkun Wu, Xiaokui Xiao, Dingxiong Deng, Gao Cong, Andy Diwen Zhu, Shuigeng Zhou

 

Shortest Path and Distance Queries on Road Networks: An Experimental Evaluation

 

notes

 




OOPSLA'12

 

Tao Bao, Yunhui Zheng, Xiangyu Zhang

 

White box sampling in uncertain data processing enabled by program analysis

 

notes

 




TOCS'30

 

George Candea, Vitaly Chipounov, Volodymyr Kuznetsov

 

The S2E Platform: Design, Implementation, and Applications

 

notes

 




ASPLOS'12

 

Michael Ferdman, Almutaz Adileh, Onur Kocberber, Stavros Volos, Mohammad Alisaface, Djordje Jevdjic, Cansu Kaynak, Adian Daniel Popescu, Anastasia Ailamaki, Babak Falsafi

 

Clearing the clouds: a study of emerging scale-out workloads on modern hardware

 

notes

 




ASPLOS'12

 

Austin T. Clements, M Frans Kaashoek, Zickolai Zeldovich

 

Scalable address spaces using RCU balanced trees

 

notes

 




TACO'9

 

Juan Carlos Juega, Sven Verdoolaege, Albert Cohen, Jose Ignacio Gomez, Christian Tenllado, Francky Catthoor

 

Polyhedral parallel code generation for CUDA

 

notes

 




OOPSLA'12

 

Xudong Wang, Xuanzhe Liu, Ying Zhang, Gang Huang

 

Migration and execution of JavaScript applications between mobile devices and cloud

 

notes

 




VLDB'12

 

Arlei Silva, Wagner Meira Jr, Mohammed J. Zaki

 

Mining Attribute-structure Correlated Patterns in Large Attributed Graphs

 

notes

 




CCS'12

 

Jose Bacelar Almeida, Manuel Barbosa, Endre Bangerter, Gilles Barthe, Stephan Krenn, Santiago Zanella Beguelin

 

Full proof cryptography: verifiable compilation of efficient zero-knowledge protocols

 

notes

 




TACO'9

 

Huimin Cui, Qing Li, Jingling Xue, Xiaobing Feng

 

Layout-oblivious compiler optimization for matrix computations

 

notes

 




OOPSLA'12

 

Ulrik Pagh Schultz

 

Programming language abstractions for self-reconfigurable robots

 

notes

 




SOSP'11

 

Hyeontaek Lim, Bin Fan, David G. Andersen, Michael Kaminsky

 

SILT: a memory-efficient, high-performance key-value store

 

notes

 




TACO'9

 

Vincent Loechner, Rachid Seghir, Benoit Meister

 

Integer affine transformations of parametric ;-polytopes and applications to loop nest optimization

 

notes

 




TOPLAS'34

 

Xing Wu, Frank Mueller

 

ScalaExtrap: Trace-based communication extrapolation for SPMD programs

 

notes

 




PLDI'12

 

Michael Carbin, Deokhwan Kim, Sasa Misailovic, Martin C. Rinard

 

Proving acceptability properties of relaxed nondeterministic approximate programs

 

notes

 




OOPSLA'12

 

Yuriy Solodkyy, Gabriel Dos Reis, Bjarne Stroustrup

 

Open and efficient type switch for C++

 

notes

 




OOPSLA'12

 

K. Rustan M. Leino, Aleksandar Milicevic

 

Program extrapolation with jennisys

 

notes

 




SIGMOD'12

 

Ruomin Jin, Ning Ruan, Saikat Dey, Jeffrey Yu Xu

 

SCARAB: scaling reachability computation on large graphs

 

notes

 




VLDB'12

 

Kyriakos Mouratidis, Man Lung Yiu

 

Shortest Path Computation with No Information Leakage

 

notes

 




PLDI'12

 

Marc de Kruijf, Karthikeyan Sankaralingam, Somesh Jha

 

Static analysis and compiler design for idempotent processing

 

notes

 




TOPLAS'34

 

Lakshminarayanan Renganarayanan, Daegon Kim, Michelle Mills Strout, Sanjay Rajopadhye

 

Parameterized loop tiling

 

notes

 




SIGMOD'12

 

Jelle Hellings, George H.L. Fletcher, Herman Haverkort

 

Efficient external-memory bisimulation on DAGs

 

notes

 




SOSP'11

 

Patrick Colp, Mihir Nanavati, Jun Zhu, William Aiello, George Coker, Tim Deegan, Peter Loscocco, Andrew Warfield

 

Breaking up is hard to do: security and functionality in a commodity hypervisor

 

notes

 




PLDI'12

 

Ming Kawaguchi, Patrick Maxim Rondon, Alexander Bakst, Ranjit Jhala

 

Deterministic parallelism via liquid effects

 

notes

 




VLDB'12

 

Francois Goasdoue, Konstantinos Karanasos, Julien Leblay, Ioana Manolescu

 

View Selection in Semantic Web Databases

 

notes

 




TACO'9

 

Kishore Kumar Pusukuri, Rajix Gupta, Laxmi N. Bhuyan

 

ADAPT: A framework for coscheduling multithreaded programs

 

notes

 




TOPLAS'34

 

Frank Tip, Mandana Vaziri, Jan Vitek, Julian Dolby, Christian Hammer, Daniel Marino

 

A data-centric approach to synchronization

 

notes

 




SOSP'11

 

Yair Sovran, Russell Power, Marcos K. Aguilera, Jinyang Li

 

Transactional storage for geo-replicated systems

 

notes

 




OOPSLA'12

 

Kohei Suenaga, Ryota Fukuda, Atsushi Igarashi

 

Type-based safe resource deallocation for shared-memory concurrency

 

notes

 




TOCS'30

 

Hadi Esmaeilzadeh, Emily R. Blem, Renee St. Amant, Karthikeyan Sankaralingam, Doug Burger

 

Power Limitations and Dark Silicon Challenge the Future of Multicore

 

notes

 




CCS'12

 

Rui Chen, Gergely Acs, Calude Castelluccia

 

Differentially private sequential data publication via variable-length n-grams

 

notes

 




PLDI'12

 

Volodymyr Kuznetsov, Johannes Kinder, Stefan Bucur, George Candea

 

Efficient state merging in symbolic execution

 

notes

 




SIGMOD'12

 

Piero Fraternali, Davide Martinenghi, Marco Tagliasacchi

 

Top-k bounded diversification

 

notes

 




OOPSLA'12

 

Michael Cohen, Haitao Steve Zhu, Emgin Ezgi Senem, Yu David Liu

 

Energy types

 

notes

 




SIGMOD'12

 

Russell Sears, Raghu Ramakrishnan

 

bLSM: a general purpose log structured merge tree

 

notes

 




VLDB'12

 

Kaibo Wang, Yin Huai, Rubao Lee, Fusheng Wang, Xiaodong Zhang, Joel H. Saltz

 

Accelerating Pathology Image Data Cross-Comparison on CPU-GPU Hybrid Systems

 

notes

 




TODS'37

 

Dmitri V Kalashnikov, Rabia Nuray-Turan, Sharad Mehrotra

 

Exploiting Web querying for Web people search

 

notes

 




VLDB'12

 

Yi Zhang, Jun Yang

 

Optimizing I/O for Big Array Analytics

 

notes

 




VLDB'12

 

Jeffrey Jestes, Ke Yi, Feifei Li

 

Building Wavelet Histograms on Large Data in MapReduce

 

notes

 




VLDB'12

 

Arun Kumar, Christopher Re

 

Probabilistic Management of OCR Data using an RDBMS

 

notes

 




TACO'9

 

Fabien Coelho, Francois Irigoin

 

API compilation for image hardware accelerators

 

notes

 




OOPSLA'12

 

Filip Voracek, Zdenek Tronicek

 

JaDaRD: java data-race detector

 

notes

 




SOSP'11

 

Wyatt Lloyd, Michael J. Freedman, Michael Kaminsky, David G. Andersen

 

Don't settle for eventual: scalable causal consistency for wide-area storage with COPS

 

notes

 




TOPLAS'34

 

Maartie De Jonge, Lennart C. L. Kats, Eelco Visser, Emma Soderberg

 

Natural and Flexible Error Recovery for Generated Modular Language Environments

 

notes

 




CCS'12

 

Dimitris Geneiatakis, Georgios Portokalidis, Vasileios P. Kemerlis, Angelos D. Keromytis

 

Adaptive defenses for commodity software through virtual application partitioning

 

notes

 




TACO'9

 

Riyadh Baghdadi, Albert Cohen, Sven Verdoolaege, Konrad Trifunovic

 

Improved loop tiling based on the removal of spurious false dependences

 

notes

 




CCS'12

 

Kathy Wain Yee Au, Yi Fan Zhou, Zhen Huang, David Lie

 

PScout: analyzing the Android permission specification

 

notes

 




VLDB'12

 

Hongzhi Yin, Bin Cui, Jing Li, Junjie Yao, Chen Chen

 

Challenging the Long Tail Recommendation

 

notes

 




CCS'12

 

Marten van Dijk, Ari Juels, Alina Oprea, Ronald L. Rivest, Emil Stefanov, Nikos Triandopoulos

 

Hourglass schemes: how to prove that cloud files are encrypted

 

notes

 




OOPSLA'12

 

Gabriel D. P. Konat, Vlad A. Vergu, Lennart C. L. Kats, Guido Wachsmuth, Eelco Visser

 

The spoofax name binding language

 

notes

 




TODS'37

 

Dmitri V Kalashnikov, Rabia Nuray-Turan, Sharad Mehrotra, Yaming Yu

 

Attribute and object selection queries on objects with probabilistic attributes

 

notes

 




VLDB'12

 

Georgios John Fakas, Zhi Cai, Nikos Mamoulis

 

Size-l Object Summaries for Relational Keyword Search

 

notes

 




OOPSLA'12

 

Jie Yu, Satish Narayanasamy, Cristiano Pereira, Gilles Pokam

 

Maple: a coverage-driven testing tool for multithreaded programs

 

notes

 




13 All Other Purported Building (108)

The original study claimed that the paper did build, and nobody has initiated re-examination.

SIGMOD'12

 

Danupon Nanongkai, Ashwin Lall, Atish Das Sarma, Kazuhisa Makino

 

Interactive regret minimization

 

notes

 




OOPSLA'12

 

Anton Willy Dubrau, Laurie J. Hendren

 

Taming MATLAB

 

notes

 




VLDB'12

 

Robert Isele, Christian Bizer

 

Learning Expressive Linkage Rules using Genetic Programming

 

notes

 




TISSEC'15

 

Norman Danner, Sam DeFabbia-Kane, Danny Krizanc, Marc Liberatore

 

Effectiveness and detection of denial-of-service attacks in tor

 

notes

 




OSDI'12

 

Sangjin Han, Scott Marshall, Byung-Gon Chun, Sylvia Ratnasamy

 

Megapipe: a new programming interface for scalable network i/o

 

notes

 




SIGMOD'12

 

Nikos Giatrakos, Antonios Deliginnakis, Minos Garadalakis, Izchak Shafman, Assaf Schuster

 

Prediction-based geometric monitoring over distributed data streams

 

notes

 




SOSP'11

 

Emin Gun Sirer, Willem de Bruijn, Patrick Reynolds, Alan Shieh, Kevin Walsh, Dan Williams, Fred B. Schneider

 

Logical attestation: an authorization architecture for trustworthy computing

 

notes

 




CCS'12

 

Weining Yang, Ninghui Li, Yuan Qi, Wahbeh Qardaji, Stephen McLaughlin, Patrick McDaniel

 

Minimizing private data disclosures in the smart grid

 

notes

 




VLDB'12

 

Paolo Boldi, Francesco Bonchi, Aristides Gionis, Tamir Tassa

 

Injecting Uncertainty in Graphs for Identity Obfuscation

 

notes

 




OOPSLA'12

 

Alessandro Ricci, Andrea Santi

 

From actors to agent-oriented programming abstractions in simpAL

 

notes

 




SIGMOD'12

 

Ruoming Jin, Ning Ruan, Yang Xiang, Victor E. Lee

 

A highway-centric labeling approach for answering distance queries on large sparse graphs

 

notes

 




VLDB'12

 

Mateusz Pawlik, Nikolaus Augsten

 

RTED: A Robust Algorithm for the Tree Edit Distance

 

notes

 




VLDB'12

 

Chao Li, Gerome Miklau

 

An Adaptive Mechanism for Accurate Query Answering under Differential Privacy

 

notes

 




VLDB'12

 

Ziyu Guan, Xifeng Yan, Lance M. Kaplan

 

Measuring Two-Event Structural Correlations on Graphs

 

notes

 




CCS'12

 

Hooman Mohajeri Moghaddam, Baiyu Li, Mohammad Derakhshani, Ian Goldberg

 

SkypeMorph: protocol obfuscation for Tor bridges

 

notes

 




CCS'12

 

Qiyan Wang, Xun Gong, Giang T. K Nguyen, Amir Houmansadr, Nikita Borisov

 

CensorSpoofer: asymmetric communication using IP spoofing for censorship-resistant web browsing

 

notes

 




OOPSLA'12

 

Gustavo Soares Soares

 

Automated behavioral testing of refactoring engines

 

notes

 




TODS'37

 

Boduo Li, Edward Mazur, Yanlei Diao, Andrew Mcgregor, Prashant Shenoy

 

SCALLA: A Platform for Scalable One-Pass Analytics Using MapReduce

 

notes

 




SIGMOD'12

 

Steven Begley, Zhen He, Yi-Ping Phoebe Chen

 

MCJoin: a memory-constrained join for column-store main-memory databases

 

notes

 




TODS'37

 

Daniel Lemire, Owen Kaser, Eduardo Gutarra

 

Reordering rows for better compression: Beyond the lexicographic order

 

notes

 




VLDB'12

 

Di Yang, Elke A. Rundensteiner, Matthew O. Ward

 

Summarization and Matching of Density-Based Clusters in Streaming Environments

 

notes

 




VLDB'12

 

Xiaochun Yang, Honglei Liu, Bin Wang

 

ALAE: Accelerating Local Alignment with Affine Gap Exactly in Biosequence Databases

 

notes

 




TACO'9

 

Yi Yang, Ping Xiang, Jingfei Kong, Mike Mantor, Huiyang Zhou

 

A unified optimizing compiler framework for different GPGPU architectures

 

notes

 




OOPSLA'12

 

Sebastien Bocq, Koen Daenen

 

Molecule: using monadic and streaming I/O to compose process networks on the JVM

 

notes

 




PLDI'12

 

Rachid Guerraoui, Viktor Kuncak, Giuliano Losa

 

Speculative linearizability

 

notes

 




OSDI'12

 

Mark S Gordon, D Anoushe Jamshidi, Scott Mahlke, Z Morley Mao, Xu Chen

 

COMET: code offload by migrating execution transparently

 

notes

 




OSDI'12

 

Mona Attariyan, Michael Chow, Jason Flinn

 

X-ray: automating root-cause diagnosis of performance anomalies in production software

 

notes

 




SIGMOD'12

 

Prashanth Mohan, Abhradeep Thakurta, Elaine Shi, Dawn Song, David E. Culler

 

GUPT: privacy preserving data analysis made easy

 

notes

 




PLDI'12

 

Charisee Chiw, Gordon Kindlmann, John Reppy, Lamont Samuels, Nick Seltzer

 

Diderot: a parallel DSL for image analysis and visualization

 

notes

 




VLDB'12

 

Dan Lin, Christian S. Jensen, Rui Zhang, Lu Xiao, Jiaheng Lu

 

A MovingObject Index for Efficient Query Processing with Peer-Wise Location Privacy

 

notes

 




CCS'12

 

Sascha Fahl, Marian Harbach, Thomas Muders, Matthew Smith, Lars Baumgartner, Bernd Freisleben

 

Why eve and mallory love android: an analysis of android SSL (in)security

 

notes

 




TACO'9

 

David Yuste, Erven Rohou, Kevin Williams

 

Vectorization technology to improve interpreter performance

 

notes

 




VLDB'12

 

Pirooz Chubak, Davood Rafiei

 

Efficient Indexing and Querying over Syntactically Annotated Trees

 

notes

 




SIGMOD'12

 

Shengqi Yang, Xifeng Yan, Bo Zong, Arijit Khan

 

Towards effective partition management for large graphs

 

notes

 




TODS'37

 

Wenjie Zhang, Xuemin Lin, Ying Zhang, Muhammad Aamir Cheema, Qing Zhang

 

Stochastic skylines

 

notes

 




VLDB'12

 

Yanif Ahmad, Oliver Kennedy, Christoph Koch, Milos Nikolic

 

DBToaster: Higher-order Delta Processing for Dynamic, Frequently Fresh Views

 

notes

 




OOPSLA'12

 

Grigore Rosu, Andrei Stefanescu

 

Checking reachability using matching logic

 

notes

 




OOPSLA'12

 

Paolo G. Giarrusso

 

Reifying and optimizing collection queries for modularity

 

notes

 




VLDB'12

 

Yongxin Tong, Lei Chen, Yurong Cheng, Philip S. Yu

 

Mining Frequent Itemsets over Uncertain Databases

 

notes

 




SIGMOD'12

 

James Cheng, Shumo Chu, Yiping Ke, Carter Cheng

 

Efficient processing of distance queries in large graphs: a vertex cover approach

 

notes

 




PLDI'12

 

Jeffrey Fischer, Rupak Majumdar, Shahram Esmaeilsabzali

 

Engage: a deployment management system

 

notes

 




PLDI'12

 

Yong Hun Eom, Brian Demsky

 

Self-stabilizing Java

 

notes

 




VLDB'12

 

Graham Cormode, Justin Thaler, Ke Yi

 

Verifying Computations with Streaming Interactive Proofs

 

notes

 




PLDI'12

 

Greg Morrisett, Gang Tan, Joseph Tassarotti, Jean-Baptiste Tristan, Edward Gan

 

RockSalt: better, faster, stronger SFI for the x86

 

notes

 




CCS'12

 

Antonio Bianchi, Yan Shoshitaishvili, Christopher Kruegel, Giovanni Vigna

 

Blacksheep: detecting compromised hosts in homogeneous crowds

 

notes

 




OOPSLA'12

 

Xiaomin Xu, Sheng Huang, Yanghua Xiao, Wei Wang

 

SCuV: a novel software clustering and visualization tool

 

notes

 




TOPLAS'34

 

Bruno De Fraine, Erik Ernst, Mario Sudholt

 

Essential AOP: The a calculus

 

notes

 




OOPSLA'12

 

Youngjoon Jo, Milind Kulkarni

 

Automatically enhancing locality for tree traversals with traversal splicing

 

notes

 




ASPLOS'12

 

Nedeljko Vasic, Dejan Novakovic, Svetozar Miucin, Dejan Kostic, Ricardo Bianchini

 

DejaVu: accelerating resource allocation in virtualized environments

 

notes

 




ASPLOS'12

 

Peter Feiner, Angela Demke Brown, Ashvin Goel

 

Comprehensive kernel instrumentation via dynamic binary translation

 

notes

 




VLDB'12

 

Wei Lu, Yanyan Shen, Beng Chin Ooi, Su Chen

 

Efficient Processing of k Nearest Neighbor Joins using MapReduce

 

notes

 




VLDB'12

 

Ahmed M. Aly, Walid G. Aref, Mourad Ouzzani

 

Spatial Queries with Two kNN Predicates

 

notes

 




OOPSLA'12

 

Christian Kastner, Klaus Ostermann, Sebastian Erdweg

 

A variability-aware module system

 

notes

 




ASPLOS'12

 

David Meisner, Thomas F. Wenisch

 

DreamWeaver: architectural support for deep sleep

 

notes

 




CCS'12

 

Mads Dam, Gurvan Le Guernic, Andreas Lundblad

 

TreeDroid: a tree automaton based approach to enforcing data processing policies

 

notes

 




TACO'9

 

Christian Wimmer, Michael Haupt, Michael L. VanDeVanter, Mick Jordan, Laurent Daynes, Douglas Simon

 

Maxine: An approachable virtual machine for, and in, java

 

notes

 




PLDI'12

 

Brian Hackett, Shu-yu Guo

 

Fast and precise hybrid type inference for JavaScript

 

notes

 




VLDB'12

 

Christopher Hoobin, Simon J. Puglisi, Justin Zobel

 

Relative Lempel-Ziv Factorization for Efficient Storage and Retrieval of Web Collections

 

notes

 




SOSP'11

 

Tongping Liu, Charlie Curtsinger, Emery D. Berger

 

Dthreads: efficient deterministic multithreading

 

notes

 




VLDB'12

 

Michael Armbrust, Kristal Curtis, Tim Kraska, Armando Fox, Michael J. Franklin, David A. Patterson

 

PIQL: Success-Tolerant Query Processing in the Cloud

 

notes

 




CCS'12

 

Zachary Weinberg, Jeffrey Wang, Vinod Yegneswaran, Linda Briesemeister, Steven Cheung, Frank Wang, Dan Boneh

 

StegoTorus: a camouflage proxy for the Tor anonymity system

 

notes

 




OOPSLA'12

 

Shams Mahmood Imam, Vivek Sarkar

 

Integrating task parallelism with actors

 

notes

 




SIGMOD'12

 

YongChul Kwon, Magdalena Balazinska, Bill Howe, Jerome Rolia

 

SkewTune: mitigating skew in mapreduce applications

 

notes

 




CCS'12

 

Myrto Arapinis, Loretta Mancini, Eike Ritter, Mark Ryan, Nico Golde, Kevin Redon, Ravishankar Borgaonkar

 

New privacy issues in mobile telephony: fix and verification

 

notes

 




SIGMOD'12

 

Barzan Mozafari, Kai Zeng, Carlo Zaniolo

 

High-performance complex event processing over XML streams

 

notes

 




TODS'37

 

Leonid Libkin, Claire David, Tony Tan

 

Efficient reasoning about data trees via integer linear programming

 

notes

 




VLDB'12

 

Haohan Zhu, George Kollios, Vassilis Athitsos

 

A Generic Framework for Efficient and Effective Subsequence Retrieval

 

notes

 




SIGMOD'12

 

Manish Singh, Arnab Nandi, Hosagrahar V. Jagadish

 

Skimmer: rapid scrolling of relational query results

 

notes

 




VLDB'12

 

Jun Zhang, Zhenjie Zhang, Xiaokui Xiao, Yin Yang, Marianne Winslett

 

Functional Mechanism: Regression Analysis under Differential Privacy

 

notes

 




TODS'37

 

Yufei Tao, Cheng Sheng, Jianzhong Li

 

Exact and approximate algorithms for the most connected vertex problem

 

notes

 




VLDB'12

 

Guoliang Li, Dong Deng, Jiannan Wang, Jianhua Feng

 

PASS-JOIN: A Partition-based Method for Similarity Joins

 

notes

 




OOPSLA'12

 

Vincent St-Amour, Sam Tobin-Hochstadt, Matthias Felleisen

 

Optimization coaching: optimizers learn to communicate with programmers

 

notes

 




VLDB'12

 

Fabian M. Suchanek, Serge Abiteboul, Pierre Senellart

 

PARIS: Probabilistic Alignment of Relations, Instances,and Schema

 

notes

 




TACO'9

 

Cedric Nugteren, Pieter Custers, Henk Corporal

 

Algorithmic species: A classification of affine loop nests for parallel programming

 

notes

 




TOPLAS'34

 

Patrick Lam, Eric Bodden, Laurie Hendren

 

Partially Evaluating Finite-State Runtime Monitors Ahead of Time

 

notes

 




OSDI'12

 

Philip Levis

 

Experiences from a decade of TinyOS development

 

notes

 




OSDI'12

 

Aapo Kyrola, Guy Blelloch, Carlos Guestrin

 

GraphChi: Large-scale graph computation on just a PC

 

notes

 




TISSEC'15

 

Jan Camenisch, Thomas Gross

 

Efficient Attributes for Anonymous Credentials

 

notes

 




VLDB'12

 

Hoang Tam Vo, Sheng Wang, Divyakant Agrawal, Gang Chen, Beng Chin Ooi

 

LogBase: A Scalable Log-structured Database System in the Cloud

 

notes

 




CCS'12

 

Reza Shokri, George Theodorakopoulos, Carmela Troncoso, Jean-Pierre Hubaux, Jean-Yves Le Boudec

 

Protecting location privacy: optimal strategy against localization attacks

 

notes

 




VLDB'12

 

Jia Wang, James Cheng

 

Truss Decomposition in Massive Networks

 

notes

 




VLDB'12

 

Essam Mansour, Amin Allam, Spiros Skiadopoulos, Panos Kalnis

 

ERA: Efficient Serial and Parallel Suffix Tree Construction for Very Long Strings

 

notes

 




ASPLOS'12

 

Sungpack Hong, Hassan Chafi, Eric Sedlar, Kunle Olukotun

 

Green-Marl: a DSL for easy and efficient graph analysis

 

notes

 




OOPSLA'12

 

Kivanc Muslu, Yuriy Brun, Reid Holmes, Michael D. Ernst, David Notkin

 

Speculative analysis of integrated development environment recommendations

 

notes

 




SIGMOD'12

 

Haibo Hu, Jianliang Xu, Qian Chen, Ziwei Yang

 

Authenticating location-based services without compromising location privacy

 

notes

 




VLDB'12

 

Theodoros Lappas, Marcos R. Vieira, Dimitrios Gunopulos, Vassilis J. Tsotras

 

On The Spatiotemporal Burstiness of Terms

 

notes

 




PLDI'12

 

Paul Gazzillo, Robert Grimm

 

SuperC: parsing all of C by taming the preprocessor

 

notes

 




TOPLAS'34

 

Jan Hoffmann, Klaus Aehlig, Martin Hofmann

 

Multivariate amortized resource analysis

 

notes

 




VLDB'12

 

Bo Zhao, Benjamin I P Rubinstein, Jim Gemmell, Jiawei Han

 

A Bayesian Approach to Discovering Truth from Conflicting Sources for Data Integration

 

notes

 




CCS'12

 

Joan Calvet, Jose M. Fernandez, Jean-Yves Marion

 

Aligot: cryptographic function identification in obfuscated binary programs

 

notes

 




VLDB'12

 

Dora Erdos, Vatche Ishakian, Andrei Lapets, Evimaria Terzi, Azer Bestavros

 

The Filter-Placement Problem and its Application to Minimizing Information Multiplicity

 

notes

 




CCS'12

 

Max Schuchard, John Geddes, Christopher Thompson, Nicholas Hopper

 

Routing around decoys

 

notes

 




VLDB'12

 

Yizhou Sun, Charu C. Aggarwal, Jiawei Han

 

Relation Strength-Aware Clustering of Heterogeneous Information Networks with Incomplete Attributes

 

notes

 




TODS'37

 

Davide Martinenghi, Marco Tagliasacchi

 

Proximity measures for rank join

 

notes

 




VLDB'12

 

Ganzhao Yuan, Zhenjie Zhang, Marianne Winslett, Xiaokui Xiao, Yin Yang, Zhifeng Hao

 

Low-Rank Mechanism: Optimizing Batch Queries under Differential Privacy

 

notes

 




OOPSLA'12

 

Wei Huang, Ana Milanova, Werner Dietl, Michael D. Ernst

 

Reim {\&} ReImInfer: checking and inference of reference immutability and method purity

 

notes

 




CCS'12

 

Dominique Schroder, Heike Schroder

 

Verifiable data streaming

 

notes

 




TACO'9

 

Michele Tartara, Stefano Crespi Reghizzi

 

Continuous learning of compiler heuristics

 

notes

 




OOPSLA'12

 

Fadi Meawad, Gregor Richards, Floreal Morandat, Jan Vitek

 

Eval begone!: semi-automated removal of eval from javascript programs

 

notes

 




TOPLAS'34

 

Jan Smans, Bart Jacobs, Frank Piessens

 

Implicit dynamic frames

 

notes

 




TACO'9

 

Bita Mazloom, Shashidhar Mysore, Mohit Tiwari, Banit Agrawal, Tim Sherwood

 

Dataflow Tomography: Information Flow Tracking For Understanding and Visualizing Full Systems

 

notes

 




TOCS'30

 

Tyler Harter, Chris Dragga, Michael Vaughn, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau

 

A File Is Not a File: Understanding the I/O Behavior of Apple Desktop Applications

 

notes

 




CCS'12

 

Peter Williams, Radu Sion, Alin Tomescu

 

PrivateFS: a parallel oblivious file system

 

notes

 




VLDB'12

 

Amit Goyal, Francesco Bonchi, Laks V. S. Lakshmanan

 

A Data-Based Approach to Social Influence Maximization

 

notes

 




VLDB'12

 

Francesco Gullo, Andrea Tagarelli

 

Uncertain Centroid based Partitional Clustering of Uncertain Data

 

notes

 




OSDI'12

 

J Gonzalez, Yucheng Low, Haijie Gu, Danny Bickson, Carlos Guestrin

 

PowerGraph: Distributed graph-parallel computation on natural graphs

 

notes

 




OOPSLA'12

 

Paolo G. Giarrusso, Klaus Ostermann, Michael Eichberg, Tillmann Rendel, Christian Kastner

 

Reifying and optimizing collection queries for modularity

 

notes

 




TACO'9

 

Yan Cui, Yingxin Wang, Yu Chen, Yuanchun Shi

 

Lock-contention-aware scheduler: A scalable and energy-efficient method for addressing scalability collapse on multicore systems

 

notes

 




14 Threats to Validity

Some threats to the validity of our attempt to validate their results include:
  • We are using a slightly different standard of evaluation.

  • Some artifacts may have changed since the time they were tested by Collberg’s team.

  • Some authors may have fixed their artifacts after reading the build logs on the original site.

  • Different platforms may yield different build issues.