I'm heartened to see Professor Shriram Krishnamurthi spearheading an effort to help evaluate yesterday's reproducibility project.
We are evaluating the results presented by the study at the University of Arizona. Our goal is to allow authors and any other interested parties to review that study’s findings about an individual paper, and attempt to reconstruct their findings. We will summarize the results here.
We are grateful to Collberg, et al. for initiating this discussion and making all their data available. This is a valuable service based on an enormous amount of manual labor. Even if we end up disagreeing with some of their findings, we remain deeply appreciative of their service to the community by highlighting these important issues.
We do wish disagree with the use of the term “reproducibility”, which many people associate with an independent reconstruction of the work. For instance, this paper spells out the difference between repeatability and reproducibility and provides an interesting case study.
I've already submit a dispute, where code isn't available where it was thought that it was. I plan to do a little more looking, time permitting.
Others have successfuly built projects reported as failures. I think that the community is going to help repair the results of the other study.
Combined, they're going to say much more about the state of the field!