Eeeeek

[petefred]

This sort of thing absolutely scares the crap out of me. I'm always worried that no matter how much I check and test and debug, and no matter how paranoid I am, somehow something dumb like this will happen. And I'll have worked for 5 or 10 years on what amounts to nothing. I give this guy a lot of credit for promptly acting to correct his mistakes

Comments

[finglegend]

Ouch! This kind of thing was the theme of Greg Wilson’s talk at the SciPy conference this year. Unfortunately, computational science is a bit lacking in the “repeatability” department. Of course, that’s incredibly ironic as, from a feasibility standpoint, it should be the most repeatable aspect of science.

Documenting and disseminating lab protocols? Submitting them to peer review? Keeping detailed lab notebooks? [huzzahs all around!]

Releasing the source code of your data analysis software and precise recipes for reproducing your published graphs and numbers? Code review? Source control?!? [crickets]

“An in-house data reduction program” is worth Sweet Fanny Adams as long as it stays “in-house.”

[yellowphoton]

Amen to that.

Honestly, I've yet to meet a computational paper, even one doing what seems like fairly straight-forward calculation, that we could reproduce the results of based on the publicly available paper & supplements. Somewhere or other there's always some magic handwaving involving in-house code with a bunch of parameters that never get specified.

So why can't peer-reviewed journals establish a nice little online data-base where people upload the source-code that goes along with their papers. It'd make blunders like this a lot easier to detect, not to mention eliminating the vast amounts of time everyone else has to spend reinventing their wheel!

[petefred]

Hm, intresting talk from the looks of the slides (the ppt link appears broken, but there's a pdf with the same base name

[petefred]

I guess what I'm getting at at the end is, how does one balance the desire to do work at the frontier of what is possible, with the need for reproducibility, since both are important? This is completely tangential to the discussion on the badness of concealed in-house code and magical parameters, but still an issue...