Photo by everything possible, Shutterstock.
Stanford Medicine Scope - December 15th, 2016 - by Jennie Dusheck
A team of scientists including Stanford’s John Ioannidis, MD, DSc, has proposed a set of principles to improve the transparency and reproducibility of computational methods used in all areas of research. The group’s summary of those principles, known as the Reproducibility Enhancement Principles, was published recently in a paper in Science.
The effort is an attempt to address the problem that much scientific work cannot be reproduced.
One major hurdle for reproducing scientific work is that even when researchers have access to all the original data, it’s often unclear what kinds of computational tools were used to analyze it, Ioannidis said. As the authors wrote: “Access to the computational steps taken to process data and generate findings is as important as access to data itself.”
“Research can be very complex, very granular, and with lots of numbers, equations and lots of script,” said Ioannidis, who is the co-director of the Meta-Research Innovation Center at Stanford (METRICS), which aims to improve research practices. “This is good news, because it means we have expanding horizons in what we can do. But at the same time it becomes more of a black box. And we should still be able to understand what that black box does. People who want to understand the results of a piece of research need to know why the computational black box is there, how it functions, and whether it always gives the same answer.”
The paper’s authors concede that some of the Reproducibility Enhancement Principles may be only “aspirational.” But if adopted, they could boost transparency, Ioannidis said.
Two of those principles, paraphrased, would ask researchers to:
- Share data, software, workflows, and the specific calculations they used to generate their results. Archive that shared information in open and trusted digital libraries.
- When a scientific paper is considered for publication, journals should conduct a reproducibility check and hold authors to a set of transparency and openness standards, called TOP.
“The current research environment is highly sensitized to these issues,” Ioannidis said. “We now have far more scientists than before who realize that these are major issues and it’s important to have access to data. It’s important to be able to use software that’s been used for scientific work, even run the scripts that have been used to create scientific work.”
