When I was a kid in late grade school and high school, I was taught the fundamental tenet of science was reproducibility. If two groups working on opposite sides of the planet or in adjacent labs performed the same experiment, they should get the same results.
If you didn’t get the same results as I did by doing precisely the same thing, something was seriously wrong.
According to University of Bristol’s Marcus Munafò and colleagues, something may indeed be seriously wrong in the state of science, and the problem may be getting dangerously worse.
“Data from many fields suggests reproducibility is lower than is desirable; one analysis estimates that 85 percent of biomedical research efforts are wasted, while 90 percent of respondents to a recent survey in Nature agreed that there is a ‘reproducibility crisis,'” Munafò and colleagues offered in the inaugural issue of Nature Human Behaviour in January. “Whether ‘crisis’ is the appropriate term to describe the current state or trajectory of science is debatable, but accumulated evidence indicates that there is substantial room for improvement with regard to research practices to maximize the efficiency of the research community’s use of the public’s financial investment in research.”
To address these concerns, the researchers offered “a manifesto for reproducible science,” first defining what they believed to be the myriad factors impacting reproducibility and then arguing “for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives.”
It is to this very last category—incentives—that I turn my attention, as I believe that failure here will make any attempts to otherwise alter the scientific landscape meaningless.
In short, we devalue efforts to reproduce previous results to test their validity and scope, declaring instead “Give me innovation or give me death!”
“Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy, the authors suggested. “These incentives ultimately increase the likelihood of false positives in the published literature.”
Despite the anxiety-inducing expansion of the scientific publishing industry, editorial panels will overwhelmingly prefer research that pushes boundaries over that which merely repeats or tests previous findings. And negative results, in particular, are often left in filing cabinets or on shelves, further skewing our ability to interpret scientific theories.
Novelty enhances a journal’s prestige. And even in an era of pay-to-publish journals with little or no editorial oversight, researchers remain incentivized to aim for higher impact journals whenever possible to increase the likelihood of academic employment or tenure, and continued research funding.
(I am not unaware of the culpability of lay and industry publications such as DDNews in promoting innovative over confirmatory news. In our defense, if we can’t find such research, it is difficult to write about it.)
“There will always be incentives for innovative outcomes—those who discover new things will be rewarded more than those who do not,” Munafò and colleagues stressed.
No one, for example, will win a Nobel Prize for fact-checking.
“However, there can also be incentives for efficiency and effectiveness—those who conduct rigorous, transparent and reproducible research could be rewarded more than those who do not,” the authors declared. And the authors offer recommendations on how to do just that.
Ultimately, however, it comes down to money.
Reproducing experiments to reproduce the results is expensive—particularly with clinical trials—and who is willing to pay the cost given the lack of reward?
Seen from a broader perspective, however, the cost of not doing so may be dramatically higher. Science is already hard enough without turning it into a faith-based exploit.
Because of the very innovations at the heart of this discussion, science has always been under siege, both from within and without. And if issues like anti-vaccination campaigns are any indication, the stakes will only get higher.
At the risk of sounding like Chicken Little, we all stand to suffer if we don’t address some of the systemic issues opening the door to rot.
“The key to fostering a robust metascience that evaluates and improves practices is that the stakeholders of science must not embrace the status quo, but instead pursue self-examination continuously for improvement and self-correction of the scientific process itself,” Munafò and colleagues concluded.
Ironically, challenging the status quo is who we are; it is in our DNA. How self-blinkering we would be to make this the one exception to that rule.
You can freely download the Munafò manifesto at: https://www.nature.com/articles/s41562-016-0021
Randall C Willis can be reached by email at willis@ddn-news.com
