Waste not, want not

WASHINGTON, D.C.—Aimed at accelerating the discovery of life-saving diagnostics, therapies and cures, while saving time, money and precious resources, three economists writing in PLOS Biology contend that in the United States alone, an estimated $28 billion annually is spent on preclinical research that is not reproducible.

Published June 9, “The Economics of Reproducibility in Preclinical Research,” authored by Global Biological Standards Institute (GBSI) President Leonard P. Freedman and leading economists Iain M. Cockburn and Timothy S. Simcoe, is the result of a yearlong study to examine the causes of irreproducibility in life-sciences research and quantify the cumulative cost.

The solution to the dilemma, the authors write, is to design a framework for new best practices guidelines aimed at improving preclinical research quality that would ultimately lead to improved reproducibility rates and faster discovery of effective therapies. This new, adopted protocol in life-sciences research would include a close scrutiny of funding requests before any agency or institution opens its proverbial wallet.

Best practices “are those steps, refined over years of research experience, that experts agree should be routinely observed throughout the research cycle from the idea for the study all the way through to publication,” Freedman says.

“In preclinical research, there are only two formal ANSI/ISO approved standards (one for cell authentication and one for anthrax) and no community-recognized best practices,” he adds.

In the larger picture, about $115 billion in the United States is estimated to be spent annually on life-sciences research, about half of which is spent on preclinical work, Freedman says. Of that preclinical research, a conservative estimate is that 50 percent of such research is not reproducible.

Major sources of this waste are errors in biological reagents and reference materials, followed by faulty study design, data analysis and reporting and laboratory protocols, the PLOS study states.

“We are calling for a fundamental shift from acceptance of an avoidable problem to the adoption of compelling best practices that support better research and better outcomes,” Freedman states.

Critics of the PLOS article claim the costs cited are exaggerated, but Freedman, Cockburn and Simcoe stand behind their numbers.

“Our primary goal here is not to pinpoint the exact reproducibility rate, but rather to identify root causes of the problem and develop a framework to address the highest priorities,” Freedman says. “Based on examples from within the life sciences, application of economic theory and review of lessons learned from other industries, we conclude that community-developed best practices and standards must play a central role in improving reproducibility going forward.”

Cockburn adds, “The problem of reproducibility has been widely discussed. To develop our estimate of the current reproducibility rate for preclinical research, we reviewed publicly available data from government sources, industry and analyst reports and scientific articles.”

As Simcoe notes, “While false positives are an inevitable part of scientific research, our study shows that the current level of irreproducibility in preclinical research is very costly. By categorizing the root causes of irreproducibility and estimating their relative importance, we can prioritize potential solutions and, ultimately, increase the overall return on public and private investments in research and development.”

The PLOS article authors have aligned with 21st Century Cures, a congressional initiative underway to update laws to support finding new and improved medicines and diagnostics, and keeping America competitive in healthcare and biotech in the 21st century.

Freedman told DDNews that the impetus for researching reproducibility has come from greater awareness on pressures from researchers.

“There is also a greater public awareness to the issue,” he says. “While individual, flashy and fraudulent papers have gotten a lot of media attention, scientific misconduct is actually very rare. The issues around irreproducibility—poorly validated reagents, flawed statistical analyses, for example—are more mundane, but ultimately incredibly serious.”

Feedback received from the PLOS article has been varied, ranging from “very supportive of the understanding that reproducibility is a cornerstone of scientific research to ‘you are exaggerating the numbers,'” Freedman says. “Ultimately, the hope is that the dollar values calculated in the study will help focus attention on areas where improvements will yield a big bang for the buck.”

Modest investments “in standards and best practices, for example as they relate to biological reagents such as validated cell lines and antibodies, will yield more reproducible results in published studies and, in turn, substantial returns on funding investments,” Freedman says.

Already, he notes, more than 450 cell biologists have completed their Cell Culture and Authentication Survey to help gain better understanding of and overcome barriers to cell authentication and issues with irreproducibility.

“With more than 100 stakeholders already signed on, our ‘authenticate’ campaign is raising awareness among researchers about the importance of cell authentication and the easy and low cost tests that are available—and the GBSI is working on ways to incentivize routine cell authentication practice,” he says.

“We are also working to improve the proficiency of researchers around best practices for quality research and ensure greater accountability to funders for their investments,” Freedman says. “Plans include an online community to provide coaching and mentoring, as well as online courses with the most up-to-date information in quality research practice.”