Room for IMPROVER
NEW YORK—It may not be the oddest couple to tackle better ways of handling complex life-science data, but New York City-based tobacco industry giant Philip Morris International (PMI) and Armonk-N.Y.-based computing giant IBM certainly make for a novel-seeming pair as they work toward the creation of their Industrial Methodology for Process Verification in Research (IMPROVER).
The initiative is similar to ones seen before, such as one that IBM lead earlier, called the
DREAM project (Dialogue on Reverse Engineering Assessment and Methods), but IMPROVER looks specifically at developing a more transparent and robust process for assessing complex scientific data using a “wisdom of the crowds” approach in which teams of scientists from around the world will compete in a series of scientific challenges. Through these challenges, PMI and IBM say, those scientists will “actively contribute to the development of an innovative method for verification of scientific data and concepts in systems biology research.”
As for the “why?” of this particular pairing, IBM has been involved in many healthcare and life-sciences efforts before—some of them involving systems biology—and PMI is interested in making less harmful tobacco products, with an eye toward using systems biology and computational modeling as a way to predict the health risks of such products. PMI’s efforts have been confounded to a large degree, as Dr. Hugh Browne, director of research and development for PMI, notes, by the fact there currently exists no standard
method of verifying the company’s conclusions.
“What started us thinking in this area is that as we’ve made significant investments in systems biology at PMI, we’ve realized that one of the things science is really good at these days is generating huge quantities of data in short periods of time,” Browne explains. “And, if we look at the peer review process, we can see that it was never really designed with systems biology or large, complex datasets in mind. It occurred to us that there were other techniques that could be complementary to peer review to allow the scientific community to get the most out of the data that is generated and to still have that degree of rigor—complementary to peer review, but beyond what the existing system is able to provide.”
From there, he said, PMI set about thinking who it could partner with, and the Thomas J. Watson Research Center at IBM quickly hit the top of the list, as the team there already had experience both with handling big data and with crowdsourcing, such as with the DREAM project. The work around IMPROVER—also known as SBV IMPROVER, with the SBV standing for systems biology verification—began officially in 2009.
“We felt that with IBM’s capabilities and familiarity with systems biology, they would be an ideal partner for PMI to think about how we could move the approach to data like this in a direction where the data can be reviewed and assessed in a way that traditional peer review alone simply doesn’t allow for,” Browne says.
“Peer review isn’t coming to an end, but it does have its limitations, especially when it comes to big data, because most people just can’t deal with that complexity on their own or with just a few other people,” notes IBM Research’s Dr. Jörg Sprengel. “What we are doing is applicable to a lot of industries, including environmental, animal health and food safety, but we think it’s particularly relevant to pharma and biotech. That said, there is a lot of interest in some of those other market segments, particularly with consumer products like nutrition, cosmetics and veterinary, where they want to make certain claims—particularly about health benefits—and need evidence to back those up.”
The ultimate goal is to create an industry standard—a product to help make systems biology-related work more efficient and useful. In the end, “SBV IMPROVER might bear some resemblance to things like ISO 9000 accreditations,” Browne says, “where there is a clear framework and methodologies published but the actual assessment is outsourced to a third party. That could help speed time to market.”
The collaborative initiative’s first challenge, the Diagnostic Signature Challenge, has already been completed after being launched in March of this year, with members of the global scientific and academic community invited to identify diagnostic signatures in four disease areas: psoriasis, multiple sclerosis, chronic obstructive pulmonary disease and lung cancer. The initial aim of the Diagnostic Signature Challenge was to verify whether it is possible to extract robust diagnostic signatures for each of the four diseases under investigation. If it was established that diagnostic signatures can be identified, the next goal was to identify the best diagnostic signature for each disease together with its associated discovery algorithm.
The research modality that IMPROVER employs “confronts scientists with huge challenges to ensure that their conclusions are accurate, robust and capable of translating through to innovative policies, processes and products,” says Dr. Gustavo Stolovitzky, manager of functional genomics and systems biology at the IBM Computational Biology Centre. “The outcome of the Diagnostic Signature Challenge shows us that IMPROVER has the potential to significantly influence how systems biology can be verified in industrial contexts in the years to come.”
The next challenge, to be launched in the second quarter of 2013, will be the Species Translation Challenge.
“With our first challenge, we wanted to start as simple as we could with something so complex, and confirm that our approach works, since people do all kinds of things looking at data like this, but not usually in a systematic fashion,” Sprengel says. “The next steps will be to move to a more qualitative rather than quantitative point, and determine translatability from model systems to human cell lines. The next level up from that will be to look at more predictive goals.”
Using a crowdsourcing approach was important in getting beyond peer review constraints but still retaining much of the value of what peer review provides, Sprengel and Browne maintain.
“In the first challenge, there were teams that performed very well in some areas but not as well in others,” Sprengel notes. “Our intention is to get the best overall performers and evaluate what they can offer to try and improve the approaches as much as we can.” “If you look at the 54 teams that submitted entries for the first challenge, they used subtly different approaches,” Browne continues, “and that’s the ‘wisdom of the crowds’ from which you can take elements of one solution and combine it with another and probably have a better solution overall.”