The initiative is similar to onesseen before, such as one that IBM lead earlier, called the
DREAM project (Dialogue on ReverseEngineering Assessment and Methods), but IMPROVER looks specifically at developinga more transparent and robust process for assessing complex scientific datausing a "wisdom of the crowds" approach in which teams of scientists from aroundthe world will compete in a series of scientific challenges. Through thesechallenges, PMI and IBM say, those scientists will "actively contribute to thedevelopment of an innovative method for verification of scientific data andconcepts in systems biology research."
As for the "why?" of thisparticular pairing, IBM has been involved in many healthcare and life-sciencesefforts before—some of them involving systems biology—and PMI is interested inmaking less harmful tobacco products, with an eye toward using systems biologyand computational modeling as a way to predict the health risks of suchproducts. PMI's efforts have been confounded to a large degree, as Dr. Hugh Browne,director of research and development for PMI,notes, by the fact there currently exists no standard
method of verifying the company'sconclusions.
"What started us thinkingin this area is that as we've made significant investments in systems biologyat PMI, we've realized that one of the things science is really good at thesedays is generating huge quantities of data in short periods of time," Browneexplains. "And, if we look at the peer review process, we can see that it wasnever really designed with systems biology or large, complex datasets in mind.It occurred to us that there were other techniques that could be complementaryto peer review to allow the scientific community to get the most out of thedata that is generated and to still have that degree of rigor—complementary topeer review, but beyond what the existing system is able to provide."
From there, he said, PMI set about thinking who it couldpartner with, and the Thomas J. Watson Research Center at IBM quickly hit thetop of the list, as the team there already had experience both with handlingbig data and with crowdsourcing, such as with the DREAM project. The workaround IMPROVER—also known as SBV IMPROVER, with the SBV standing for systemsbiology verification—began officially in 2009.
"We felt that with IBM's capabilities and familiarity withsystems biology, they would be an ideal partner for PMI to think about how wecould move the approach to data like this in a direction where the data can bereviewed and assessed in a way that traditional peer review alone simplydoesn't allow for," Browne says.
"Peer review isn't coming to an end, but it does have itslimitations, especially when it comes to big data, because most people justcan't deal with that complexity on their own or with just a few other people,"notes IBM Research's Dr. Jörg Sprengel. "Whatwe are doing is applicable to a lot of industries, including environmental,animal health and food safety, but we think it's particularly relevant topharma and biotech. That said, there is a lot of interest in some of thoseother market segments, particularly with consumer products like nutrition,cosmetics and veterinary, where they want to make certain claims—particularlyabout health benefits—and need evidence to back those up."
The ultimate goal is to create anindustry standard—a product to help make systems biology-related work moreefficient and useful. In the end, "SBV IMPROVER might bear some resemblance tothings like ISO 9000 accreditations," Browne says, "where there is a clearframework and methodologies published but the actual assessment is outsourcedto a third party. That could help speed time to market."
The collaborative initiative's first challenge, theDiagnostic Signature Challenge, has already been completed after being launchedin March of this year, with members of the global scientific and academiccommunity invited to identify diagnostic signatures in four disease areas:psoriasis, multiple sclerosis, chronic obstructive pulmonary disease and lungcancer. The initial aim of the Diagnostic Signature Challenge was to verifywhether it is possible to extract robust diagnostic signatures for each of thefour diseases under investigation. If it was established that diagnosticsignatures can be identified, the next goal was to identify the best diagnosticsignature for each disease together with its associated discovery algorithm.
The research modality that IMPROVER employs "confrontsscientists with huge challenges to ensure that their conclusions are accurate,robust and capable of translating through to innovative policies, processes andproducts," says Dr. Gustavo Stolovitzky, manager of functional genomics andsystems biology at the IBM Computational Biology Centre. "The outcome of theDiagnostic Signature Challenge shows us that IMPROVER has the potential tosignificantly influence how systems biology can be verified in industrialcontexts in the years to come."
The next challenge, to be launched in the second quarter of2013, will be the Species Translation Challenge.
"With our first challenge, we wanted to start as simple aswe could with something so complex, and confirm that our approach works, sincepeople do all kinds of things looking at data like this, but not usually in asystematic fashion," Sprengel says. "The next steps will be to move to a morequalitative rather than quantitative point, and determine translatability frommodel systems to human cell lines. The next level up from that will be to lookat more predictive goals."
Using a crowdsourcing approach wasimportant in getting beyond peer review constraints but still retaining much ofthe value of what peer review provides, Sprengel and Browne maintain.
"In the first challenge, therewere teams that performed very well in some areas but not as well in others,"Sprengel notes. "Our intention is to get the best overall performers andevaluate what they can offer to try and improve the approaches as much as wecan." "If you look at the 54 teams that submitted entriesfor the first challenge, they used subtly different approaches," Brownecontinues, "and that's the 'wisdom of the crowds' from which you can takeelements of one solution and combine it with another and probably have a bettersolution overall."