The life science DREAM team
YORKTOWN HEIGHTS, N.Y.—In the case of amyotrophic lateral sclerosis (ALS), why do some patients—such as renowned baseball player Lou Gehrig—die quickly, while others—Stephen Hawking comes to mind—survive for many years? This is one of the questions that Dialogue for Reverse Engineering Assessment and Methods (DREAM) Project leader and founder Gustavo Stolovitzky hopes can be answered in the coming weeks by “the wisdom of crowds.”
Established in 2006 by the IBM Computational Biology Center and the MAGNet National Center for Biomedical Computing at Columbia University, DREAM’s main objective is “to catalyze the interaction between experiment and theory in the area of cellular network inference and quantitative model building in systems biology.” The initiative’s mission, according to its website, is “to assess how well we are describing the networks of interacting molecules that underlie biological systems … and how can we know how well we are predicting the outcome of previously unseen experiments from our models?”
Three of the four challenges for this year’s DREAM 7 challenge call for the development of informatics tools and methods that will support ongoing efforts to treat cancer and ALS. Stolovitzky, a Yale University Ph.D. and manager of functional genomics and systems biology at IBM, notes that for this year’s challenges, DREAM organizers are partnering with the National Cancer Institute, Sage Bionetworks and Prize4Life, a not-for-profit organization that supports efforts to discover treatments for ALS.
The NCI-DREAM Drug Sensitivity Prediction Challenge will focus on using genomic information to build models that can estimate the sensitivity of cancer cell lines to a set of small-molecule compounds—both alone and in combination. The goal is to understand how well computational analysis of ‘omics data—including proteomics data, SNP data, gene expression data and drug dose response data—can be used to predict drug activity in cell lines, and ultimately to select the best treatments for patients based on genetic profiling of tumors, Stolovitzky explains. The best-performing team will have the opportunity to publish its results in Nature Biotechnology.
The second challenge for the year, organized with Prize4Life, will attempt to develop methods that can predict the future progression of ALS. Prize4Life is collecting data from ALS patients and, using this data, the DREAM team will use combinations of mathematical and statistical models to find a combination that defines fast and slow progression of the disease. Emerging from this data may be chemical differences that are at the causative root of ALS. Finally, the team will try to determine what drugs would work best in each patient's case, Stolovitzky says. Prize4Life is offering a $25,000 prize for the winning submission in this category.
The third challenge, created in collaboration with Sage Bionetworks, aims to develop algorithms for predicting breast cancer survival. The best performer in this category will have a paper published in Science Translational Medicine.
The final challenge is “pure informatics” and focuses on network topology and parameter inference. Participants are expected to develop optimization methods that accurately estimate parameters, predict outcomes of perturbations and rewire networks in systems biology network models. This challenge is meant to help researchers “understand how to do experimental design when it comes to reconstruction of gene regulatory networks, and understand how to create mathematical models, including what parameters to use,” Stolovitzky says.
Stolovitzky notes that in DREAM 1, he “created the scaffolding and did all the scoring.” The contest has grown since, and now includes a number of collaborators to help him with the various evaluations. “In 7,” he observes, “results will be scrutinized as part of a peer-review process.” For example, the journals that have agreed to publish results from two of the challenges—Nature Biotech and Science Translational Medicine—will provide what Stolovitzky refers to as “challenge-assisted peer review.” He notes that DREAM 7 has reached out to participants involved in previous challenges as well as individuals who attend conferences on related subjects.
“We are as inclusive as possible,” he says. “The more the merrier.”
As its centerpiece, the DREAM process relies upon the wisdom of crowds, a process based on the title of a book published in 2004 and written by James Surowiecki about the aggregation of information in groups and resulting in decisions that are often better than could have been made by any single member of the group. Stolovitzky is a believer, arguing that an aggregate solution is better than any single solution.
“Everyone knows a little bit about something, but knowledge is diffused among many people. We distribute questions in such a way that participants compete to provide their own important piece of the truth,” he states.
He claims that aggregating the best results from teams in each of the categories from previous challenges has resulted in improved methods and better results.
“We have verified this in challenge after challenge,” he says.
While the timelines haven’t yet been finalized, it’s likely that most submissions will be due by mid-October, although entries for some challenges could be due as early as Oct. 1, Stolovitzky says. The winning entries from each challenge will be presented at the DREAM 7 conference to be held in San Francisco Nov. 12 to 14.
Prize4Life and ALS
Prize4Life was founded in 2006 by a group of Harvard Business School students when one of them, Avichai Kremer, then 29, was diagnosed with ALS. After his diagnosis, Kremer and his colleagues decided to pilot an innovative new way to accelerate ALS research.
Prize4Life is a results-oriented nonprofit organization founded to accelerate ALS research by offering substantial prizes to scientists who solve the most critical scientific problems preventing the discovery of an effective ALS treatment. In a statement to ddn, Dr. Neta Zach, the organization’s scientific director, said: “Prize4Life wants to generate new solutions and breakthroughs in ALS. One way to do this is to introduce new minds to the research of ALS. For this challenge, we are appealing to computational savants, many of whom don’t often have the opportunity to use their skills to generate meaningful change in human health, let alone a terrible disease like ALS. While the potential for Big Data to impact health is much discussed, ALS patients, with their short lifespan, need to see these benefits as soon as possible. We are proud to be one of the first initiatives to actually develop and launch such a challenge and hope that others will follow.”
The new Prize4Life Challenge is based on the PRO-ACT database, which upon completion will contain clinical data for more than 7,500 ALS patients from clinical trials by companies including Sanofi, Teva and Novartis. The entire PRO-ACT database will be available for research purposes by year’s end, and the ALS Prediction Prize will use a subset of this data.
Ultimately, it is expected that the solution resulting from this challenge will improve disease prediction and lead to more accurate methods of forecasting progression earlier on in the course of the disease.
DREAM 7 and breast cancer survival
Based on initial promising clinical results, computational approaches to infer molecular predictors of cancer clinical phenotypes are one of the most active areas of research in both industrial and academic institutions, leading to a flood of published reports of signatures predictive of cancer phenotypes, notes Dr. Adam Margolin, director of computational biology at Sage Bionetworks. To evaluate such predictors, the Sage/DREAM Breast Cancer Prognosis Challenge will create a community-based effort to provide an unbiased assessment of models and methodologies for the prediction of breast cancer survival. A common dataset containing gene expression, copy number, and clinical information for 2,000 breast cancer samples will be provided for participants to build models to predict survival time (median 10 years, also provided for all samples). A novel dataset of 350 samples will be generated at the end of the challenge and used to provide a final, unbiased score for each model. More than 200 participants from 20 different countries have already signed up for the challenge.
The journal Science Translational Medicine (STM) has agreed for the best performing individual or team in the final evaluation (using the newly generated data) to publish their results as a “prize” for best performance provided its score is better than the score of a pre-defined baseline set of models. STM representatives agreed that having an evaluation committee re-run and compare all models in a transparent environment can serve the role of challenge-assisted peer review, allowing the results from the winning individual or team to be published without additional review.