NEW YORK—Winning sports teams have long inspired business leaders, but now their strategies are influencing pharmaceutical researchers. The Oakland A’s upended baseball recruiting in 2002 by forgoing conventional wisdom for an objective numbers analysis called sabermetrics, made popular by the film “Moneyball.” Inspired by this moneyball approach, the study “A Data-Driven Approach to Predicting Successes and Failures of Clinical Trials,” published in Cell Chemical Biology, has gone beyond conventional wisdom in pharmaceutical research to develop an objective, machine-learning program called PrOCTOR to predict drug toxicity in humans.
“Some of us had recently read the book ‘Moneyball,’ an account of how old wisdom was discarded in favor of coldly assessing how known and overlooked features predict baseball players’ success on the field. We have applied the same philosophy to predicting the outcome of clinical trials—forget what the dogma says, analyze all the information available on clinical trials and the drugs being tested in humans, and find out what works and what does not work,” explains Dr. Olivier Elemento, senior author of the paper, associate director of the Institute for Computational Biomedicine at Weill Cornell Medicine and head of the computational biology group at the Caryl and Israel Englander Institute for Precision Medicine.
Scientists typically turn to a handful of rules-based comparisons of a drug’s molecular structure to bet on whether an untested drug is safe or toxic. But despite this industry convention, nearly one-third of drugs that fail clinical trials do so because of intolerable side effects.
“Rule-based methods have sought to quantify the extent to which new molecules are ‘drug-like’ and have been used extensively to select molecules with predicted low toxicity,” says Elemento. “The Veber rule specifically looks at molecular properties of small molecules that influence their oral bioavailability (a property that is inversely correlated with toxicity). Surprisingly, we found that the commonly used Veber rule as well as other rules (Lipinski, etc.) have very limited ability to discern molecules that are toxic in human clinical trials. People had feelings about certain factors being important in drug toxicity, and there wasn’t much science behind these judgment calls.”
When Elemento and his co-authors crunched the numbers on the conventional rules in a test model, they found that the common “Veber Rule” incorrectly predicted that more than 75 percent of FDA-approved drugs would have been too toxic for clinical trials. Lipinski’s Rule of Five incorrectly judged 73 percent of drugs that failed clinical trials due to toxicity as safe enough to pass.
“This led to a highly predictive, quantitative model (the PrOCTOR score) that predicts accurately whether a small molecule will fail or pass clinical trials in human. Most importantly, the model tells you which features are predictive, and that’s invaluable information because it may help design less toxic drugs from the start,” Elemento tells DDNews. “We are applying the general philosophy of ‘letting the data speak’ and ‘ignore old wisdom, just crunch the numbers’ to all levels of the drug discovery process. We think this approach will tremendously speed up the process of discovery and testing new drugs.”
To create their tool PrOCTOR (Predicting Odds of Clinical Trial Outcomes using Random-forest), the researchers used a decision-tree machine-learning model and tested whether overlooked data might be equally or more important to safety predictions than the conventional structure-based rules. PrOCTOR incorporates data from 48 different features, including descriptors of a drug’s structure such as molecular weight, as well as a host of details about the drug’s targets (the molecules in the body to which drugs bind to be effective).
The researchers trained PrOCTOR on a large dataset of 784 FDA-approved drugs and 100 drugs that failed clinical trials with toxicity concerns; they then tested the model on hundreds of drugs approved in Europe and Japan and on an even larger database of 3,236 drugs not included in PrOCTOR’s training set of data. Overall, PrOCTOR accurately predicted drug toxicity in test models and even flagged approved drugs that were later monitored for reports of serious side effects.
“The PrOCTOR score uses state-of-the-art machine learning (a form of artificial intelligence) to learn which features of small molecules and their mechanisms of action in cells are predictive of clinical trial outcomes. We looked at a very broad set of features, ranging from molecular weight and shape and composition, to where in the body the targets of molecules are expressed, how connected such targets are and how essential they are to cell survival, etc,” says Elemento. “Then we trained the model using a large dataset of clinical trials whose outcome was known—some trials failed for toxicity reasons, and some went all the way to FDA approval. This led to a validated predictive model that can be applied prospectively to new molecules.”
“The PrOCTOR score displays both high sensitivity and high specificity in predicting clinical trial toxicity. The area-under-the-receiver-operator curve (AUC), a widely used measure of prediction accuracy, was 0.82 (maximum achievable value is 1.00). The PrOCTOR score successfully predicted the outcome of other trials not included in the training set, such as those conducted in Europe and Asia. Among FDA-approved drugs, drugs with higher predicted toxicity scores had significantly more frequent reports of serious adverse events, such as death and renal failure, than predicted safe drugs in the OpenFDA resource of drug adverse events. One approved antidiabetic drug, rosiglitazone, was predicted as highly toxic in humans. We subsequently found that this drug has been linked with an elevated risk of heart attack, and was withdrawn from the market in Europe in 2010,” he adds.
However, researchers found that a PrOCTOR score should be assessed in context. Several FDA-approved drugs in the study were flagged as potential failures, but on closer investigation, most of these drugs were life-saving cancer treatments with an understandably high bar for toxic side effects.
The PrOCTOR model worked best when data about the drug’s target was included; however, the authors note that this information is not always available during drug development. Moreover, many drug companies don’t release details about why a particular drug failed a clinical trial.
“For us, the more data the better,” says first author Kaitlyn Gayvert, a PhD student in the Tri-Institutional Computational Biology and Medicine Program, a partnership of Weill Cornell Medicine, Cornell University and Memorial Sloan Kettering Cancer Center. “If better clinical trial data is reported in the future, we’ll be able to make better predictions.”
“Absence of clinical trial reporting means we are not able to learn from trial outcomes. Likewise, mechanisms of drug action in cells are not always well-defined. To address this problem, we are working on artificial intelligence-guided methods to predict drug targets with high accuracy, including off-targets,” Elemento adds.
Because PrOCTOR is a machine-learning-based tool, it provides an opportunity to predict more than just a toxicity score. According to Elemento, “We are working to predict specific types of toxicity, such as liver or cardiac toxicity. Our initial results look promising. If the models are fully validated, they would pave the way for 1) guiding clinical trials by enrolling only patients at low risk of certain toxic events and 2) enhancing post-approval drug surveillance by focusing on specific types of adverse events and toxicity in patients receiving a particular molecule.”
“We believe that artificial intelligence-guided methods such as PrOCTOR will increasingly provide new insights into how to optimally design drugs that are safe and highly efficient. We envision that PrOCTOR will be applied early in the drug development process to help guide optimization of properties that dictate toxicity. PrOCTOR can also be used for increased post-approval drug surveillance for drugs flagged with a high likelihood of toxicity,” Elemento says. “As discussed, we are working on a battery of other methods to help speed up the process of discovering and developing new drugs, learning how to combine them, identify biomarkers of response and starting to partner with pharma companies to test and apply these methods.”
