An ensemble model of QSAR tools for regulatory risk assessment

Springer Science and Business Media LLC - Tập 8 - Trang 1-9 - 2016
Prachi Pradeep1, Richard J. Povinelli2, Shannon White3, Stephen J. Merrill4
1National Center for Computational Toxicology (ORISE Fellow), US EPA, Research Triangle Park, USA
2Electrical and Computer Engineering Department, Marquette University, Milwaukee, USA
3Lombardi Comprehensive Cancer Center, Georgetown University Medical Center, Washington, USA
4Department of Mathematics, Statistics and Computer Science, Marquette University, Milwaukee, USA

Tóm tắt

Quantitative structure activity relationships (QSARs) are theoretical models that relate a quantitative measure of chemical structure to a physical property or a biological effect. QSAR predictions can be used for chemical risk assessment for protection of human and environmental health, which makes them interesting to regulators, especially in the absence of experimental data. For compatibility with regulatory use, QSAR models should be transparent, reproducible and optimized to minimize the number of false negatives. In silico QSAR tools are gaining wide acceptance as a faster alternative to otherwise time-consuming clinical and animal testing methods. However, different QSAR tools often make conflicting predictions for a given chemical and may also vary in their predictive performance across different chemical datasets. In a regulatory context, conflicting predictions raise interpretation, validation and adequacy concerns. To address these concerns, ensemble learning techniques in the machine learning paradigm can be used to integrate predictions from multiple tools. By leveraging various underlying QSAR algorithms and training datasets, the resulting consensus prediction should yield better overall predictive ability. We present a novel ensemble QSAR model using Bayesian classification. The model allows for varying a cut-off parameter that allows for a selection in the desirable trade-off between model sensitivity and specificity. The predictive performance of the ensemble model is compared with four in silico tools (Toxtree, Lazar, OECD Toolbox, and Danish QSAR) to predict carcinogenicity for a dataset of air toxins (332 chemicals) and a subset of the gold carcinogenic potency database (480 chemicals). Leave-one-out cross validation results show that the ensemble model achieves the best trade-off between sensitivity and specificity (accuracy: 83.8 % and 80.4 %, and balanced accuracy: 80.6 % and 80.8 %) and highest inter-rater agreement [kappa (κ): 0.63 and 0.62] for both the datasets. The ROC curves demonstrate the utility of the cut-off feature in the predictive ability of the ensemble model. This feature provides an additional control to the regulators in grading a chemical based on the severity of the toxic endpoint under study.

Tài liệu tham khảo

Kramer JA, Sagartz JE, Morris DL (2007) The application of discovery toxicology and pathology towards the design of safer pharmaceutical lead candidates. Nat Rev Drug Discov 6:636–649 Coecke S, Pelkonen O, Leite SB, Bernauer U, Bessems JG, Bois FY, Gundert-Remy U, Loizou G, Testai E, Zaldvar JM (2013) Toxicokinetics as a key to the integrated toxicity risk assessment based primarily on non-animal approaches. Toxicol In-vitro 27:1570–1577 Kaplan AV, Baim DS, Smith JJ, Feigal DA, Simons M, Jefferys D, Thomas JF, Kuntz RE, Leon MB (2004) Medical device development from prototype to regulatory approval. Circulation 109(25):3068–3072 Muster W, Breidenbach A, Fischer H, Kirchner S, Mller L, Phler A (2008) Computational toxicology in drug development. Drug Discov Today 13(7):303–310 Dearden JC (2003) In silico prediction of drug toxicity. J Comput Aided Mol Des 17(2–4):119–127 Serafimova R, Gatnik MF, Worth A (2010) Review of qsar models and software tools for predicting genotoxicity and carcinogenicity. Publications Office of the European Union. JRC Scientific and technical reports Jaworska J, Nikolova-Jeliazkova N, Aldenberg T (2005) Review of methods for qsar applicability domain estimation by the training set. Technical report. The European Commission—Joint Research Centre Institute for Health & Consumer Protection-ECVAM Cronin MT, Jaworska JS, Walker JD, Comber MH, Watts CD, Worth AP (2003) Use of QSARs in international decision-making frameworks to predict health effects of chemical substances. Environ Health Perspect 111(10):1391–1401 Hfer T, Gerner I, Gundert-Remy U, Liebsch M, Schulte A, Spielmann H, Vogel R, Wettig K (2004) Animal testing and alternative approaches for the human health risk assessment under the proposed new european chemicals regulation. Arch Toxicol 78(10):549–564 Valerio LG Jr (2011) In silico toxicology models and databases as FDA Critical Path Initiative toolkits. Hum Genomics 5(3):200–207 Zeeman M, Auer CM, Clements RG, Nabholz JV, Boethling RS (1995) US EPA regulatory perspectives on the use of QSAR for new and existing chemical evaluations. SAR QSAR Environ Res 3(3):179–201 Us EPA (2014) Estimation programs interface suite for microsoft windows. United States Environmental Protection Agency, Washington EPA T.E.S.T. http://www.epa.gov/nrmrl/std/qsar/qsar.html. Last accessed Apr 2014 Worth AP (2010) Recent advances in QSAR studies. Springer Netherlands, pp 367–382. Chap. 13: “The role of QSAR methodology in the regulatory assessment of chemicals” Jaworska JS, Comber M, Auer C, Van Leeuwen CJ (2003) Summary of a workshop on regulatory acceptance of (q) sars for human health and environmental endpoints. Environ Health Perspect 111(10):1358–1360 Gleeson MP, Modi S, Bender A, Marchese L, Robinson R, Kirchmair J, Promkatkaew M, Hannongbua S, Glen RC (2012) The challenges involved in modeling toxicity data in silico: a review. Curr Pharm Des 18(9):1266–1291 Zhao C, Boriani E, Chana A, Roncaglioni A, Benfenati E (2008) A new hybrid system of QSAR models for predicting bioconcentration factors (BCF). Chemosphere 73(11):1701–1707 VEGA. http://www.vega-qsar.eu/. Last accessed Apr 2013 Gissi A, Nicolotti O, Carotti A, Gadaleta D, Lombardo A, Benfenat E (2013) Integration of qsar models for bioconcentration suitable for reach. Sci Total Environ 456:325–332 Sanderson DM, Earnshaw CG (1991) Computer prediction of possible toxic action from chemical structure; the DEREK system. Hum Exp Toxicol 10(4):261–273 Lhasa DEREK. http://www.lhasalimited.org/products/derek-nexus.htm. Last accessed Oct 2014 Roberts G, Myatt GJ, Johnson WP, Cross KP, Blower PE (2000) Leadscope: software for exploring large sets of screening data. J Chem Inf Comput Sci 40(6):1302–1314 MultiCASE Inc. http://multicase.com/ Toxtree—Toxic Hazard Estimation by decision tree approach. http://toxtree.sourceforge.net/. Last accessed 2013 Ames BN, McCann J, Yamasaki E (1975) Methods for detecting carcinogens and mutagens with the salmonella/mammalian-microsome mutagenicity test. Mutat Res 31:347–364 Benignia R, Giulianib A (1988) Computer-assisted analysis of interlaboratory ames test variability. J Toxicol Environ Health 25:135–148 Hillebrecht A, Muster W, Brigo A, Kansy M, Weiser T, Singer T (2011) Comparative evaluation of in silico systems for ames test mutagenicity prediction: scope and limitations. Chem Res Toxicol 24(6):843–854 MDL QSAR, MDL information systems. http://www.mdli.com Contrera JF, Kruhlak NL, Matthews EJ, Benz RD (2007) Comparison of MC4PC and MDL-QSAR rodent carcinogenicity predictions and the enhancement of predictive performance by combining QSAR models. Regul Toxicol Pharmacol 49(3):172–182 Valencia A (2010) BioEpisteme—an in silico approach for predicting and understanding the underlying molecular mechanisms contributing to toxicity responses. Toxicol Lett 196(S25):1–48. doi:10.1016/j.toxlet.2010.03.117 Matthews EJ, Kruhlak NL, Benz RD, Contrera JF, Marchant CA, Yang C (2008) Combined use of MC4PC, MDL-QSAR, BioEpisteme, Leadscope PDM, and Derek for Windows software to achieve high-performance, high-confidence, mode of action-based predictions of chemical carcinogenesis in rodents. Toxicol Mech Methods 18(2–3):189–206 Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51:181–207 Svetnik V, Wang T, Tong C, Liaw A, Sheridan RP, Song Q (2005) Boosting: an ensemble learning tool for compound classification and QSAR modeling. J Chem Inf Model 45:786–799 Tan AC, Gilbert D (2003) Ensemble machine learning on gene expression data for cancer classification. Appl Bioinform 2:75–83 Oza NC, Tumer K (2008) Classifier ensembles: select real-world applications. Inf Fusion 9:4–20 Pissurlenkar RR, Khedkar VM, Iyer RP, Coutinho EC (2011) Ensemble QSAR: a QSAR method based on conformational ensembles and metric descriptors. J Comput Chem 32(10):2204–2218 OECD QSAR Toolbox. http://www.qsartoolbox.org/ Danish (Q)SAR Database. http://qsar.food.dtu.dk/ Gold LS (1984) The Carcinogenic Potency Project (CPDB). http://toxnet.nlm.nih.gov/cpdb/. Last accessed Oct 2014 Chemical Carcinogenesis Research Information System (2014) http://toxnet.nlm.nih.gov/cgi-bin/sis/htmlgen?CCRIS. Last accessed Mar 2014 Mohri M, Rostamizadeh A, Talwalkar A (2012) Foundations of machine learning. The MIT Press, Cambridge Kuncheva LI, Rodrguez JJ (2014) A weighted voting framework for classifiers ensembles. Knowl Inf Syst 38:259–275 Valpola H, Honkela A, Karhunen J, Raiko T, Giannakopoulos X, Ilin A, Oja (2001) Bayesian ensemble learning of generative models. Biennial Report of Adaptive Informatics Research Center, Aalto Univ MATLAB R2012a (2012) The MathWorks Inc Chirico N, Gramatica P: Real external predictivity of QSAR models: how to evaluate it? Comparison of different validation criteria and proposal of using the concordance correlation coeffient. J Chem Inf Model 51(9):2320–2335 Keefer CE, Kauffman GW, Gupta RR (2013) Interpretable, probability-based confidence metric for continuous quantitative structureactivity relationship models. J Chem Inf Model 53(2):368–383 Consonni V, Ballabio D, Todeschini R (2010) Evaluation of model predictive ability by external validation techniques. J Chemom 24(3–4):194–201 Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46 Viera AJ, Garrett JM (2005) Understanding interobserver agreement: the kappa statistic. Fam Med 37(5):360–363 Zou KH, OMalley AJ, Mauri L (2007) Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation 115(5):654–657