An ensemble model of QSAR tools for regulatory risk assessment
Tóm tắt
Quantitative structure activity relationships (QSARs) are theoretical models that relate a quantitative measure of chemical structure to a physical property or a biological effect. QSAR predictions can be used for chemical risk assessment for protection of human and environmental health, which makes them interesting to regulators, especially in the absence of experimental data. For compatibility with regulatory use, QSAR models should be transparent, reproducible and optimized to minimize the number of false negatives. In silico QSAR tools are gaining wide acceptance as a faster alternative to otherwise time-consuming clinical and animal testing methods. However, different QSAR tools often make conflicting predictions for a given chemical and may also vary in their predictive performance across different chemical datasets. In a regulatory context, conflicting predictions raise interpretation, validation and adequacy concerns. To address these concerns, ensemble learning techniques in the machine learning paradigm can be used to integrate predictions from multiple tools. By leveraging various underlying QSAR algorithms and training datasets, the resulting consensus prediction should yield better overall predictive ability. We present a novel ensemble QSAR model using Bayesian classification. The model allows for varying a cut-off parameter that allows for a selection in the desirable trade-off between model sensitivity and specificity. The predictive performance of the ensemble model is compared with four in silico tools (Toxtree, Lazar, OECD Toolbox, and Danish QSAR) to predict carcinogenicity for a dataset of air toxins (332 chemicals) and a subset of the gold carcinogenic potency database (480 chemicals). Leave-one-out cross validation results show that the ensemble model achieves the best trade-off between sensitivity and specificity (accuracy: 83.8 % and 80.4 %, and balanced accuracy: 80.6 % and 80.8 %) and highest inter-rater agreement [kappa (κ): 0.63 and 0.62] for both the datasets. The ROC curves demonstrate the utility of the cut-off feature in the predictive ability of the ensemble model. This feature provides an additional control to the regulators in grading a chemical based on the severity of the toxic endpoint under study.
Tài liệu tham khảo
Kramer JA, Sagartz JE, Morris DL (2007) The application of discovery toxicology and pathology towards the design of safer pharmaceutical lead candidates. Nat Rev Drug Discov 6:636–649
Coecke S, Pelkonen O, Leite SB, Bernauer U, Bessems JG, Bois FY, Gundert-Remy U, Loizou G, Testai E, Zaldvar JM (2013) Toxicokinetics as a key to the integrated toxicity risk assessment based primarily on non-animal approaches. Toxicol In-vitro 27:1570–1577
Kaplan AV, Baim DS, Smith JJ, Feigal DA, Simons M, Jefferys D, Thomas JF, Kuntz RE, Leon MB (2004) Medical device development from prototype to regulatory approval. Circulation 109(25):3068–3072
Muster W, Breidenbach A, Fischer H, Kirchner S, Mller L, Phler A (2008) Computational toxicology in drug development. Drug Discov Today 13(7):303–310
Dearden JC (2003) In silico prediction of drug toxicity. J Comput Aided Mol Des 17(2–4):119–127
Serafimova R, Gatnik MF, Worth A (2010) Review of qsar models and software tools for predicting genotoxicity and carcinogenicity. Publications Office of the European Union. JRC Scientific and technical reports
Jaworska J, Nikolova-Jeliazkova N, Aldenberg T (2005) Review of methods for qsar applicability domain estimation by the training set. Technical report. The European Commission—Joint Research Centre Institute for Health & Consumer Protection-ECVAM
Cronin MT, Jaworska JS, Walker JD, Comber MH, Watts CD, Worth AP (2003) Use of QSARs in international decision-making frameworks to predict health effects of chemical substances. Environ Health Perspect 111(10):1391–1401
Hfer T, Gerner I, Gundert-Remy U, Liebsch M, Schulte A, Spielmann H, Vogel R, Wettig K (2004) Animal testing and alternative approaches for the human health risk assessment under the proposed new european chemicals regulation. Arch Toxicol 78(10):549–564
Valerio LG Jr (2011) In silico toxicology models and databases as FDA Critical Path Initiative toolkits. Hum Genomics 5(3):200–207
Zeeman M, Auer CM, Clements RG, Nabholz JV, Boethling RS (1995) US EPA regulatory perspectives on the use of QSAR for new and existing chemical evaluations. SAR QSAR Environ Res 3(3):179–201
Us EPA (2014) Estimation programs interface suite for microsoft windows. United States Environmental Protection Agency, Washington
EPA T.E.S.T. http://www.epa.gov/nrmrl/std/qsar/qsar.html. Last accessed Apr 2014
Worth AP (2010) Recent advances in QSAR studies. Springer Netherlands, pp 367–382. Chap. 13: “The role of QSAR methodology in the regulatory assessment of chemicals”
Jaworska JS, Comber M, Auer C, Van Leeuwen CJ (2003) Summary of a workshop on regulatory acceptance of (q) sars for human health and environmental endpoints. Environ Health Perspect 111(10):1358–1360
Gleeson MP, Modi S, Bender A, Marchese L, Robinson R, Kirchmair J, Promkatkaew M, Hannongbua S, Glen RC (2012) The challenges involved in modeling toxicity data in silico: a review. Curr Pharm Des 18(9):1266–1291
Zhao C, Boriani E, Chana A, Roncaglioni A, Benfenati E (2008) A new hybrid system of QSAR models for predicting bioconcentration factors (BCF). Chemosphere 73(11):1701–1707
VEGA. http://www.vega-qsar.eu/. Last accessed Apr 2013
Gissi A, Nicolotti O, Carotti A, Gadaleta D, Lombardo A, Benfenat E (2013) Integration of qsar models for bioconcentration suitable for reach. Sci Total Environ 456:325–332
Sanderson DM, Earnshaw CG (1991) Computer prediction of possible toxic action from chemical structure; the DEREK system. Hum Exp Toxicol 10(4):261–273
Lhasa DEREK. http://www.lhasalimited.org/products/derek-nexus.htm. Last accessed Oct 2014
Roberts G, Myatt GJ, Johnson WP, Cross KP, Blower PE (2000) Leadscope: software for exploring large sets of screening data. J Chem Inf Comput Sci 40(6):1302–1314
MultiCASE Inc. http://multicase.com/
Toxtree—Toxic Hazard Estimation by decision tree approach. http://toxtree.sourceforge.net/. Last accessed 2013
Ames BN, McCann J, Yamasaki E (1975) Methods for detecting carcinogens and mutagens with the salmonella/mammalian-microsome mutagenicity test. Mutat Res 31:347–364
Benignia R, Giulianib A (1988) Computer-assisted analysis of interlaboratory ames test variability. J Toxicol Environ Health 25:135–148
Hillebrecht A, Muster W, Brigo A, Kansy M, Weiser T, Singer T (2011) Comparative evaluation of in silico systems for ames test mutagenicity prediction: scope and limitations. Chem Res Toxicol 24(6):843–854
MDL QSAR, MDL information systems. http://www.mdli.com
Contrera JF, Kruhlak NL, Matthews EJ, Benz RD (2007) Comparison of MC4PC and MDL-QSAR rodent carcinogenicity predictions and the enhancement of predictive performance by combining QSAR models. Regul Toxicol Pharmacol 49(3):172–182
Valencia A (2010) BioEpisteme—an in silico approach for predicting and understanding the underlying molecular mechanisms contributing to toxicity responses. Toxicol Lett 196(S25):1–48. doi:10.1016/j.toxlet.2010.03.117
Matthews EJ, Kruhlak NL, Benz RD, Contrera JF, Marchant CA, Yang C (2008) Combined use of MC4PC, MDL-QSAR, BioEpisteme, Leadscope PDM, and Derek for Windows software to achieve high-performance, high-confidence, mode of action-based predictions of chemical carcinogenesis in rodents. Toxicol Mech Methods 18(2–3):189–206
Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51:181–207
Svetnik V, Wang T, Tong C, Liaw A, Sheridan RP, Song Q (2005) Boosting: an ensemble learning tool for compound classification and QSAR modeling. J Chem Inf Model 45:786–799
Tan AC, Gilbert D (2003) Ensemble machine learning on gene expression data for cancer classification. Appl Bioinform 2:75–83
Oza NC, Tumer K (2008) Classifier ensembles: select real-world applications. Inf Fusion 9:4–20
Pissurlenkar RR, Khedkar VM, Iyer RP, Coutinho EC (2011) Ensemble QSAR: a QSAR method based on conformational ensembles and metric descriptors. J Comput Chem 32(10):2204–2218
OECD QSAR Toolbox. http://www.qsartoolbox.org/
Danish (Q)SAR Database. http://qsar.food.dtu.dk/
Gold LS (1984) The Carcinogenic Potency Project (CPDB). http://toxnet.nlm.nih.gov/cpdb/. Last accessed Oct 2014
Chemical Carcinogenesis Research Information System (2014) http://toxnet.nlm.nih.gov/cgi-bin/sis/htmlgen?CCRIS. Last accessed Mar 2014
Mohri M, Rostamizadeh A, Talwalkar A (2012) Foundations of machine learning. The MIT Press, Cambridge
Kuncheva LI, Rodrguez JJ (2014) A weighted voting framework for classifiers ensembles. Knowl Inf Syst 38:259–275
Valpola H, Honkela A, Karhunen J, Raiko T, Giannakopoulos X, Ilin A, Oja (2001) Bayesian ensemble learning of generative models. Biennial Report of Adaptive Informatics Research Center, Aalto Univ
MATLAB R2012a (2012) The MathWorks Inc
Chirico N, Gramatica P: Real external predictivity of QSAR models: how to evaluate it? Comparison of different validation criteria and proposal of using the concordance correlation coeffient. J Chem Inf Model 51(9):2320–2335
Keefer CE, Kauffman GW, Gupta RR (2013) Interpretable, probability-based confidence metric for continuous quantitative structureactivity relationship models. J Chem Inf Model 53(2):368–383
Consonni V, Ballabio D, Todeschini R (2010) Evaluation of model predictive ability by external validation techniques. J Chemom 24(3–4):194–201
Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46
Viera AJ, Garrett JM (2005) Understanding interobserver agreement: the kappa statistic. Fam Med 37(5):360–363
Zou KH, OMalley AJ, Mauri L (2007) Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation 115(5):654–657