Users identified challenges in applying GRADE to complex interventions and suggested an extension to GRADE

Guyatt, 2008, GRADE: an emerging consensus on rating quality of evidence and strength of recommendations, BMJ, 336, 924, 10.1136/bmj.39489.470347.AD Balshem, 2011, GRADE guidelines: 3. Rating the quality of evidence, J Clin Epidemiol, 64, 401, 10.1016/j.jclinepi.2010.07.015 Available at http://www.gradeworkinggroup.org. Accessed October 22, 2014. Durrheim, 2010, Modifying the GRADE framework could benefit public health, J Epidemiol Community Health, 64, 387, 10.1136/jech.2009.103226 Rehfuess, 2013, Current experience with applying the GRADE approach to public health interventions: an empirical study, BMC Public Health, 13, 9, 10.1186/1471-2458-13-9 Rehfuess, 2011, GRADE for the advancement of public health, J Epidemiol Community Health, 65, 559, 10.1136/jech.2010.130013 Kelly, 2010, Evidence based public health: a review of the experience of the National Institute of Health and Clinical Excellence (NICE) of developing public health guidance in England, Soc Sci Med, 71, 1056, 10.1016/j.socscimed.2010.06.032 Petticrew, 2011, When are complex interventions “complex”? When are simple interventions “simple”?, Eur J Public Health, 21, 397, 10.1093/eurpub/ckr084 Craig, 2008 Henggeler, 2002 Littell, 2005, Multisystemic therapy for social, emotional, and behavioral problems in youth aged 10-17, Cochrane Database Syst Rev, 4, CD004797 Movsisyan, 2015, Outcomes of complex interventions more frequently result in “low” and “very low” GRADE ratings when compared to the outcomes of simple interventions, J Clin Epidemiol Petticrew, 2015, Time to rethink the systematic review catechism? Moving from “what works” to “what happens”, Syst Rev, 4, 36, 10.1186/s13643-015-0027-1 Squires, 2013, Systematic reviews of complex interventions: framing the review question, J Clin Epidemiol, 66, 1215, 10.1016/j.jclinepi.2013.05.013 Dangour, 2013, Interventions to improve water quality and supply, sanitation and hygiene practices, and their effects on the nutritional status of children, Cochrane Database Syst Rev, 8, CD009382 Rana, 2013, Pharmacological interventions for self-injurious behaviour in adults with intellectual disabilities, Cochrane Database Syst Rev, 4, CD009084 Pope, 2002, Qualitative methods in research on healthcare quality, Qual Saf Health Care, 11, 148, 10.1136/qhc.11.2.148 Weinstein Gale, 2013, Using the framework method for the analysis of qualitative data in multi-disciplinary health research, BMC Med Res Methodol, 13, 117, 10.1186/1471-2288-13-117 Ram, 2008, A methodology for a structured survey of the healthcare literature related to medical device users, Evaluation, 14 Ritchie, 1994, Qualitative data analysis for applied policy research Fletcher-Watson, 2014, Interventions based on the Theory of Mind cognitive model for autism spectrum disorder (ASD), Cochrane Database Syst Rev, 3, CD008785 Langford, 2014, The WHO Health Promoting School framework for improving the health and well-being of students and their academic achievement, Cochrane Database Syst Rev, 4, CD008958 Martin, 2014, Lifestyle interventions for improving school achievement in overweight or obese children and adolescents, Cochrane Database Syst Rev, 3, CD009728 Oono, 2013, Parent-mediated early intervention for young children with autism spectrum disorders (ASD), Cochrane Database Syst Rev, 4, CD009774 Pega, 2013, In-work tax credits for families and their impact on health status in adults, Cochrane Database Syst Rev, 8, CD009963 Barlow, 2012, Group-based parent training programmes for improving parental psychosocial health, Cochrane Database Syst Rev, 6, CD002020 Petrosino, 2013, Scared straight and other juvenile awareness programs for preventing juvenile delinquency: a systematic review, Campbell Syst Rev, 9 Rosendal, 2013, Enhanced care by generalists for functional somatic symptoms and disorders in primary care, Cochrane Database Syst Rev, 10, CD008142 Thomson, 2013, Housing improvements for health and associated socio-economic outcomes, Cochrane Database Syst Rev, 2, CD008657 Winokur, 2014, Kinship care for the safety, permanency, and well-being of children removed from the home for maltreatment, Cochrane Database Syst Rev, 1, CD006546 Hazell, 2013, Tricyclic drugs for depression in children and adolescents, Cochrane Database Syst Rev, 6, CD002317 Lazzerini, 2013, Specially formulated foods for treating children with moderate acute malnutrition in low- and middle-income countries, Cochrane Database Syst Rev, 6, CD009584 Magni, 2013, Fluoxetine versus other types of pharmacotherapy for depression, Cochrane Database Syst Rev, 7, CD004185 Purgato, 2014, Cochrane Database Syst Rev, 4, CD006531 Schoonees, 2013, Ready-to-use therapeutic food for home-based treatment of severe acute malnutrition in children from six months to five years of age, Cochrane Database Syst Rev, 6, CD009000 Wang, 2013, Iron therapy for improving psychomotor development and cognitive function in children under the age of three with iron deficiency anaemia, Cochrane Database Syst Rev, 6, CD001444 Rothbart, 2013, Pharmacotherapy for trichotillomania, Cochrane Database Syst Rev, 11, CD007662 Wijkstra, 2013, Pharmacological treatment for psychotic depression, Cochrane Database Syst Rev, 11, CD004044 Williams, 2013, Selective serotonin reuptake inhibitors (SSRIs) for autism spectrum disorders (ASD), Cochrane Database Syst Rev, 8, CD004677 Sterne Montgomery, 2013, The Oxford Implementation Index: a new tool for incorporating implementation data into systematic reviews and meta-analyses, J Clin Epidemiol, 66, 874, 10.1016/j.jclinepi.2013.03.006 Hasson, 2010, Systematic evaluation of implementation fidelity of complex interventions in health and social care, Implement Sci, 5, 67, 10.1186/1748-5908-5-67 Tugwell, 2013, Complex interventions—how should systematic reviews of their impact differ from reviews of simple or complicated interventions?, J Clin Epidemiol, 66, 1195, 10.1016/j.jclinepi.2013.09.003 Meader, 2014, A checklist designed to aid consistency and reproducibility of GRADE assessments: development and pilot validation, Syst Rev, 3, 82, 10.1186/2046-4053-3-82 Andrews, 2013, GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength, J Clin Epidemiol, 66, 726, 10.1016/j.jclinepi.2013.02.003 Djulbegovic, 2015, Quality of evidence is a key determinant for making a strong GRADE guidelines recommendation, J Clin Epidemiol, 68, 727, 10.1016/j.jclinepi.2014.12.015 Nasser, 2015, Strength of recommendations in WHO guidelines using GRADE was associated with uptake in national policy, J Clin Epidemiol, 68, 703, 10.1016/j.jclinepi.2014.11.006 McPheeters, 2015 Bonell, 2006, Should structural interventions be evaluated using RCTs? the case of HIV prevention, Soc Sci Med, 63, 1135, 10.1016/j.socscimed.2006.03.026 Victora, 2004, Evidence-based public health: moving beyond randomized trials, Am J Public Health, 94, 400, 10.2105/AJPH.94.3.400 Glenton, 2013, Nonrandomized studies are not always found even when selection criteria for health systems intervention reviews include them: a methodological study, J Clin Epidemiol, 66, 367, 10.1016/j.jclinepi.2012.11.009 Ioannidis, 2001, Comparison of evidence of treatment effects in randomized and nonrandomized studies, JAMA, 286, 821, 10.1001/jama.286.7.821 Clair, 2014, Examining the internal validity and statistical precision of the comparative interrupted time series design by comparison with a randomized experiment, Am J Eval, 12, 1 Fretheim, 2013, Interrupted time-series analysis yielded an effect estimate concordant with the cluster-randomized controlled trial result, J Clin Epidemiol, 66, 883, 10.1016/j.jclinepi.2013.03.016 Golder, 2011, Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview, PLos Med, 8, e1001026, 10.1371/journal.pmed.1001026 Golder, 2013, Comparison of pooled risk estimates for adverse effects from different observational study designs: methodological overview, PLoS One, 8, e71813, 10.1371/journal.pone.0071813 Guise, 2014 Howick, 2009, The evolution of evidence hierarchies: what can Bradford Hill's 'guidelines for causation' contribute?, J R Soc Med, 102, 186, 10.1258/jrsm.2009.090020 Anderson, 2013, Investigating complexity in systematic reviews of interventions by using a spectrum of methods, J Clin Epidemiol, 66, 1223, 10.1016/j.jclinepi.2013.06.014 Petticrew, 2013, Synthesizing evidence on complex interventions: how meta-analytical, qualitative, and mixed-method approaches can contribute, J Clin Epidemiol, 66, 1230, 10.1016/j.jclinepi.2013.06.005 Boutron, 2008, Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration, Ann Intern Med, 148, 295, 10.7326/0003-4819-148-4-200802190-00008 Montgomery, 2013, Protocol for CONSORT-SPI: an extension for social and psychological interventions, Implement Sci, 8, 1, 10.1186/1748-5908-8-99 Creswell, 2000, Determining validity in qualitative inquiry, Theory Pract, 39, 124, 10.1207/s15430421tip3903_2 Daly, 2007, A hierarchy of evidence for assessing qualitative health research, J Clin Epidemiol, 60, 43, 10.1016/j.jclinepi.2006.03.014 Shih, 2008, Comparing response rates in e-mail and paper surveys: a meta-analysis, Educ Res Rev, 4, 26, 10.1016/j.edurev.2008.01.003 Teunis, 2015, Do corresponding authors take responsibility for their work? A covert survey, Clin Orthop Relat Res, 473, 729, 10.1007/s11999-014-3868-3 2011