Designing evaluation studies to optimally inform policy: what factors do policy-makers in China consider when making resource allocation decisions on healthcare worker training programmes?
Tóm tắt
In light of the gap in evidence to inform future resource allocation decisions about healthcare provider (HCP) training in low- and middle-income countries (LMICs), and the considerable donor investments being made towards training interventions, evaluation studies that are optimally designed to inform local policy-makers are needed. The aim of our study is to understand what features of HCP training evaluation studies are important for decision-making by policy-makers in LMICs. We investigate the extent to which evaluations based on the widely used Kirkpatrick model – focusing on direct outcomes of training, namely reaction of trainees, learning, behaviour change and improvements in programmatic health indicators – align with policy-makers’ evidence needs for resource allocation decisions. We use China as a case study where resource allocation decisions about potential scale-up (using domestic funding) are being made about an externally funded pilot HCP training programme. Qualitative data were collected from high-level officials involved in resource allocation at the national and provincial level in China through ten face-to-face, in-depth interviews and two focus group discussions consisting of ten participants each. Data were analysed manually using an interpretive thematic analysis approach. Our study indicates that Chinese officials not only consider information about the direct outcomes of a training programme, as captured in the Kirkpatrick model, but also need information on the resources required to implement the training, the wider or indirect impacts of training, and the sustainability and scalability to other settings within the country. In addition to considering findings presented in evaluation studies, we found that Chinese policy-makers pay close attention to whether the evaluations were robust and to the composition of the evaluation team. Our qualitative study indicates that training programme evaluations that focus narrowly on direct training outcomes may not provide sufficient information for policy-makers to make decisions on future training programmes. Based on our findings, we have developed an evidence-based framework, which incorporates but expands beyond the Kirkpatrick model, to provide conceptual and practical guidance that aids in the design of training programme evaluations better suited to meet the information needs of policy-makers and to inform policy decisions.
Tài liệu tham khảo
Lindblom C, Cohen D. Usable Knowledge: Social Science and Social Problem Solving. New Haven, CT: Yale University Press; 1979.
Hyder AA, Corluka A, Winch PJ, El-Shinnawy A, Ghassany H, Malekafzali H, Lim MK, Mfutso-Bengo J, Segura E, Ghaffar A. National policy-makers speak out: are researchers giving them what they need? Health Policy Plan. 2011;26:73–82.
Weiss C. The many meanings of research utilization. Public Adm Rev. 1979;39:426–31.
Hawkins B, Parkhurst J. The ‘good governance' of evidence in health policy. Evid Policy. 2016;12:575–92.
Aaserud M, Lewin S, Innvaer S. Translating research into policy and practice in developing countries: a case study of magnesium sulphate for pre-eclampsia. BMC Health Serv Res. 2005;5:68.
Albert M, Fretheim A, Maiga D. Factors influencing the utilization of research findings by health policy-makers in a developing country: the selection of Mali’s essential medicines. Health Res Policy Syst. 2007;5:2.
Hennik M, Stephenson R. Using research to inform health policy: barriers and strategies in developing countries. J Health Commun. 2005;10:163–80.
Trostle J, Bronfman M, Langer A. How do researchers influence decision-makers? Case studies of Mexican policies. Health Policy Plan. 1999;14:103–14.
Lavis JDH, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;10:35–48.
Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers' perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7:239–44.
Dobbins M, Ciliska D, Cockerill R, Barnsley J, DiCenso A. A framework for the dissemination and utilization of research for health-care policy and practice. Online J Knowl Synth Nurs. 2002;9:7.
Kouri D. Introductory Module: Introduction to Decision Theory and Practice. Saskatoon: HEALNet; 1997.
Hanney S, Gonzalez-Block M, Buxton M, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1:2.
Schneider A, Ingram H. Policy Design for Democracy. Lawrence, KS: University Press of Kansas; 1997.
Beaglehole R, Dal Poz MR. Public health workforce: challenges and policy issues. Hum Resour Health. 2003;1:4.
World Health Organization. The World Health Report 2006: Working Together For Health. Geneva: WHO; 2006.
Figueroa-Munoz J, Palmer K, Dal Poz M, Blanc L, Bergström K, Raviglione M. The health workforce crisis in TB control: a report from high-burden countries. Hum Resour Health. 2005;3:2.
Wu Q, Zhao L, Ye XC. Shortage of healthcare professionals in China. BMJ. 2016;354:i4860.
Bowser D, Sparkes SP, Mitchell A, Bossert TJ, Barnighausen T, Gedik G, Atun R. Global Fund investments in human resources for health: innovation and missed opportunities for health systems strengthening. Health Policy Plan. 2014;29:986–97.
Wu S, Roychowdhury I, Khan M. Evaluations of training programs to improve human resource capacity for HIV, malaria and TB control: a systematic review of methods applied and outcomes assessed. Trop Med Health. 2017;45:16.
World Health Organization. Evaluating Training in WHO. Geneva: WHO; 2010.
Kirkpatrick D. Evaluating Training Programs: The Four Levels (3rd edition). San Francisco, CA: Berrett-Koehler Publishers; 2006.
Phillips PPJ. Symposium on the evaluation of training. Int J Train Dev. 2001;5:240–7.
Kraiger KFJ, Salas E. Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. J Appl Psychol. 1993;78:311–28.
Arthur WBW, Edens P, Bell S. Effectiveness of training in organizations: a meta-analysis of design and evaluation features. J Appl Psychol. 2003;88:234–45.
Guskey T. Five Levels of Professional Development Evaluation: North Central Regional Educational Laboratory (NCREL); 2002.
Kaufman R, Keller J, Watkins R. What works and what doesn't: evaluation beyond kirkpatrick. Perform Improv. 1996;35:8–12.
Kearns P, Miller T. Measuring the Impact of Training and Development on the Bottom Line. Upper Saddle River, NJ: Financial Times Prentice Hall; 1997.
Hamblin AC. Evaluation and Control of Training. Maidenhead: McGraw-Hill Co.; 1974.
Brauchle P, Schmidt K. Contemporary approaches for assessing outcomes on training, education, and HRD programs. J Ind Teach Educ. 2004;41:17.
O'Malley G, Perdue T, Petracca F. A framework for outcome-level evaluation of in-service training of health care workers. Hum Resour Health. 2013;11:50.
Alvarez K, Salas E, Garofano C. An integrated model of training evaluation and effectiveness. Hum Resour Dev Rev. 2004;3:385–416.
Bates R. A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Eval Program Planning. 2004;27:341–7.
Naude CE, Zani B, Ongolo-Zogo P, Wiysonge CS, Dudley L, Kredo T, Garner P, Young T. Research evidence and policy: qualitative study in selected provinces in South Africa and Cameroon. Implement Sci. 2015;10:126.
Sackett PR, Mullen EJ. Beyond formal experimental design: towards an expanded view of the training evaluation process. Pers Psychol. 1993;46:613–28.
Bowen S, Zwi AB. Pathways to “evidence-informed” policy and practice: a framework for action. PLoS Med. 2005;2:e166.
Hunter DJ. Relationship between evidence and policy: a case of evidence-based policy or policy-based evidence? Public Health. 2009;123:583–6.
Leir S, Parkhurst J. What is Good Evidence for Policy? London: London School of Hygiene and Tropical. Medicine. 2016;
Hutchison C, Khan MS, Yoong J, Lin X, Coker RJ. Financial barriers and coping strategies: a qualitative study of accessing multidrug-resistant tuberculosis and tuberculosis care in Yunnan, China. BMC Public Health. 2017;17:221.
Ritchie J, Lewis J, Nicholls CM, Ormston R. Qualitative Research Practice: A Guide for Social Science Students and Researchers. Thousand Oaks, CA: Sage; 2013.
Rice P, Ezzy D. Qualitative Research Methods: A Health Focus. Melbourne: Oxford University Press; 1999.
Boyatzis R. Transforming Qualitative Information: Thematic Analysis and Code Development. Thousand Oaks, CA: Sage; 1998.
Saldana J. The Coding Manual for Qualitative Researchers. Thousand Oaks, CA: Sage; 2009.
Glaser B, Strauss A. The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago: Aldine; 1967.
Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19:349–57.
Bramley P. Evaluating Training Effectiveness. Maidenhead: McGraw-Hill; 1996.
Warr P, Bird M, Rackcam N. Evaluation of Management Training. London: Gower; 1978.
Foxon M. Evaluation of training and development programs: a review of the literature. Aust J Educ Technol. 1989;5:89–104.
Wu S, Roychowdhury I, Khan M. Evaluating the impact of healthcare provider training to improve tuberculosis management: a systematic review of methods and outcome indicators used. Int J Infect Dis. 2017;56:105–10.
Johns B, Baltussen R, Hutubessy R. Programme costs in the economic evaluation of health interventions. Cost Eff Resour Alloc. 2003;1:1.
Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998;13:87–108.
Hyder AA, Bloom G, Leach M, Syed SB, Peters DH. Future Health Systems: Innovations for Equity. Exploring health systems research and its influence on policy processes in low income countries. BMC Public Health. 2007;7:309.
Onwuegbuzie AJ. Expanding the Framework of Internal and External Validity in Quantitative Research. 2000. https://eric.ed.gov/?id=ED448205. Accessed 12 Feb 2018.
Smithson J. Using and analysing focus groups: limitations and possibilities. Int J Social Research Methodology. 2000;3:103–19.
Black N. Evidence based policy: proceed with care. BMJ. 2001;323:275–9.