Effectiveness of incentives and follow-up on increasing survey response rates and participation in field studies
Tóm tắt
Questionnaires are valuable data collection instruments in public health research, and can serve to pre-screen respondents for suitability in future studies. Survey non-response leads to reduced effective sample sizes and can decrease representativeness of the study population, so high response rates are needed to minimize the risk of bias. Here we present results on the success of different postal questionnaire strategies at effecting response, and the effectiveness of these strategies at recruiting participants for a field study on the effects of aircraft noise on sleep. In total, we mailed 17 rounds of 240 questionnaires (total n = 4080) to randomly selected households around Atlanta International Airport. Different mailing rounds were varied in the length of the questionnaire (11, 26 or 55 questions), survey incentive (gift card or $2 cash), number of follow-up waves (0, 2 or 3), incentive for participating in a 5-night in-home sleep study ($100, $150 or $200), and address personalization. We received completed questionnaires from 407 respondents (response rate 11.4%). Personalizing the address, enclosing a $2 cash incentive with the initial questionnaire mailing and repeated follow-up mailings were effective at increasing response rate. Despite the increased expense of these approaches in terms of each household mailed, the higher response rates meant that they were more cost-effective overall for obtaining an equivalent number of responses. Interest in participating in the field study decreased with age, but was unaffected by the mailing strategies or cash incentives for field study participation. The likelihood that a respondent would participate in the field study was unaffected by survey incentive, survey length, number of follow-up waves, field study incentive, age or sex. Pre-issued cash incentives and sending follow-up waves could maximize the representativeness and numbers of people from which to recruit, and may be an effective strategy for improving recruitment into field studies.
Tài liệu tham khảo
Dillman DA, Smyth JD, Christian LM. Internet, phone, mail, and mixed-mode surveys : the tailored design method, 4th edition. Edn. Hoboken: Wiley; 2014.
Teddlie C, Yu F. Mixed methods sampling a typology with examples. J Mixed Methods Res. 2007;1(1):77–100.
Singer E. Introduction - nonresponse bias in household surveys. Public Opin Q. 2006;70(5):637–45.
Baruch Y. Response rate in academic studies - a comparative analysis. Hum Relat. 1999;52(4):421–38.
Groves RM. Nonresponse rates and nonresponse bias in household surveys. Public Opin Q. 2006;70(5):646–75.
Johnson TP, Wislar JS. Response rates and nonresponse errors in surveys. Jama. 2012;307(17):1805–6.
Groves RM, Peytcheva E. The impact of nonresponse rates on nonresponse bias - a meta-analysis. Public Opin Q. 2008;72(2):167–89.
Response rates - an overview [https://www.aapor.org/Education-Resources/For-Researchers/Poll-Survey-FAQ/Response-Rates-An-Overview.aspx].
Cook C, Heath F, Thompson RL. A meta-analysis of response rates in web- or internet-based surveys. Educ Psychol Meas. 2000;60(6):821–36.
American Association for Public Opinion Research. Standard definitions: final dispositions of case codes and outcome rates for surveys. 9th ed. Lenexa: AAPOR; 2016.
Czajka JL, Beyler A. Declining response rates in federal surveys: trends and implications. Final report - volume 1. In: Mathematica policy research; 2016.
Armstrong JS, Overton TS. Estimating nonresponse Bias in mail surveys. J Mark Res. 1977;14(3):396–402.
Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev 2009;(3).
Nakash RA, Hutton JL, Jorstad-Stein EC, Gates S, Lamb SE. Maximising response to postal questionnaires--a systematic review of randomised trials in health research. BMC Med Res Methodol. 2006;6:5.
Knoll M, Soller L, Ben-Shoshan M, Harrington D, Fragapane J, Joseph L, La Vieille S, St-Pierre Y, Wilson K, Elliott S, et al. The use of incentives in vulnerable populations for a telephone survey: a randomized controlled trial. BMC Res Notes. 2012;5:572.
Basner M, Dinges DF. Sleep duration in the United States 2003–2016: first signs of success in the fight against sleep deficiency? Sleep 2018; 41(4).
Cirelli C, Tononi G. Is sleep essential? PLoS Biol. 2008;6(8):e216.
Watson NF, Badr MS, Belenky G, Bliwise DL, Buxton OM, Buysse D, Dinges DF, Gangwisch J, Grandner MA, Kushida C, et al. Joint consensus statement of the American Academy of sleep medicine and Sleep Research Society on the recommended amount of sleep for a healthy adult: methodology and discussion. Sleep. 2015;38(8):1161–83.
World Health Organization. Burden of disease from environmental noise. Quantification of healthy life years lost in Europe. In: WHO Regional Office for Europe. Copenhagen: World Health Organization; 2011: 101.
Miedema HM, Vos H. Associations between self-reported sleep disturbance and environmental noise based on reanalyses of pooled data from 24 studies. Behav Sleep Med. 2007;5(1):1–20.
Iber C, Ancoli-Israel S, Chesson A, Quan SF. The AASM manual for the scoring of sleep and associated events; rules, terminology and technical specifications. 1st ed. Westchester: American Academy of Sleep Medicine; 2007.
Winser MA, McBean AL, Montgomery-Downs HE. Minimum duration of actigraphy-defined nocturnal awakenings necessary for morning recall. Sleep Med. 2013;14(7):688–91.
Münzel T, Gori T, Babisch W, Basner M. Cardiovascular effects of environmental noise exposure. Eur Heart J. 2014;35(13):829–36.
McGuire S, Witte M, Kallarackal A, Basner M. Pilot study examining the effects of aircraft noise on sleep in communities near Philadelphia international airport. Sleep. 2017;40:A60–1.
American Community Survey (ACS) [https://www.census.gov/programs-surveys/acs].
Buysse DJ, Reynolds CF 3rd, Monk TH, Berman SR, Kupfer DJ. The Pittsburgh sleep quality index: a new instrument for psychiatric practice and research. Psychiatry Res. 1989;28(2):193–213.
Weinstein ND. Individual differences in reactions to noise: a longitudinal study in a college dormitory. J Appl Psychol. 1978;63(4):458–66.
Fields JM, De Jong RG, Gjestland T, Flindell IH, Job RFS, Kurra S, Lercher P, Vallet M, Yano T, Guski R, et al. Standardized general-purpose noise reaction questions for community noise surveys: research and a recommendation. J Sound Vib. 2001;242(4):641–79.
Scott P, Edwards P. Personally addressed hand-signed letters increase questionnaire response: a meta-analysis of randomised controlled trials. BMC Health Serv Res. 2006;6:11.
Sahlqvist S, Song Y, Bull F, Adams E, Preston J, Ogilvie D, Consortium I. Effect of questionnaire length, personalisation and reminder type on response rate to a complex postal survey: randomised controlled trial. BMC Med Res Methodol. 2011;11:62.
Möller-Leimkühler AM, Möller H-J, Maier W, Gaebel W, Falkai P. EPA guidance on improving the image of psychiatry. Eur Arch Psychiatry Clin Neurosci. 2016;266(2):139–54.
Halpern SD, Ubel PA, Berlin JA, Asch DA. Randomized trial of $5 versus $10 monetary incentives, envelope size, and candy to increase physician response rates to mailed questionnaires. Med Care. 2002;40(9):834–9.
Guski R, Schreckenberg D, Schuemer R. WHO environmental noise guidelines for the European Region: a systematic review on environmental noise and annoyance. Int J Environ Res Public Health. 2017;14(12):E1539.
Sinclair M, O'Toole J, Malawaraarachchi M, Leder K. Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches. BMC Med Res Methodol. 2012;12:132.
Singer E, Van Hoewyk J, Neugebauer RJ. Attitudes and behavior - the impact of privacy and confidentiality concerns on participation in the 2000 census. Public Opin Q. 2003;67(3):368–84.
Bowling A. Mode of questionnaire administration can have serious effects on data quality. J Public Health-Uk. 2005;27(3):281–91.
Groves RM, Singer E, Corning A. Leverage-saliency theory of survey participation - description and an illustration. Public Opin Q. 2000;64(3):299–308.
Ryan C. Language use in the United States: 2011. In.: U.S. Department of Commerce. Economics and Statistics Administration. U.S. Census Bureau. 2013.
Brick JM, Montaquila JM, Han DF, Williams D. Improving response rates for Spanish speakers in two-phase mail surveys. Public Opin Q. 2012;76(4):721–32.
Guo YM, Kopec JA, Cibere J, Li LC, Goldsmith CH. Population survey features and response rates: a randomized experiment. Am J Public Health. 2016;106(8):1422–6.
Ryan JM, Corry JR, Attewell R, Smithson MJ. A comparison of an electronic version of the SF-36 general health questionnaire to the standard paper version. Qual Life Res. 2002;11(1):19–26.
Patrick ME, Couper MP, Laetz VB, Schulenberg JE, O'Malley PM, Johnston LD, Miech RA. A sequential mixed-mode experiment in the U.S. National Monitoring the future study. J Surv Stat Methodol. 2018;6(1):72–97.
Medway RL, Fulton J. When more gets you less: a meta-analysis of the effect of concurrent web options on mail survey response rates. Public Opin Q. 2012;76(4):733–46.
McMaster HS, LeardMann CA, Speigle S, Dillman DA, Tea MCFS. An experimental comparison of web-push vs. paper-only survey procedures for conducting an in-depth health survey of military spouses. BMC Med Res Methodol. 2017;17:73.
Halbesleben JRB, Whitman MV. Evaluating survey quality in health services research: a decision framework for assessing nonresponse Bias. Health Serv Res. 2013;48(3):913–30.
2010 Dicennial census of population and housing [https://www.census.gov/programs-surveys/decennial-census/decade.2010.html].
Funkhouser E, Vellala K, Baltuck C, Cacciato R, Durand E, McEdward D, Sowell E, Theisen SE, Gilbert GH, Grp NDPC. Survey methods to optimize response rate in the National Dental Practice-Based Research Network. Eval Health Prof. 2017;40(3):332–58.
Lin IF, Schaeffer NC. Using survey participants to estimate the impact of nonparticipation. Public Opin Q. 1995;59(2):236–58.
Singer E, Ye C. The use and effects of incentives in surveys. Ann Am Acad Polit Soc Sci. 2013;645(1):112–41.
Curtin R, Singer E, Presser S. Incentives in random digit dial telephone surveys: a replication and extension. J Off Stat. 2007;23(1):91–105.
Church AH. Estimating the effect of incentives on mail survey response rates - a Metaanalysis. Public Opin Q. 1993;57(1):62–79.
Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, Kwan I. Increasing response rates to postal questionnaires: systematic review. Br Med J. 2002;324(7347):1183–5.
Mercer A, Caporaso A, Cantor D, Townsend R. How much gets you how much? Monetary incentives and response rates in household surveys. Public Opin Q. 2015;79(1):105–29.
Han DF, Montaquila JM, Brick JM. An evaluation of incentive experiments in a two-phase address-based sample mail survey. Surv Res Methods-Ger. 2013;7(3):207–18.
Weil J, Mendoza AN, McGavin E. Recruiting older adults as participants in applied social research: applying and evaluating approaches from clinical studies. Educ Gerontol. 2017;43(12):662–73.
Sharp EC, Pelletier LG, Levesque C. The double-edged sword of rewards for participation in psychology experiments. Can J Behav Sci. 2006;38(3):269–77.
Mayfield A, Amaya A, Carris K. A matter of time: the value and optimal timing of follow-up questionnaire mailings in a multimode survey. In: 68th Annual Conference of the American Association for Public Opinion Research Boston, Boston: American Statistical Association; 2013.
Voigt LF, Koepsell TD, Daling JR. Characteristics of telephone survey respondents according to willingness to participate. Am J Epidemiol. 2003;157(1):66–73.
Basner M, McGuire S. WHO environmental noise guidelines for the European Region: a systematic review on environmental noise and effects on sleep. Int J Environ Res Public Health. 2018;15(3):E519.
Saleh A, Bista K. Examining factors impacting online survey response rates in educational research: perceptions of graduate students. J Multi Discip Eval. 2017;13(29):63–74.