A practitioner's guide to meta-analysis
Tóm tắt
Meta-analysis can be a powerful and useful technique. In the short term, the results of meta-analysis can increase understanding and influence the future endeavors of researchers interested in a particular field. Over the long term, meta-analysis provides a vehicle by which large bodies of research can be integrated and focused on public policy issues. Administrative structures currently exist to foster the relationship between research findings and public policy. As one example, Saxe (1986) has described the activities of the Office of Technology Assessment (OTA) established by Congress to provide lawmakers with scientific information in such a way as to be useful for policy decision-making. OTA is commissioned by Congress to provide assessments of scientific findings, and meta-analyses are an ideal resource in this regard. Saxe (1986) notes “In almost all cases, the emphasis is on making sense of already collected data, thus literature reviews and methods for synthesizing research results are relied upon” (pp. 61–62). Whether meta-analysis actually plays an important role in public policy depends on many factors, not the least of which is the care with which meta-analyses are conducted. Hopefully, our comments here will help investigators conduct methodologically rigorous meta-analyses of research relevant to the field of community psychology.
Tài liệu tham khảo
Abelson, R. P. (1985). A variance explanation paradox: When a little is a lot.Psychological Bulletin, 97, 129–133.
Abrami, P. C., Cohen, P. A., & d'Apollonia, S. (1988). Implementation problems in meta-analysis.Review of Educational Research, 58, 151–179.
Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity.Psychological Bulletin, 101, 213–232.
Bangert-Drowns, R. L. (1986). Review of developments in meta-analytic method.Psychological Bulletin, 99, 388–399.
Bullock, R. J., & Svyantek, D. J. (1985). Analyzing meta-analysis: Potential problems, an unsuccessful replication, and evaluation criteria.Journal of Applied Psychology, 70, 108–115.
Carver, R. P. (1975). The Coleman Report: Using inappropriately designed achievement tests.American Educational Research Journal, 12, 77–86.
Chalmers, T. C., Smith, H., Blackburn, B., Silverman, B., Schroeder, B., Reitman, D., & Ambroz, A. (1981). A method for assessing the quality of a randomized control trial.Controlled Clinical Trials, 2, 31–49.
Chambless, D. L. (1989). Measurement effects on outcome of treatment for agoraphobia.Behavior Therapy, 20, 465–466.
Cohen, J. (1977).Statistical power analysis for the behavioral sciences (rev. ed.). New York: Academic Press.
Cohen, J. (1988).Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
Cook, T. D., & Campbell, D. T. (1979).Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally.
Cooper, H. (1984).The integrative research review: A systematic approach. Beverly Hills, CA: Sage.
Cordray, D. S. (1990). Strengthening causal interpretations of non-experimental data: The role of meta-analysis. In L. Sechrest, E. Perrin, & J. Bunker (Eds.),Research methodology: Strengthening causal interpretations of nonexperimental data (pp. 151–172). Washington, D.C.: U.S. Department of Health and Human Services, Agency for Health Care Policy and Research.
Cordray, D. S., & Orwin, R. G. (1983). Improving the quality of evidence: Interconnections among primary evaluation, secondary analysis, and quantitative synthesis. In R. J. Light (Ed.),Evaluation studies review annual (Vol. 8, pp. 91–119). Beverly Hills, CA: Sage.
Curlette, W. L. (1987). The meta-analysis effect size calculator: A BASIC program for reconstructing unbiased effect sizes.Educational and Psychological Measurement, 47, 107–109.
Devine, E. C., & Cook, T. D. (1983). Effects of psychoeducational interventions on length of post-surgical hospital stay: A meta-analytic review of thirty-four studies. In R. J. Light (Ed.),Evaluation studies review annual (Vol. 8). Beverly Hills, CA: Sage.
Dobson, K. S. (1989). A meta-analysis of the efficacy of cognitive therapy for depression.Journal of Consulting and Clinical Psychology, 57, 414–419.
Durlak, J. A., Lampman, C., & Wells, A. (1991).Effectiveness of child psychotherapy: A re-evaluation. Manuscript submitted for publication.
Durlak, J. A., Fuhrman, T., & Lampman, C. (in press). Effectiveness of cognitive behavior therapy for maladapting children: A meta-analysis.Psychological Bulletin.
Dush, D. M., Hirt, M. L., & Schroeder, H. E. (1983). Self-statement modification with adults: A meta-analysis.Psychological Bulletin, 94, 408–422.
Fuchs, D., & Fuchs, L. S. (1989). Effects of examiner familiarity on Black, Caucasian, and Hispanic children: A meta-analysis.Exceptional Children, 55, 303–308.
Garfield, S. L. (Ed.). (1983). Meta-analysis and psychotherapy (special section).Journal of Consulting and Clinical Psychology, 51(1).
Gibbs, L. E. (1989). Quality of study rating form: An instrument for synthesizing evaluation studies:Journal of Social Work Education, 25, 55–67.
Glass, G. V. (1976). Primary, secondary, and meta-analysis of research.Educational Researcher, 5, 3–8.
Glass, G. V., McGaw, B., & Smith, M. L. (1981).Meta analysis in social research. Beverly Hills, CA: Sage Publications.
Guzzo, R. A., Jackson, S. E., & Katzell, R. A. (1987). Meta-analysis analysis.Research in Organizational Behavior, 9, 407–442.
Hartmann, D. P. (Ed.). (1982).Using observers to study behavior. New directions for methodology of social and behavioral sciences. San Francisco: Jossey-Bass.
Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators.Journal of Educational Statistics, 6, 107–128.
Hedges, L. V. (1982a). Estimation of effect size from a series of independent experiments.Psychological Bulletin, 92, 490–499.
Hedges, L. V. (1982b). Fitting categorical models to effect sizes from a series of experiments.Journal of Educational Statistics, 7, 119–137.
Hedges, L. V. (1982c). Fitting continuous models to effect size data.Journal of Educational Statistics, 7, 245–270.
Hedges, L. V. (1983). A random effects model for effect sizes.Psychological Bulletin, 93, 388–395.
Hedges, L. V. (1984). Advances in statistical methods for meta-analysis.New Directions for Program Evaluation, 24, 25–42.
Hedges, L. V., & Olkin, I. (1985).Statistical methods for meta-analysis. New York: Academic Press.
Holmes, C. T. (1984). Effect size estimation in meta-analysis.Journal of Experimental Education, 52, 106–109.
Hovell, M. F. (1982). The experimental evidence for weight-loss treatment of essential hypertension: A critical review.American Journal of Public Health, 72, 359–368.
Hunter, J. E., & Schmidt, F. L. (1990).Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
Hunter, J. E., Schmidt, F. L., & Jackson, G. B. (1982).Meta-analysis: Cumulating research findings across studies. Beverly Hills, CA: Sage.
Hyde, J. S. (1981). How large are cognitive gender differences: A meta-analysis using omega andd. American Psychologist, 36, 892–901.
Jackson, G. B. (1980). Methods for integrative reviews.Review of Educational Research, 50, 438–460.
Jackson, S. E. (1984, August).Can meta-analysis be used for theory development in organizational psychology? Paper presented at the Meeting of the American Psychological Association, Toronto, Canada.
Jorm, A. F. (1987). Sex differences in neuroticism: A quantitative synthesis of published research.Australian and New Zealand Journal of Psychiatry, 21, 501–506.
Kazdin, A. E., & Bass, D. (1989). Power to detect differences between alternative treatments in comparative psychotherapy outcome research.Journal of Consulting and Clinical Psychology, 57, 138–147.
Kiesler, C. A. (1985). Meta-analysis, clinical psychology, and social policy.Clinical Psychology Review, 5, 3–12.
Kulik, J. A. (1984, April).The uses and misuses of meta-analysis. Paper presented at the meeting of the American Educational Research Association, New Orleans.
Levin, H. M., Glass, G. V., & Meister, G. R. (1984).Cost-effectiveness of four education interventions. Stanford, CA: Stanford University.
Light, R. J., & Pillemer, D. B. (1984).Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press.
Lipsey, M. W. (1988). Practice and malpractice in evaluation research.Evaluation Practice, 9, 5–24.
Lipsey, M. W. (1990).Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage.
Lipsey, M. W. (1991). Juvenile delinquency treatment: A meta-analytic inquiry into the variability of effects. InMeta-analysis for explanation: A casebook. New York: Russell Sage Foundation.
Lipsey, M. W., Crosse, S., Dunkle, J., Pollard, J., & Stobart, G. (1985). Evaluation: The state of the art and the sorry state of the science.New Directions for Program Evaluation, 27, 7–28.
Loeber, R., & Stouthamer-Loeber, M. (1986). Family factors as correlates and predictors of juvenile conduct problems and delinquency. In M. Tonry & N. Morris (Eds.),Crime and justice: An annual review of research (Vol. 7). Chicago, IL: University of Chicago Press.
Martocchio, J. J., & O'Leary, A. M. (1989). Sex differences in occupational stress: A meta-analytic review.Journal of Applied Psychology, 74, 495–501.
Matt, G. E. (1989). Decision rules for selecting effect sizes in meta-analysis: A review and reanalysis of psychotherapy outcome studies.Psychological Bulletin, 105, 106–115.
Messick, S., & Jungeblut, A. (1981). Time and method in coaching for the SAT.Psychological Bulletin, 89, 191–216.
Miller, R. C., & Berman, J. S. (1983). The efficacy of cognitive behavior therapies: A quantitative review of the research evidence.Psychological Bulletin, 94, 39–53.
Nurius, P. S., & Yeaton, W. H. (1987). Research synthesis reviews: An illustrated critique of “hidden” judgments, choices, and compromises.Clinical Psychology Review, 7, 695–714.
O'Grady, K. E. (1982). Measures of explained variance: Cautions and limitations.Psychological Bulletin, 92, 766–777.
Okun, M. A., & Stock, W. A. (1987). The construct validity of subjective well-being measures: An assessment via quantitative research synthesis.Journal of Community Psychology, 15, 481–492.
Orwin, R. G. (1983). A fail-safeN, 8, 157–159.
Orwin, R. G., & Cordray, D. S. (1985). Effects of deficient reporting on meta-analysis: A conceptual framework and reanalysis.Psychological Bulletin, 97, 134–147.
Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling multivariate effect sizes.Psychological Bulletin, 103, 111–120.
Robinson, L. A., Berman, J. S., & Neimeyer, R. A. (1990). Psychotherapy for the treatment of depression: A comprehensive review of the controlled outcome research.Psychological Bulletin, 108, 30–49.
Rosenthal, R. (1979). The “file drawer problem” and tolerance for null results.Psychological Bulletin, 86, 638–641.
Rosenthal, R. (1984).Meta-analytic procedures for social research. Beverly Hills, CA: Sage.
Rosenthal, R., & Rubin, D. B. (1982a). Comparing effect sizes of independent studies.Psychological Bulletin, 92, 500–504.
Rosenthal, R., & Rubin, D. B. (1982b). A simple, general purpose display of magnitude of experimental effect.Journal of Educational Psychology, 74, 166–169.
Russell Sage Foundation (1991).Meta-analysis for explanation: A casebook. New York: Russell Sage Foundation.
Sackett, P. R., Harris, M. M., & Orr, J. M. (1986). On seeking moderator variables in the meta-analysis of correlational data: A Monte Carlo investigation of statistical power and resistance to Type I error.Journal of Applied Psychology, 71, 302–310.
Saxe, L. (1986). Policymakers' use of social science research: Technology assessment in the U.S. Congress.Knowledge: Creation, Diffusion, Utilization, 8, 59–78.
Sechrest, L., & Yeaton, W. H. (1981). Empirical bases for estimating effect size. In R. F. Boruch, P. M. Wortman, D. S. Cordray, & Assocs. (Eds.),Reanalyzing program evaluations: Policies and practices for secondary analysis of social and educational programs (pp. 212–224). San Francisco, CA: Jossey-Bass.
Sechrest, L., & Yeaton, W. H. (1982). Magnitudes of experimental effects in social science research.Evaluation Review, 6, 579–600.
Shadish, Jr., W. R., Doherty, M., & Montgomery, L. M. (1989). How many studies are in the file drawer? An estimate from the family/marital psychotherapy literature.Clinical Psychology Review, 9, 589–603.
Shapiro, D. A., & Shapiro, D. (1983). Comparative therapy outcome research: Methodological implications of meta-analysis.Journal of Consulting and Clinical Psychology, 51, 42–53.
Skiba, R. J. (1989). The importance of construct validity: Alternative models for the assessment of behavioral disorders.Behavioral Disorders, 14, 175–185.
Slavin, R. E. (1986). Best evidence synthesis: An alternative to meta-analytic and traditional reviews.Educational Researcher, 15, 5–11.
Smith, M. L. (1980). Publication bias and meta-analysis.Evaluation and Education, 4, 22–24.
Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies.American Psychologist, 32, 752–760.
Smith, M. L., Glass, G. V., & Miller, T. I. (1980).The benefits of psychotherapy. Baltimore, MD: Johns Hopkins University Press.
Spector, P. E., & Levine, E. L. (1987). Meta-analysis for integrating study outcomes: A Monte-Carlo study of its susceptibility to Type I and Type II errors.Journal of Applied Psychology, 72, 3–9.
Stevenson, M. R., & Black, K. N. (1988). Paternal absence and sex role development: A meta-analysis.Child Development, 59, 793–814.
Stock, W. A., Okun, M. A., Haring, M. J., Miller, W., Kinney, C., & Seurvorst, R. W. (1982). Rigor in data synthesis: A case study of reliability in meta-analysis.Educational Researcher, 11, 10–14.
Strube, M. J., Gardner, W., & Hartmann, D. P. (1985). Limitations, liabilities, and obstacles in reviews of the literature: The current status of meta-analysis.Clinical Psychology Review, 5, 63–78.
Suls, J., & Wan, C. K. (1989). The relation between Type A behavior and chronic emotional distress: A meta-analysis.Journal of Personality and Social Psychology, 57, 503–512.
Tail, M., Padgett, M. Y., & Baldwin, T. T. (1989). Job and life satisfaction: A re-evaluation of the strength of the relationship and gender effects as a function of the date of the study.Journal of Applied Psychology, 74, 502–507.
Trull, T. J., Nietzel, M. T., & Main, A. (1988). The use of meta-analysis to assess the clinical significance of behavior therapy for agoraphobia.Behavior Therapy, 19, 527–538.
Wanous, J. P., Sullivan, S. E., & Malinak, J. (1989). The role of judgment calls in meta-analysis.Journal of Applied Psychology, 74, 259–264.
Weisz, J. R., Weiss, B., Alicke, M. D., & Klotz, M. L. (1987). Effectiveness of psychotherapy with children and adolescents: A meta-analysis for clinicians.Journal of Consulting and Clinical Psychology, 55, 542–549.
Wilson, G. T. (1985). Limitations of meta-analysis in the evaluation of the effects of psychological therapy.Clinical Psychology Review, 5, 35–47.
Wolf, F. M. (1986).Meta-analysis: Quantitative methods for research synthesis, Beverly Hills, CA: Sage.
Wortman, P. M. (1983). Evaluation research: A methodological perspective.Annual Review of Psychology, 34, 223–260.
Wortman, P. M., & Bryant, F. B. (1985). School desegregation and black student achievement: An integrative review.Sociological Methods and Research, 13, 289–324.
Yates, B. T. (1985). Cost-effectiveness analysis and cost-benefit analysis: An introduction.Behavioral Assessment, 7, 207–234.
Yeaton, W. H., & Wortman, P. M. (in press). On the reliability of meta-analytic reviews: The role of intercoder agreement.