American Journal of Evaluation

A general inductive approach for analysis of qualitative evaluation data is described. The purposes for using an inductive approach are to (a) condense raw textual data into a brief, summary format; (b) establish clear links between the evaluation or research objectives and the summary findings derived from the raw data; and (c) develop a framework of the underlying structure of experiences or processes that are evident in the raw data. The general inductive approach provides an easily used and systematic set of procedures for analyzing qualitative data that can produce reliable and valid findings. Although the general inductive approach is not as strong as some other analytic strategies for theory or model development, it does provide a simple, straightforward approach for deriving findings in the context of focused evaluation questions. Many evaluators are likely to find using a general inductive approach less complicated than using other approaches to qualitative data analysis.

An important final step in the life cycles of programs and their evaluation involves assessing new programs’ or innovations’ sustainability. This review and synthesis of 19 empirical studies of the sustainability of American and Canadian health-related programs examines the extent of sustainability achieved and summarizes factors contributing to greater sustainability. Three definitions for measuring sustainability were examined: continued program activities (18 studies), continued measured benefits or outcomes for new clients (2 studies), and maintained community capacity (6 studies). Methods of studying sustainability were also assessed. In 14 of 17 studies covering the continuation of program activities, at least 60% of sites reported sustaining at least one program component. Although these studies’ methods had substantial limitations, cross-study analysis showed consistent support for five important factors influencing the extent of sustainability: (a) A program can be modified over time, (b) a “champion” is present, (c) a program “fits” with its organization’s mission and procedures, (d) benefits to staff members and/or clients are readily perceived, and (e) stakeholders in other organizations provide support.

Increasingly, collaboration between business, non-profit, health and educational agencies is being championed as a powerful strategy to achieve a vision otherwise not possible when independent entities work alone. But the definition of collaboration is elusive and it is often difficult for organizations to put collaboration into practice and assess it with certainty. Program evaluators can assist practitioners concerned with the development of a strategic alliance predicated on collaboration by understanding and utilizing principles of collaboration theory. The Strategic Alliance Formative Assessment Rubric (SAFAR) is an assessment tool that captures central principles of collaboration and has been used as part of a four-step evaluation process to help alliance leaders, managers, and members in Safe School/Healthy Student Initiatives to quantitatively and qualitatively gauge, celebrate, and communicate the relative strength of their collaborative endeavor over time. The collaboration principles and corresponding assessment processes described in this article can be used by evaluators of large- or small-scale initiatives that seek to capitalize on the synergistic power of the “collaborative effort.”

Evaluators sometimes wish for a Fairy Godmother who would make decision makers pay attention to evaluation findings when choosing programs to implement. The U.S. Department of Education came close to creating such a Fairy Godmother when it required school districts to choose drug abuse prevention programs only if their effectiveness was supported by “scientific” evidence. The experience showed advantages of such a procedure (e.g., reduction in support for D.A.R.E., which evaluation had found wanting) but also shortcomings (limited and in some cases questionable evaluation evidence in support of other programs). Federal procedures for identifying successful programs appeared biased. In addition, the Fairy Godmother discounted the professional judgment of local educators and did little to improve the fit of programs to local conditions. Nevertheless, giving evaluation more clout is a worthwhile way to increase the rationality of decision making. The authors recommend research on procedures used by other agencies to achieve similar aims.

The authors present a case study examining the potential for policies to be “evidence-based.” To what extent is it possible to say that a decision to implement a complex social intervention is warranted on the basis of available empirical data? The case chosen is whether there is sufficient evidence to justify banning smoking in cars carrying children. The numerous assumptions underpinning such legislation are elicited, the weight and validity of evidence for each is appraised, and a mixed picture emerges. Certain propositions seem well supported; others are not yet proven and possibly unknowable. The authors argue that this is the standard predicament of evidence-based policy. Evidence does not come in finite chunks offering certainty and security to policy decisions. Rather, evidence-based policy is an accumulative process in which the data pursue but never quite capture unfolding policy problems. The whole point is the steady conversion of “unknowns” to “knowns.”

Environment and sustainable development show how policies are becoming ever more complex and ambiguous. This trend calls for new evaluation approaches. They need to be more clearly focused on specific, explicit concerns. They must be driven by a strategic concept of use to overcome the vulnerability to manipulation of many integrative, essentially procedural, approaches to policy making and evaluation. This article presents a conceptual framework for such evaluations and a four-step approach: defining the focal concern; developing criteria and synthesizing observations on the extent to which the focal concern is met; identifying and assessing all policies contributing to this outcome; and complementing this with the evaluation of policies specifically aimed at meeting the focal concern. Examples are taken essentially from wetland-related policies in France and Senegal. The article discusses how this approach tackles some crucial issues in evaluation research and practice and advocates closer connections between evaluation and critical research on policies.

Sweden has a low prevalence of bullying and Swedish schools are legally obliged to have anti-bullying policies. Many commercial programs are available. A mixed methods, quasi-experimental, concurrent evaluation of 8 programs, chosen from a pool of 21 widely used anti-bullying programs, was planned. Preliminary data, based on 835 stakeholder interviews, both individual and group, revealed extensive program-contamination in "treatment" schools ( n = 31), and extensive program use in nominally designated control/comparison schools ( n = 8). An evolved evaluation strategy focused on arrays of program components, use of which was assessed through detailed qualitative data gathered from principals, teaching and non-teaching staff, anti-bullying teams, and pupils. Bullying victimization, Grades 4-9, in 39 schools, was assessed by questionnaire, administered in three waves. Response rates were 76.1% (with N = 10,919) at the first wave, 74.3% (of 10,660) at the second wave, and 70.6% (of 10,523) at the final wave. Longitudinal aggregate and individual outcomes (3,487 pupils answered all three questionnaires) were assessed. Prevalence of victimization was 8.1% at first wave and (7.4% at last). Based on individual data, persistent victimization (bullied at both waves) was estimated at 1.5% with 4.7% experiencing a worsening situation (not bullied at the first wave but bullied at last) and 4.8% with an improved situation. Hierarchical cluster analysis was used to designate intervention types (shared arrays of strong, weak, or absent component use) and schools with similar approaches. A quasi-experimental, longitudinal design with individual data and an aggregate cross-sectional follow-up indicated differences in effective, ineffective, and iatrogenic components, component clusters (intervention types), and successful schools.