Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

Partnership evaluation: An application of a developmental framework to the Governance and Local Development project in Senegal

This study explores the valued-added of USAID/Senegal’s Governance for Local Development (GOLD) project’s integrated approach to working with four sector projects: two in health, one in water and sanitation, and one in nutrition. The study builds on a partnership evaluation framework that identifies practices related to preconditions, factors linked with success, implementation structures and processes, contribution of collaboration to performance, and collaboration outcomes. A survey instrument developed statements of positive practices and asked respondents to agree or disagree with each one. The analysis includes exploration of differences in perception. Respondents generally expressed agreement with the statements in the survey instrument associated with the various dimensions of effective partnerships. All projects mentioned the presence of partnership champions. Problems with joint action (e.g., planning, budgeting, managing) were the most frequently cited constraints. The study confirms most of the hypotheses embedded in the evaluation framework regarding partnership’s potential for creating synergies and improved outcomes and affirms the framework’s usefulness as a developmental evaluation tool. Positive outcomes result from intentional efforts to build relationships, cultivate champions, and design systems and processes that support a partnership culture and contribute to each partners’ success as well as the success of the partnership itself.     

Workforce training in family therapy and systemic practices: An evaluation framework and case study

Internationally, while the interdisciplinary field of family therapy and systemic practices is well established and evidence for therapeutic impact is advanced, evaluation of training and measurement of its impact lags behind. This paper addresses this gap, utilising a single case study at the Bouverie Centre, the largest family therapy and workforce development service in Australia. We describe (a) the scope of workforce training and implementation at the Centre, (b) the development of our training research and evaluation framework, and (c) standardised data collection methodology, including a pre-post training reflective survey. Collectively, this work comprises our current evaluation framework through which we will subsequently examine training impact. Future research and practice recommendations are offered to address challenges to workforce training evaluation and progress efficacy of family therapy training.     

An application of the Impact Evaluation Process for designing a performance measurement and evaluation framework in K-12 environments

This article illustrates the application of the Impact Evaluation Process for the design of a performance measurement and evaluation framework for an urban high school. One of the key aims of this framework is to enhance decision-making by providing timely feedback about the effectiveness of various performance improvement interventions. The framework design process is guided by the Impact Evaluation Process, and included the participation of key stakeholders including administrative and teaching staff who all contributed to the performance measurement and evaluation framework design process. Key performance indicators at the strategic, tactical, and operational levels were derived from the school vision, and linked to specific interventions to facilitate the continuous evaluation and improvement process.     

An application of the Impact Evaluation Process for designing a performance measurement and evaluation framework in K-12 environments

This article illustrates the application of the Impact Evaluation Process for the design of a performance measurement and evaluation framework for an urban high school. One of the key aims of this framework is to enhance decision-making by providing timely feedback about the effectiveness of various performance improvement interventions. The framework design process is guided by the Impact Evaluation Process, and included the participation of key stakeholders including administrative and teaching staff who all contributed to the performance measurement and evaluation framework design process. Key performance indicators at the strategic, tactical, and operational levels were derived from the school vision, and linked to specific interventions to facilitate the continuous evaluation and improvement process.     

A comprehensive health service evaluation and monitoring framework

ObjectiveTo develop a framework for evaluating and monitoring a primary health care service, integrating hospital and community services.MethodA targeted literature review of primary health service evaluation frameworks was performed to inform the development of the framework specifically for remote communities. Key principles underlying primary health care evaluation were determined and sentinel indicators developed to operationalise the evaluation framework. This framework was then validated with key stakeholders.ResultsThe framework includes Donabedian's three seminal domains of structure, process and outcomes to determine health service performance. These in turn are dependent on sustainability, quality of patient care and the determinants of health to provide a comprehensive health service evaluation framework. The principles underpinning primary health service evaluation were pertinent to health services in remote contexts. Sentinel indicators were developed to fit the demographic characteristics and health needs of the population. Consultation with key stakeholders confirmed that the evaluation framework was applicable.ConclusionData collected routinely by health services can be used to operationalise the proposed health service evaluation framework. Use of an evaluation framework which links policy and health service performance to health outcomes will assist health services to improve performance as part of a continuous quality improvement cycle.     

Inpatient practice. An integrated framework.

A framework for practice that integrates nursing concepts and processes with system concepts has distinct advantages. Nursing concepts provide direction for nursing care; the nursing process provides the structure in which nursing care is delivered and evaluated. Application of system concepts enables the CNS to navigate and negotiate the larger institutional context. Inpatient practice is not just the delivery of quality patient care; it also involves managing the system so it is possible to deliver care. An integrated framework has been offered as a guide to inpatient practice. Although further use and testing are indicated, so far it has proved useful in the practice of a CNS.     

An organization-environment framework for assessing program implementation

Evaluations of health programs are typically conducted from a rational bureaucratic framework in which client change and client outcome are seen as logical outcomes of program activity. This paper proposes a theoretical framework for examining organizational consequences of program implementation. Factors in the internal environment (e.g., the hospital) and the external environment (e.g., the community) which contribute to the program's success or failure, are considered. Observational and structured interview data are used to compare two state funded perinatal projects, one an organizational "success" and the other a costly failure. Attributes of external environments which may be important to successful implementation include demand for services, provider competition, access to information, social beliefs, and professional commitment. Important features of the internal environment include host organization commitment and conflict, and structural attributes: complexity, formalization, centralization, and coordination. The paper concludes with theoretically-based questions which serve as guidelines for process evaluation.     

Diversity training for the community aged care workers: A conceptual framework for evaluation

Older Australians are an increasingly diverse population, with variable characteristics such as culture, sexual orientation, socioeconomic status, and physical capabilities potentially influencing their participation in healthcare. In response, community aged care workers may need to increase skills and uptake of knowledge into practice regarding diversity through appropriate training interventions. Diversity training (DT) programs have traditionally existed in the realm of business, with little research attention devoted to scientifically evaluating the outcomes of training directed at community aged care workers. A DT workshop has been developed for community aged care workers, and this paper focuses on the construction of a formative evaluative framework for the workshop. Key evaluation concepts and measures relating to DT have been identified in the literature and integrated into the framework, focusing on five categories: Training needs analysis; Reactions; Learning outcomes, Behavioural outcomes and Results The use of a mixed methods approach in the framework provides an additional strength, by evaluating long-term behavioural change and improvements in service delivery. As little is known about the effectiveness of DT programs for community aged care workers, the proposed framework will provide an empirical and consistent method of evaluation, to assess their impact on enhancing older people’s experience of healthcare.     

Empowerment evaluation: An approach that has literally altered the landscape of evaluation

The quest for credible and actionable evidence to improve decision making, foster improvement, enhance self-determination, and promote social betterment is now a global phenomenon. Evaluation theorists and practitioners alike have responded to and overcome the challenges that limited the effectiveness and usefulness of traditional evaluation approaches primarily focused on seeking rigorous scientific knowledge about social programs and policies. No modern evaluation approach has received a more robust welcome from stakeholders across the globe than empowerment evaluation.Empowerment evaluation has been a leader in the development of stakeholder involvement approaches to evaluation, setting a high bar. In addition, empowerment evaluation’s respect for community knowledge and commitment to the people’s right to build their own evaluation capacity has influenced the evaluation mainstream, particularly concerning evaluation capacity building. Empowerment evaluation’s most significant contributions to the field have been to improving evaluation use and knowledge utilization.     

Evaluation readiness: improved evaluation planning using a data inventory framework

Factors intrinsic to many programs, such as ambiguously stated objectives, inadequately defined performance measures, and incomplete or unreliable databases, often conspire to limit the evaluability of these programs. Current evaluation planning approaches are somewhat constrained in their ability to overcome these obstacles and to achieve full preparedness for evaluation. In this paper, the concept of evaluation readiness is introduced as a complement to other evaluation planning approaches, most notably that of evaluability assessment. The basic products of evaluation readiness--the formal program definition and the data inventory framework--are described, along with a guide for assuring more timely and appropriate evaluation response capability to support the decision making needs of program managers. The utility of evaluation readiness for program planning, as well as for effective management, is also discussed.     

The manifestations of politics in evaluation: An exploratory study across the evaluation process

Evaluation has been described as a political act. Programs and policies are generated from a political process, and the decision to evaluate and how to use the evaluation are manifestations of the political dynamic. This exploratory study was conducted with practicing evaluators to understand what they view as political situations in the evaluation process and how they responded to these situations. Findings suggest that, in relation to the potential evaluation phases in which each respondent has been involved, evaluations are susceptible to politics when initially attempting to identify stakeholders and when it’s time to report the evaluation findings. Evaluators have also developed multiple strategies for dealing with these situations, including finding allies for the evaluation and working to explain the evaluation process and its implications. We hope that this study will help to inform novice and expert evaluators about the various political situations they may encounter in their practice.     

The priority of justice: a framework approach to ethics in program evaluation

The situations of evaluation are inevitably complex and various, often involving conflicts between ethical principles as well as among aims or stakeholders. To meet this challenge, evaluators and stakeholders need an interpretive framework to clarify the issues at hand and open the way to workable solutions. This paper provides a three-part framework of justice to guide practitioners in this interpretive task.First, evaluation is instituted to serve the public with fidelity to the values, standards, and ideals that characterize it as a profession and as an office deserving the public trust, here called public justice. Second, these requirements evoke and entail the more general demands of fair process, mutual respect, and right action known as procedural justice. Third, the operations of office and just process must serve the public good, meeting the demands of distributive justice that govern the allocation of goods and benefits.This framework takes as a starting point the priority of justice as articulated by John Rawls: “Justice is the first virtue of social institutions, as truth is to thought. A theory however elegant and economical must be rejected or revised if it is untrue; likewise laws and institutions no matter how efficient and well-arranged must be reformed or abolished if they are unjust” (1971, p. 3).     

The design of a contextualized responsive evaluation framework for fishery management in Benin

The main question addressed by this article is how to adapt the responsive evaluation (RE) approach to an intervention context characterized by repetition of ineffective interventions, ambiguous intervention action theories among stakeholders, and high complexity. The context is Grand-Popo, a fishing municipality located on Benin's southwest Atlantic coast. The fishery management interventionists and the fishing communities in the municipality all espoused concern for the sustainable improvement of fishing actors' livelihood conditions, but differed about the reasons for this livelihood impairment, and about what should be done, when, where, and by whom. Given this ambiguity, we identified RE as a promising action research approach to facilitate dialogue and mutual learning, and consequently to improve stakeholders' ability to resolve problems. However, this approach seems to have some shortcomings in the Grand-Popo context, regarding the repetitive ineffectiveness of interventions, high complexity, and uncertainty. Therefore, based on our empirical study, we add three dimensions to the existing RE framework: historical analysis to deal with routine interventions, exploration and discussion of incongruities of action theories to trigger double-loop learning, and system analysis to deal with complexity and uncertainty. This article does not intend to address the implications or impact of this adapted RE framework. Instead, we suggest some criteria and indicators for evaluating whether the proposed amended RE approach has assisted in resolving the fishery problems in Grand-Popo after the approach has been applied.