Building Evaluation Capacity with Community Stakeholders

I Introduction



Most health promotion programs involve a partnership model of delivery. While inviting community stakeholders to participate in the planning of a program is becoming the norm and these programs are increasingly evaluated, how to involve community stakeholders in program evaluation can still be challenging.



This article will discuss building evaluation capacity with stakeholders by describing the involvement of stakeholders, benefits of program evaluation, and the role of logic models. Practical tips will be shared to increase the evaluation capacity of community stakeholders and the utilization of the evaluation results.

II Involvement of Stakeholders



The involvement of stakeholders in program evaluation is imperative for the success of the program and the utilization of the evaluation results. Stakeholders are defined as individuals or groups who have an interest in the evaluation findings (Patton, 1997). The first step in program evaluation is to clarify the program and this includes the engagement of stakeholders (The Health Communication Unit, 2002).



Engagement of stakeholders includes

* defining who they are,

* understanding their interests and expectations, and

* ensuring their participation and including them in all aspects of decision making about the evaluation (The Health Communication Unit, 2002).



This reflects a participatory approach to evaluation. The benefits to this type of approach include

* facilitating the process of empowerment,

* increasing the likelihood the results will be utilized because stakeholders participated in the decision making,

* fostering greater understanding of the program, and

* ensuring that evaluation questions are appropriate.



As most health promotion work involves partnerships, any assessment of health promotion programs requires involving the partners or stakeholders. Milstein and Wetterhall (1999) support this belief by identifying three principal groups of stakeholders that are critical to have involved in program evaluation:

* those involved in program operations,

* those served or affected by the program,

* and primary users of the evaluation.



Health promotion practitioners need to ensure that these three principal stakeholders are involved in the program evaluation and that the evaluation addresses the needs of the program based on the understanding of these stakeholders. Moreover, "primary users of the evaluation are the specific persons who are in a position to do or decide something regarding the program "(Milstein and Wetterhall, 1999). It makes sense, therefore, to involve community partners in program evaluation.



The Canadian Evaluation Society (1999) identified that evaluation utilization is enhanced when

* decisions made from the evaluation are significant to stakeholders and answered through formal data collection,

* evaluation findings are consistent with the beliefs and expectations of the stakeholders, and

* stakeholders are involved in the evaluation process.



Patton (1997) further notes that "a particular evaluation may have multiple levels of stakeholders and therefore need multiple levels of stakeholder involvement." This is true for health promotion program as there are community stakeholders, funding stakeholders, agency stakeholders, and provincial stakeholders and each of these stakeholders may have differing expectations for a program evaluation and, therefore, need to be included in the design and implementation. Moreover, they need to be included in order for them to value and use the results.



~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ *



II Benefits of Program Evaluation



Porteous, Sheldrick and Stewart (1997) define program evaluation as "the systematic gathering, analysis, and reporting of data about a program to assist in decision-making."



Benefits of engaging in program evaluation are

* examining aspects of the program for effectiveness;

* producing data to support or not support decision making about how the program is delivered;

* identifying how best to use available resources;

* contributing to ongoing decisions about programs such as staffing and the optimal use of time and resources;

* identifying whether the program is meeting the needs of participants;

* improving the program and demonstrate the effectiveness of the program to stakeholders;

* identifying gaps in information, skills, or resources;

* helping to identify priority areas and what things are working well and what things need improvement (Porteous, Sheldrick and Stewart, 1997).



Program evaluation is about making things better and increasing everyone's knowledge about how health promotion programs can achieve what they set out to do.



A Clarifies Program Goals and Objectives



Program evaluation establishes clarity around program goals and objectives and identifies measurable indicators. As presented in the Evaluating Health Promotion Programs workbook by The Health Communication Unit (2002), the first step in program evaluation is to clarify the program by defining the program goals, population of interest, outcome objectives, program activities and outputs, and measurable program indicators. Therefore, another benefit of program evaluation is that each of these items is clearly defined and can be shared with others.



In creating a shared understanding of what the program hopes to achieve, how it will be achieved, and what will be measured to ensure achievement; the results of the evaluation increases the evaluation capacity of stakeholders as the information discovered becomes usable information. This understanding is critical when working with community stakeholders. It ensures those that are working on the program can share their energy and talents to move the program towards the same goal.



B Leads to Best Practices



Evaluating clearly defined goals and objectives for programs contributes to the best practice research for health promotion.



This information provides operative guidance to those directly involved in the program evaluation as well as gives credibility to the global perception of health promotion practice. Evaluation of programs leads to best practices within health promotion. Best practices under the what works approach is "a set of criteria is used to judge the effectiveness of specific health promotion actions" (Kahan, Goodstadt and Rajkumar, 1999).



~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ *



IV Role of Logic Models: A tangible tool to share with stakeholders



Logic models have a role in the evaluation of health promotion programs. Logic models are a one-page visual summary of the decisions made to implement a program. Logic models include program goals, population of interest, longer-term outcome objectives, short-term outcome objectives, outputs and activities (The Health Communication Unit, 2002). The model reflects the relationship between the programs goals, objectives, and activities.



When a program can be clearly defined through a logic model, it increases the quality of the evaluation as it focuses the evaluation of your program around essential linkages and engage(s) the stakeholders in the evaluation" (The Health Communication Unit, 2002). The Canadian Evaluation Society (1999) also identified that developing a program logic model provides a "useful framework for examining outcomes [and i]dentifies key program components that need to be tracked to assess program effectiveness."



Building on this theme of logic models for developing a common understanding, Rush and Ogborne (1991) state logic models "also aid in the identification of the key issues and questions that need to be pursued in the evaluation, thereby using evaluation resources efficiently and increasing the chances of utilization."



Identifying the key issues and questions that need to be addressed in the evaluation is critical to the program. Nobody wants to waste time and resources evaluating a program only to find out that the wrong questions were asked. Therefore, the logic model assists in the development of choosing evaluation questions about the program by providing the parameters of what the program intends to do and how this can be measured.



A Asking the right questions



The Canadian Evaluation Society (1999) identified the following questions to reflect on when developing practical and useful evaluation questions.

"* Can it be answered through a data collection process?

* Does the phrasing of the questions predetermine the answers to the questions? If so, rephrase the question.

* Have the intended users of the evaluation indicated how they would use the answers?

* Have the users of the evaluation specified the relevance of the answers for future action?"



Engaging in program evaluation also means taking the time to thoughtfully develop evaluation questions in order to design an evaluation that creates results that can be used.



B Means of Communicating



Another relationship of logic models to evaluation is as a means to communicate clearly with stakeholders. Kirkpatrick (2001) identified that "the program logic model can help organizations to take their program evaluations a step further by involving stakeholders in considering not only whether objectives are achieved, but also whether programs address problems that are appropriate for the particular program and organization." Thus, it provides a tool to communicate to stakeholders the basic nature and purpose of the program and to open the door for a discussion about the appropriateness of the program.



Some health promotion practitioners struggle with discussing whether a program has moved away from the original purpose and view the logic model, as part of the program evaluation, as an excellent way to ensure that the program is appropriate for that organization and doing what it set out to do. Kirkpatrick (2001) sums up the overall benefits of logic models in evaluation with "while logic model development can be a time consuming and detail-oriented process, the end product is a model that is very useful in guiding the implementation and evaluation of a program. Furthermore, the process can facilitate team building and stakeholder buy in."



~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ *



V Practical Tips to Build Evaluation Capacity with Community Stakeholders



A Dialogue about Evaluation:

* Do a quick go around at a stakeholder meeting asking "What feelings do you experience when thinking about evaluating a program." Write a positive and negative sign on a flip chart and write down the different feelings under the headings. Discuss as a group the different feelings about evaluation and ways to address concerns.

* Talk about evaluation at the beginning of a program and not at the end. This allows everyone to design a well laid out program to be evaluated.

* Talk about barriers to participation. Why are stakeholders not interested in participating? What are their learning needs? What are your learning needs?



B Share

* Share knowledge and resources. Provide them with web sites and training material on the topic. Highlight the different resources used to guide the work of health promotion.

* Sell the benefits of program evaluation. Demonstrate the "WIIFT" (What's in it for Them) rule. People need to know what they get out of a process and what they can contribute. Share this.



C Facilitate

* Build evaluation capacity in your community. Become a leader in putting program evaluation on the table with community stakeholders. This leads to strong partnerships and stronger community programs.

* Reflect on the role of a facilitator. When facilitating a process, what are your strengths? Your challenges? What are your views about "group process"? How do you incorporate others' values, views, and interests?

* Find and cultivate people who want to learn.



~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ * ~ *



VI Conclusion



Involving stakeholders, evaluating programs, and using logic models increases the possibility that the results of the program evaluation will be utilized. Patton (1997) states that "utilization-focused evaluation is a process for making decisions about these issues in collaboration with an identified group of primary users focusing on their intended uses of evaluation." It makes sense that if we are going to go to all the work of involving stakeholders that we want to make sure that the results of the evaluation are used. The process presented here increases the odds of using the results of the evaluation as stakeholders have a vested interest.

VII References



Canadian Evaluation Society (1999). Building an Evaluation Framework. Essential Skills

Series, Toronto, Ontario, 1-70.



Canadian Evaluation Society. (1999). Evaluating for Results. Essential Skills Series,

Toronto, Ontario, 1-80.



Kahan, B., Goodstadt, M. & Rajkamar, E. (1999). Best Practices in Health Promotion. Ontario Health Promotion E-Bulletin, 100.1, 1-7.



Kirkpatrick, S. (2001) The Program Logic Model: What, Why and How? Ontario Health

Promotion E-Bulletin, #229.1, 1-4.



Milstein, R. & Wetterhall, S. (1999). Framework for Program Evaluation in Public

Health. Centres for Disease Control and Prevention. Retrieved on September 5, 2003 from http://www.cdc.gov/epo/mmwr/preview/mmwrhtml/rr4811a1.htm.



Patton, M.Q. (1997). Utilization-focused Evaluation, 3rd edition. SAGE Publications,

Thousand Oaks, California, 1-431.



Porteous, N.L., Sheldrick, B.J. & Stewart, P.J. (1997) Program Evaluation Tool Kit a

blue print for public health management. Public Health Research Education and Devleopment Program, Ottawa-Carlton Health Department.



The Health Communicaton Unit. (2002). Evaluating Health Promotion Programs,

Version 3.3, Centre for Health Promotion, University of Toronto, 1-102.



Rush, B. & Ogborne, A. (1991). Program logic models: Expanding their role and structure for program planning and evaluation. The Canadian Journal of Program Evaluation, Vol 6, Number 2, 93-105.