Back to top

Ever wonder whether policy makes a difference? Conduct an evaluation to find out

Contents

I Introduction
II The Ten Steps
III Conclusion
IV References
V Additional Resources

--Submitted by Allison Meserve and Kim Bergeron, Health Promotion Consultants, Health Promotion Capacity Building Services at Public Health Ontario

I Introduction

A policy is a course of action that drives decision making to set priorities and allocate resources, and is often developed to address an issue or a problem. [1] Healthy public policies are developed to improve physical, social, economic or environmental conditions. [1] These types of policies can typically be categorized into three broad levels – government (federal, provincial and municipal), public institutions (e.g. hospitals, schools, day care centres) and organizations/workplaces. [2] Building healthy public policy is one of five key action areas under the Ottawa Charter for Health Promotion. [3]

Evaluation “is the systematic gathering, analysis, and reporting of data about a program to assist in decision-making”. [4] Policy evaluations apply evaluation principles and methods to examine the content, implementation or impact of a policy. [5] We evaluate policies to:

  • Document and inform development, adoption and implementation.
  • Determine effectiveness and build an evidence base.
  • Gauge support.
  • Assess compliance.
  • Inform future efforts. [6]

This article will highlight how to use Public Health Ontario’s Ten Steps for Conducting an Evaluation to evaluate a healthy public policy. [7]

II The Ten Steps

Step 1: Clarify what is to be evaluated

An evaluation cannot be undertaken if what is being evaluated and the purpose of the evaluation is unclear. An existing policy most likely has an underlying theory of change or logic, so the first step is to determine the policy’s goals, populations of interest, outcomes, outputs and activities. This is often identified by developing a logic model. [8] If you are at the beginning stages of policy development you may want to review existing policies from other jurisdictions or organizations to determine the logic model. The importance of this step is to ensure that there is agreement on the evaluation’s purpose and how results will be used. 

The purpose of the evaluation is usually a broad statement, rather than specific questions to be addressed. Examples include:

  • The purpose of this evaluation is to determine how other jurisdictions have implemented policies for calorie-labeling at fast-food restaurants. Results will be used to create a policy for the municipality.
  • The evaluation will examine whether the intended outcomes of the organization’s food policy for meetings have been attained. Based on the results, the executive team will determine if the policy should remain the same, be changed, or be eliminated. 

Step 2: Engage stakeholders

Stakeholders are anyone with an interest in the evaluation results. Begin by defining stakeholders and their interests or expectations. Some stakeholders often involved with healthy public policy evaluations include:

  • Decision makers (e.g., government officials, business owners).
  • Those affected positively and or negatively by the policy.
  • Enforcement and first responder agencies (e.g., public health staff, municipal staff, police).
  • Subject matter, policy or evaluation experts. [6]

Step 3: Assess resources and evaluability

Unfortunately, resources can be a major factor in decisions made in subsequent steps. Evaluation resources include funds, staff time, in-kind support (such as consultant or expert support, equipment, software), materials, and the timeline for the evaluation. A realistic assessment of available resources will ensure that you are able to implement a logical and thoughtful evaluation as planned.

This step includes an assessment of whether you are ready to conduct a useful evaluation by using four standards for determining readiness, drawn from program evaluation literature:

  1. Defined, agreed-upon, and realistic goals (existing or planned policies).
  2. Well-defined information needs (agreement on evaluation focus).
  3. Obtainable evaluation data.
  4. Intended users willing and able to use the evaluation information. [9]

If these standards cannot be met, it may not be the right time for an evaluation. You may be better served by waiting, or by setting monitoring or performance measurement systems in place (for example, monitoring enforcement efforts through the number of citations issued).

Step 4: Determine your evaluation questions

When determining the purpose of the evaluation with your stakeholders, you probably generated some initial questions to address through an evaluation. In this step, you will further prioritize and refine these questions based on time and resources, decisions to be made based on the results, and existing information or evaluations.

Evaluation questions are often then organized by type. The National Center for Injury Prevention and Control at the Centers for Disease Control and Prevention describes three policy evaluation types:

  1. Content: How was an existing policy developed – under what conditions or context? How do different policies addressing the same issue compare? Are the necessary components (goals, implementation and underlying logic) articulated?
  2. Implementation: Was the policy implemented as intended? What were barriers and facilitators to implementation? Is there continued support for the policy?
  3. Impact: Were the intended short-term and long-term outcomes produced? Any unintended consequences? [5]

It is recommended the evaluation team engage in facilitated dialogue to identify the evaluation questions first, which will identify the type of evaluation needed.

Step 5: Determine methods of measurement and procedures

This step determines the most feasible and credible methods of measurement based on available resources, evaluation questions and stakeholder desires. In this step you will determine:

  • What you will measure (indicators).
  • When you will collect data.
  • How you will collect data (qualitative, quantitative or mixed methods).
  • Who you will collect data from (staff, specific sub-groups or sample of population of interest).
  • Who will use the data. [7]

Based on responses to these questions, procedures for recruitment, data collection and analysis can be developed. Any ethical concerns regarding the data collection methods will be addressed at this time as well.

Some quantitative data collection which can be used in policy evaluation include questionnaires and surveys, media tracking, existing research, surveillance systems, and analysis of existing administrative data. Possible qualitative data collection methods include content analysis of the policy or meeting minutes, key informant interviews, case studies, and retrospective review of charts or case notes. [6] 

Step 6: Develop evaluation plan

It may seem late in the process to develop an evaluation plan, but most information needed has been developed in previous steps. Creating the plan later also allows you to create a realistic plan based on discussions and decisions to date. The plan includes the purpose of the evaluation, methods, procedures, a data collection matrix (e.g., evaluation questions, indicators, methods, data source, timing, roles and responsibilities, budget), and ethical considerations (e.g., informed consent, confidentiality and anonymity, cultural sensitivity). [10] An evaluation plan acts as a “to do” list for the process, ensuring everyone has a clear understanding of what needs to happen, how, and when.

Step 7: Collect data

The focus of this step is implementing the evaluation plan. Evaluation results and recommendations depend upon data quality. Quality can be ensured by pilot testing data collection or extraction tools, adequately training data collectors, meeting frequently at the start to ensure problems are addressed in a timely way, and checking through the data early to ensure procedures are being followed correctly. [10]

Step 8: Process data and analyze results

Now is the time to synthesize the information from all data sources, to permit interpretation and to answer your evaluation questions. This step requires you to verify the quality of the data collected, organize your data and then analyze it. [10] Therefore, accurate data entry is very important and requires planning, since a lot of resources are invested in data management and analysis. Depending on the type of data collected (quantitative or qualitative), procedures for processing and analyzing it will vary. For example, for evaluations with quantitative data, statistical analyses will be used to interpret results, whereas for qualitative analysis themes of particular interest can be identified using the evaluation questions. [10]

Step 9: Interpret and disseminate results

Once initial analyses are complete, you can work with stakeholders to interpret the results. It is important to anchor recommendations in the evaluation results. Sharing the results provides an opportunity to demonstrate innovation and creativity beyond writing an evaluation report. Consider developing communication products such as an infographic, short informational video, a slide show presentation, or a briefing note. When communicating results, especially to policymakers, keep in mind they may have limited time for review of evaluation findings. To increase the likelihood policymakers will use evaluation results, consider:

  • Framing findings in relation to local context.
  • Providing real-life illustrations.
  • Visualizing statistical data clearly.
  • Communicating with brevity and clarity.
  • Conducting cost-benefit analyses. [6]

Step 10: Apply evaluation findings

This step is about taking action. Now is the time to consider which actions are needed. One way to do this is to consider: What was learned? How can this information be used? What concrete changes are needed? Answers to these questions may generate a number of recommended actions related to the policy, so you will need to prioritize the most important and feasible changes to implement. [10] Once actions are prioritized, develop a plan to implement strategic decisions about the policy.

At this time you can also evaluate your evaluation by asking the evaluation stakeholders to reflect on the process and/or outcomes of the evaluation itself. You can stimulate discussion by addressing the implementation and actual or anticipated outcomes of the evaluation. You can use the evaluation process itself to build shared meaning and understanding among the groups involved.

 

III Conclusion

Policy evaluation can be challenging, time-consuming, and can require many resources. However, evaluation of a policy’s content, implementation, or impact improves existing healthy public policies and builds the evidence base for continuing to use policies as a health promotion strategy. For more information about this step process and the concepts presented, please watch the OHPE for the release of our introductory evaluation workbook. If you are considering conducting a policy evaluation, but would like further support, please e-mail Health Promotion Capacity Building Services (HPCB) at [email protected] for a tailored consultation.

 

IV References

1 Milo N. Glossary: Healthy public policy. Journal of Epidemiology and Community Health. 2001;55(9):622-3.

2. Ontario Agency for Health Protection and Promotion (Public Health Ontario). Evaluating healthy public policies: A ten step process. Webinar-February 6, 2015. Toronto, ON: Queens Printer of Ontario; 2015. Available from: http://www.publichealthontario.ca/en/LearningAndDevelopment/Events/Pages/Evaluating_Healthy_Public_Policies.aspx

3. World Health Organization. The Ottawa Charter for Health Promotion: First International Conference on Health Promotion [Internet]. Geneva: World Health Organization; 1986 [cited 2014 Dec 2]. Available from: http://www.who.int/healthpromotion/conferences/previous/ottawa/en/

4. Ministry of Health and Long-Term Care. Ontario Public Health Standards 2008. Toronto, ON: Queen's Printer for Ontario; 2014. Available from: http://www.health.gov.on.ca/en/pro/programs/publichealth/oph_standards/docs/ophs_2008.pdf

5. National Center for Injury Prevention and Control. Step by step - evaluating violence and injury prevention policies. Brief 1: Overview of policy evaluation [Internet]. Atlanta, GA: Centers for Disease Control and Protection; 2013 [cited 2014 Dec 3]. Available from: http://www.cdc.gov/injury/pdfs/policy/Brief%201-a.pdf

6. Centers for Disease Control and Prevention, Office of the Associate Director for Policy. Policy evaluation in public health: CDC's Office on Smoking and Health Surveillance and Evaluation Webinar. January 29, 2015. Atlanta, GA: Centers for Disease Control and Prevention; 2015. Available from: http://www.tacenters.emory.edu/documents/netconference_docs/SE2015/012915Policy-Evaluation_final.pdf

7. Ontario Agency for Health Protection and Promotion (Public Health Ontario). At A Glance: The ten steps for conducting an evaluation. 1st ed. Toronto, ON: Queen’s Printer for Ontario; 2015. Available from: http://www.publichealthontario.ca/en/eRepository/At_A_Glance_Evaluation_2015.pdf

8. Morestin F, Castonguay J.  Constructing a Logic Model for a Healthy Public Policy: Why and How? Montréal, Québec: National Collaborating Centre for Healthy Public Policy; 2013. Available from: http://www.ncchpp.ca/docs/LogicModeleLogique_En.pdf

9. Wholey JS. Exploratory evaluation. In: Wholey JS, Hatry HP, Newcomer KE, editors. Handbook of practical program evaluation. 3rd ed. San Francisco, CA: Jossey-Bass; 2010.

10. Ontario Agency for Health Protection and Promotion (Public Health Ontario). Evaluating health promotion programs: introductory workbook. Toronto, ON: Queen's Printer for Ontario; 2015.

 

V Additional Resources

A key step in any evaluation is to determine what you are evaluating. The National Collaborating Centre for Healthy Public Policy developed a useful resource: Constructing a logical model for a healthy public policy: Why and how? available at:  http://www.ncchpp.ca/docs/LogicModeleLogique_En.pdf

The National Center for Injury Prevention and Control, Centers for Disease Control and Prevention created a series of briefs on policy evaluation available at: http://www.cdc.gov/injury/about/policy/evaluation.html

Policy evaluations require a diverse set of stakeholders. The Robert Wood Johnson Foundation created A Practical Guide for Engaging Stakeholders in Developing Key Questions: http://www.rwjf.org/content/dam/web-assets/2009/01/a-practical-guide-for-engaging-stakeholders-in-developing-evalua to aid in the process of engaging stakeholders.

An example of a briefing note used to report policy evaluation findings is available here: https://www.pembina.org/reports/ab-climate-policy-briefing-note-12162011.pdf

Examples of policy evaluation

Restricting hours of alcohol sales to prevent excessive alcohol consumption and related injuries. Available from: http://www.cdc.gov/injury/pdfs/policy/Appendices‐a.pdf

Smoke-free housing policy evaluation-findings from the 2013 Waterloo Region Housing and Region of Waterloo Community Housing Inc. household tenant survey. Available from:

https://uwaterloo.ca/propel/sites/ca.propel/files/uploads/files/smoke_free_housing_policy_tenant_survey_2013.pdf   

Evaluating the implementation and active living impacts of state government policy designed to create walkable neighborhoods in Perth, Western Australia. Available from: http://www.ajhpcontents.com/doi/pdf/10.4278/ajhp.130503-QUAN-226

On today’s menu: Evaluation of a menu labelling initiative in hospital cafeterias in Ottawa, Canada. Available from: http://resources.cpha.ca/CPHA/Conf/Data/2014/a14-252e.pptx