Back to top

Participatory Evaluation: The case for setting a place for ‘young people’ at the ‘adult’ table

Contents

I Introduction
II Principles of Youth Engagement
III Participatory Evaluation Benefits Youth
IV Participatory Data Collection and Analysis
V Facilitators for Participatory Data Collection and Analysis
VI Conclusion

--Submitted by Allison Meserve, Health Promotion Consultant, Health Promotion Capacity Building, Public Health Ontario and Kristy Ste Marie, Acting Manager, YATI & Youth Engagement, Youth Advocacy Training Institute, Ontario Lung Association

I Introduction

Too often, even in youth-targeted interventions, youth are not considered a stakeholder of evaluations.[1] They are often engaged as subjects, respondents and informants, but are not routinely included as partners in evaluations of interventions seeking to improve their own health. Although youth programs are committed to engaging youth in all levels of their programming, many struggle with how to meaningfully engage young people in evaluation. Participatory evaluation is an evaluation theory that calls for the inclusion of all who have a stake in the outcome of health promotion interventions in the evaluation process. [1,2] Youth can be involved as stakeholders in all phases of an evaluation—planning, implementation and utilization. This feature article focuses on how to involve youth aged 12–24 in the implementation phase of the evaluation.

II Principles of Youth Engagement

In 2010, the Youth Engagement Advisory Group (YEAG) was created to advise the then Ministry of Health Promotion and Sport on new models for engaging Ontario’s youth in tobacco and health promotion initiatives. This group identified 11 key principles to youth engagement: inclusiveness, positive youth development, accountability, operational practices, strengths-based approach, flexibility and innovation, space for youth, transparency, sustainability of resources, cross-sector alignment, and collaboration. [3] These principles are often seen as the foundation to applying youth development approaches.

The Principles of Youth Engagement also can inform and guide participatory evaluation with youth. For example, positive youth development refers to intentional efforts of youth, adults, schools, communities, and government agencies to provide opportunities for youth to enhance their interests, skills, and abilities. This development can be influenced by allies (such as schools and community groups) as they provide services, opportunities, and supports to encourage youth development in a positive direction. [3] In a participatory evaluation approach, youth are not seen as recipients of evaluation processes created by adults, but rather as evaluators themselves. Participatory evaluation provides young people with opportunities to develop and enhance new skills as well as influence and shape the evaluation. [1]  

III Participatory Evaluation Benefits Youth

A participatory approach to an evaluation can be thought of not only as a principle of the evaluation, but as a key component of the intervention itself. As Springett (2000) explains, “The strength of participatory approaches lies in their contribution to empowerment and social change. In other words, participatory evaluation is a health promotion strategy.” [2, p. 83]

Some of the benefits to youth include:

  • Reinforced leadership, communication, negotiation and collaboration skills
  • Increased understanding of research and development of related skills (including how to design and plan a project, data collection methodologies and tool development, data analysis and critical appraisal)
  • Enhanced skills and opportunities for self-reflection and development of critical consciousness
  • Improved self-confidence
  • Increased understanding of particular health topics and the “power to make positive health choices” [4, p. 6)
  • Expanded social network (social capital)
  • Increased understanding of diverse viewpoints and experiences among their peers
  • Empowered role within their own communities. [1, 4–6]

It is important to note that the benefits of participatory evaluation are not limited to youth.  Participatory evaluation benefits young people, adult facilitators, programs, communities, and society.[1,4,7] The quality of the evaluation is improved as well by increasing participation of hard to reach populations, improving the quality and completeness of the data collected, as well as providing alternative viewpoints when interpreting the data and developing recommendations. [1,4,5]  

IV Participatory Data Collection and Analysis

Any data collection method can be transformed into a participatory method. We will describe below how to adapt two common data collection methods, surveys and interviews, to include youth in the instrument design, collection and analysis of data.

A questionnaire is “a set of questions administered consistently to different respondents”. [8, p. 2] Youth can be involved in all phases of an evaluation with questionnaires – including designing the questions to be asked, recruiting participants to complete the survey, analyzing the data, developing recommendations based on the results, as well as presenting the results.

To illustrate, for a needs assessment carried out in Toronto, a Youth Advisory Committee (YAC) made up of 12 teens aged 13–17 designed a survey and study protocol to determine how to improve sexual health services at a network of clinics targeting youth. Prior to the development of the survey, the YAC members received training on “topics related to qualitative and quantitative methodologies, sexual health, anti-oppression analysis, and the social determinants of health.” [5, p. 137] At the end of each training or working session, youth were provided an opportunity to provide feedback on the session which was incorporated into the design of subsequent sessions. The YAC members worked with the researchers to develop and eventually administer the survey. [5]

An interview is “a one-on-one, guided conversation used to gather in-depth information about someone’s feelings, thoughts and experiences related to a specific subject.” [9, p. 2] In another Canadian example, First Nations high school students were co-researchers in a study to investigate tobacco knowledge and use among children and youth in their community (grades 2–12). The student co-researchers contributed to the development of the interview guide and research procedures, introduced the Photovoice (a type of participatory data collection method) study, and interviewed study participants through structured and semi-structured interviews. Prior to the interviews, the student co-researchers received five training sessions on research ethics, interviews (including developing guides, obtaining consent and interviewing skills), Photovoice methods, and recording equipment. The student co-researchers were supported by the research team as well as school staff. [4]

In addition to designing data collection tools and collecting data, youth can be involved in analyzing and interpreting evaluation data as well. The student co-researchers in the above example determined the major themes of the collected interviews and photos, as well as directed how and in what form the data should be shared with the rest of their community. [4] Large printouts of interview notes or transcripts also can be used in groups for participatory analysis. [10]

For quantitative data, youth have been taught how to enter and analyze data in SPSS (a statistical software program) [11] and can participate in analysis through data parties [12] or the use of data placemats. [13] Data parties and data placemats involve breaking up the quantitative analysis into smaller chunks of data, allowing for group discussion and interpretation of quantitative data at a more manageable and age-appropriate level. Including youth in the steps of data collection and analysis is guided by the Principles of Youth Engagement and ensures that some of the benefits of participatory evaluation, particularly in the area of job and life skill development, occur for the involved youth.

V Facilitators for Participatory Data Collection and Analysis

Although there are many benefits to including youth in your evaluation, there may be some challenges. The list below provides some considerations to increase the likelihood of success when involving youth in your evaluation:

  • Physical accessibility – place and time of meetings, payment for transportation costs
  • Cultural and age-appropriate accessibility – structure of trainings and meetings, how information is presented and disseminated
  • Building and maintaining trust
  • Meaningful involvement of youth
  • Allowing adequate time for training and the research process
  • Adults trained in research, group processes and youth-engagement principles. [1,4,5,14,15]

VI Conclusion

Including youth in all stages of an evaluation will improve not only your evaluation, but will empower the youth you are working with, increase their ability to be informed consumers of research and provide them with marketable skills. To learn more about additional participatory data collection methods, including digital storytelling and Most Significant Change, please attend our upcoming webinar on September 22nd. More information can be found at: http://sept222016hpcbseminar.eventbrite.ca.
 

References

  1. Flores KS. Youth participatory evaluation: Strategies for engaging young people. John Wiley & Sons; 2007.
  2. Springett J. Participatory approaches to evaluation in health promotion . Denmark: World Health Organization; 2001. Available from: http://www.euro.who.int/__data/assets/pdf_file/0007/108934/E73455.pdf
  3. Youth Engagement Advisory Group. Youth engagement principles. Toronto, ON: Ministry of Health Promotion and Sport; 2010. Available from: https://lin.ca/sites/default/files/attachments/Youth_Engagement_Principl...
  4. Jardine C, James A. Youth researching youth: benefits, limitations and ethical considerations within a participatory research process. Int J Circumpolar Health. 2012;71(0). Available from:  http://www.circumpolarhealthjournal.net/index.php/ijch/article/view/18415
  5. Flicker S, Guta A, Larkin J, Flynn S, Fridkin A, Travers R, et al. Survey design from the ground up: collaboratively creating the Toronto Teen Survey. Health Promot Prac. 2010;11(1):112-22. Available from: http://flicker.blog.yorku.ca/files/2013/02/FINALflickerMAR8.pdf
  6. Powers JL, Tiffany JS. Engaging youth in participatory research and evaluation. J Public Health Manag Pract. 2006;12:S79-87. Available from:
  7. Cousins JB, Whitmore E. Framing participatory evaluation. New directions for evaluation. 1998;1998(80):5-23.
  8. Ontario Centre of Excellence for Child and Youth Mental Health. Method mini-toolkit-collecting information using questionnaires: An overview. Toronto, ON: Ontario Centre of Excellence for Child and Youth Mental Health. Available from: http://www.excellenceforchildandyouth.ca/sites/default/files/docs/miniki...
  9. Ontario Centre of Excellence for Child and Youth Mental Health. Method mini-toolkit-qualitative interviewing: An overview. Toronto, ON: Ontario Centre of Excellence for Child and Youth Mental Health. Available from: http://www.excellenceforchildandyouth.ca/sites/default/files/docs/miniki...
  10. Jackson SF. A participatory group process to analyze qualitative data. Prog Community Health Partnersh: research, education, and action. 2008;2(2):161-70. Available from: https://www.researchgate.net/profile/Suzanne_Jackson/publication/4180886...
  11. Lee JP, Lipperman-Kreda S, Saephan S, Kirkpatrick S. Tobacco environment for Southeast Asian American youth: Results from a participatory research project. J Ethn Subst Abuse. 2013;12(1):30-50. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3661407/
  12. Franz NK. The data party: Involving stakeholders in meaningful data analysis. J Extension. 2013;51(10):1IAW2. Available from: http://www.joe.org/joe/2013february/iw2.php
  13. Pankaj V. Data Placemats: a DataViz technique to improve stakeholder understanding of evaluation results [Internet]. Presented at: Evaluation 2014: American Evaluation Association. 2014 Oct 16[cited 2015 Nov 2]; Denver, Co. Available from: http://www.slideshare.net/InnoNet_Eval/data-placemats-share
  14. Bradbury-Jones C, Taylor J. Engaging with children as co-researchers: challenges, counter-challenges and solutions. International Journal of Social Research Methodology. 2015;18(2):161-73.
  15. Ford T, Rasmus S, Allen J. Being useful: achieving indigenous youth involvement in a community-based participatory research project in Alaska. Int J Circumpolar Health. 2012;71. Available from: http://www.circumpolarhealthjournal.net/index.php/ijch/article/view/1841...  

Additional resources

Public Health Ontario has many resources, including a workbook and audio presentations, which provide guidance on how to conduct any type of evaluation which can be found on the PHO website: http://www.publichealthontario.ca/HPevaluation.

The Paloma Foundation and Wellesley Institute have developed a resource on participatory evaluation called Working together: The Paloma-Wellesley guide to participatory program evaluation. The guide is available at http://www.wellesleyinstitute.com/wp-content/uploads/2010/03/WorkingToge....

Public Profit, a consulting organization in the US, has developed a report on various ways to include stakeholders, including youth, in analyzing data. The report, Dabbling in the Data, can be found at http://www.publicprofit.net/Dabbling-In-The-Data.

Hallie Preskill and Darlene Russ-Eft developed recently updated their book, Building Evaluation Capacity: 72 Activities for Teaching and Training. May of the described evaluation capacity-building activities are feasible to use with older youth.

Youth Research and Evaluation Exchange, recently launched at York University, aims to increase the capacity of youth-serving organizations in research and evaluation. Their website (http://youthrex.com/) includes blog posts, resources and recorded webinars