Contents
I Introduction
II Overview of web properties study
III Methodology
IV Summary of findings
–Submitted by Brian Cugelman. PhD, Online strategy and research consultant, AlterSpark Consulting
I Introduction
Organizations produce publications for a number of reasons: Publications help organizations disseminate critical information to their stakeholders, support their constituents, and help achieve core goals. In some cases, the success of an organization rests on the success of its publications. Organizations such as research groups or think tanks, and organizations with education mandates often treat the publication production process as a core activity.
Yet it’s not easy to produce impactful, high quality publications. The production process can take many months of careful research, internal discussion, external consultations, fact checking, and production work. Not to mention, the huge effort involved in disseminating publications.
Despite these huge efforts, it is surprising that few organizations have a strong understanding of their readership and how readers regard their publications. Many organizations have access to ad hoc feedback, but few have conducted in-depth studies of their readership, and few fully understand the demographics of their readers, how their readers regard their publications, and which publications have the most or least impact.
II Overview of web properties study
The Health Communication Unit (THCU) is a health promotion unit (formerly at the University of Toronto), with a mandate to provide training and support in health communication, planning, evaluation, and policy change. Until recently, THCU staff largely relied on anecdotal and occasional survey feedback to understand how their clients regarded their publications. So, to gain more insight into its readership, THCU commissioned an independent study of its web properties and key publications.
“This study gave us a deeper understanding of our online readership, far beyond the simple figures provided by Google Analytics,” said Larry Hershfield, Manager of THCU from 1993-2011 (currently still involved through his position at Public Health Ontario). “For the first time ever, we had hard evidence that our online outreach and publications were being promoted by a large and diverse group of Canadian health and non-profit organizations. We were glad to see many health and safety organizations were treating our publications as valuable resources, and that 82% of the organizations were from Ontario.”
Overall, the study revealed how organizations regarded THCU’s publications, which publications were gaining the most traction among stakeholders, which topics were most or least popular, and which types of organizations were most influenced by their publications. The network diagram above illustrates THCU’s online network, which comprises three broad groups: (1) Canadian health and non-profit organizations, (2) US health organizations, and (3) workplace health and safety organizations.
The study was performed by AlterSpark Consulting (http://www.alterspark.com) and the Statistical Cybermetric Research Group (http://cybermetrics.wlv.ac.uk). Dr. Brian Cugelman, founder of AlterSpark Consulting and EvaluationPal (http://www.evaluationpal.com) oversaw this study.
III Methodology
The research team employed an innovative methodology based on practices used by social scientists who study online phenomena. Their study employed a large-scale data collection process, random sampling, and content analysis of hyperlink and online properties.
“We developed this research process to help organizations understand their online readership, to help them produce publications that matter to their audiences”, said Dr. Cugelman. “By employing online social science research methods, we were able to show THCU their online environment, their stakeholders, and their publication’s impact. It’s one thing to believe your publications are making an impact. It’s another to actually see it in detail.”
“Our methodology goes beyond the simple metrics commonly available in web analytics packages,” Dr. Cugelman continued. “Web analytics tools can reveal how often a publication is being viewed over time. However, these tools do not reveal who is talking about that publication on third party websites, how people regard it, or whether these publications are making an impact on their target audiences. By combining large-scale data collection with random sampling, we were able to take a look at THCU’s overall impact in cyberspace. Our team hopes this study will help THCU continue producing relevant and valued works.”
IV Summary of findings
Following is a summary of the findings from the study:
The vast majority of citing host websites come from Canadian sources (63%) of which 82% are from Ontario. Following this, USA sources (10%) are the second most popular.
THCU’s online network is roughly comprised of three major groupings (1) Canadian health and non-profit organizations, (2) US health organizations, and (3) Workplace health and safety organizations.
THCU is largely sustained by a larger number of coalitions, academic institutes, and associations with medium levels of online influence. However, there are also a small number of highly influential organizations from academia, associations, health agencies, and government.
THCU’s publications and websites are largely considered resources, and frequently promoted as a resource by highly credible actors with significant online influence.
Workplace health and safety appears to be one of the most popular themes. A significant part of THCU’s online network is comprised of health and safety websites. THCU’s publication with the largest number of citations is the Comprehensive Workplace Health Promotion which was just published in 2010.
The impact of THCU largely appears to be on health organizations such as associations, academic, and health agencies. About 8% of the citations were from irrelevant web services, the majority are from key players in the health community.
Website operators appear to place importance on the pages that cite THCU. They are often posted these pages just two levels from the home page. Moreover, 17% of these citations are exclusively focused on THCU.