Emergency Medicine and Critical Care Blogs and Podcasts: Establishing an International Consensus on Quality

doi:10.1016/j.annemergmed.2015.03.002

Annals of Emergency Medicine

Volume 66, Issue 4, October 2015, Pages 396-402.e4

https://doi.org/10.1016/j.annemergmed.2015.03.002 Get rights and content

Study objective

This study identified the most important quality indicators for online educational resources such as blogs and podcasts.

Methods

A modified Delphi process that included 2 iterative surveys was used to build expert consensus on a previously defined list of 151 quality indicators divided into 3 themes: credibility, content, and design. Aggregate social media indicators were used to identify an expert population of editors from a defined list of emergency medicine and critical care blogs and podcasts. Survey 1 consisted of the quality indicators and a 7-point Likert scale. The mean score for each quality indicator was included in survey 2, which asked participants whether to “include” or “not include” each quality indicator. The cut point for consensus was defined at greater than 70% “include.”

Results

Eighty-three percent (20/24) of bloggers and 90.9% (20/22) of podcasters completed survey 1 and 90% (18/20) of bloggers and podcasters completed survey 2. The 70% inclusion criteria were met by 44 and 80 quality indicators for bloggers and podcasters, respectively. Post hoc, a 90% cutoff was used to identify a list of 14 and 26 quality indicators for bloggers and podcasters, respectively.

Conclusion

The relative importance of quality indicators for emergency medicine blogs and podcasts was determined. This will be helpful for resource producers trying to improve their blogs or podcasts and for learners, educators, and academic leaders assessing their quality. These results will inform broader validation studies and attempts to develop user-friendly assessment instruments for these resources.

Introduction

During the past two decades, there has been a substantial increase in the number of digital medical education resources, particularly in emergency medicine and critical care.¹ The proliferation of resources such as blogs and podcasts, driven in part by the popularity of the free open-access medical education movement,² has paralleled their increasing use by emergency medicine residents.3, 4 Recent surveys from Canada³ and the United States⁴ have found high rates of use of resources including blogs, podcasts, video podcasts, and wikis.

Editor’s Capsule Summary

What is already known on this topic

Podcasts and blogs are an increasingly popular way for emergency physicians to keep up with changes in their field.

What question this study addressed

Which items, from a list of 151 possibilities, do experts in emergency medicine social media believe are the best indicators of quality?

What this study adds to our knowledge

The experts used a modified Delphi method to select 31 items they deemed particularly important. They did so with a high degree of agreement despite the exercise being conducted without an explicit definition of “quality.”

How this is relevant to clinical practice

This will not change practice but provides the basis for further research into how physicians may identify the most reliable social media sources.

Despite their widespread use, these resources have been met with caution by the academic community because of uncertainty about their influence and quality.5, 6, 7, 8, 9, 10, 11, 12 Despite their parallels to traditional resources such as textbooks and lectures,¹³ few tools have been developed to assess blogs and podcasts for use in medical education. The Social Media Index is a tool that aggregates social media and Web-analytic metrics to rank Web sites by their relative impact.¹⁴ When applied to journals, both the Social Media Index and its components were found to correlate with traditional impact metrics.¹⁴

That said, impact is not equivalent to quality. Having a method to assess the quality of novel educational resources such as blogs and podcasts could make it easier for learners to assess their quality, educators to recommend appropriate resources to their learners, resource producers to improve the quality of their products, and academic leaders to assess the contributions of resource producers. In a previous study, we conducted a literature search, qualitative analysis, and series of focus groups to develop a list of quality indicators that are applicable to online educational resources such as blogs and podcasts.¹⁵ The qualitative analysis subcategorized an extensive list of quality indicators into three major themes (credibility, content, and design) and 151 items.¹⁵

The primary focus of this study was to determine the relative importance of each quality indicator for blogs and podcasts, using an iterative, consensus-building, modified Delphi process among an expert group of emergency medicine and critical care bloggers and podcasters.

Section snippets

Materials and Methods

This study received ethics approval from the Hamilton Integrated Research Ethics Board in Hamilton, ON, Canada.

Results

A total of 22 podcasters and 24 bloggers (2 podcasts and 4 blogs had more than 1 editor) were invited to complete the surveys (Table 1); 83.3% (20/24) of bloggers and 90.9% (20/22) of podcasters completed survey 1, whereas 90% of the invited bloggers (18/20) and podcasters (18/20) completed survey 2. The Web sites of the bloggers and podcasters who were invited to participate according to the rankings of the Social Media Index are outlined in the Figure. The majority of participants had greater

Limitations

Rather than using a snowball technique to identify experts, as is often done in Delphi surveys,16, 17, 18 we used the Social Media Index. Although this was beneficial because it provided a reproducible method of identifying the highest-impact bloggers and podcasters, it may be viewed as a limitation because it has not previously been used for this purpose.¹⁴ However, as outlined in Table 1, the Social Media Index effectively identified groups of bloggers and podcasters with substantial,

Discussion

Because blogs and podcasts are proliferating¹ and increasingly being used by learners,3, 4 assessing their quality is an important task. Through the widespread adoption of free open-access medical education, emergency medicine and critical care has become a leader in online medical education,² and to our knowledge this study represents the first collaborative effort by experts in this field to develop consensus around quality. These efforts may be informative to other medical specialties as the

References (25)

M.H. Gelula
Effective lecture presentation skills
Surg Neurol
(1997)
M. Cadogan et al.
Free open access meducation (FOAM): the rise of emergency medicine and critical care blogs and podcasts (2002-2013)
Emerg Med J
(2014)
C.P. Nickson et al.
Free open access medical education (FOAM) for the emergency physician
Emerg Med Australas
(2014)
E. Purdy et al.
The use of free online educational resources by Canadian emergency medicine residents and program directors
CJEM
(2015)
M. Mallin et al.
A survey of the current utilization of asynchronous education among emergency medicine residents in the United States
Acad Med
(2014)
Genes N. Pro/con: why #FOAMed is not essential to EM education. Emerg Physicians Mon. 2014. Available at:...
W.W. Savage
The transom: you can’t spill mustard on a blog
J Sch Publ
(2006)
F.J. Grajales et al.
Social media: a review and tutorial of applications in medicine and health care
J Med Internet Res
(2014)
T. Brabazon
The Google effect: Googling, blogging, wikis and the flattening of expertise
Libri
(2006)
G. Kirkup
Academic blogging: academic practice and academic identity
London Rev Educ
(2010)

Ostermayer D. Podcasts are great, but.... ACEP Now. 2013. Available at: http://www.acepnow.com/article/podcasts-great/....

L. Zanussi et al.

Podcasting in medical education: can we turn this toy into an effective learning tool?

Adv Health Sci Educ Theory Pract

(2012)

Cited by (61)

Digital Impact Factor: A Quality Index for Educational Blogs and Podcasts in Emergency Medicine and Critical Care
2023, Annals of Emergency Medicine
Given the popularity of educational blogs and podcasts in medicine, learners and educators need tools to identify trusted and impactful sites. The Social Media Index was a multi-sourced formula to rank the effect of emergency medicine and critical care blogs. In 2022, a key data point for the Social Media Index became unavailable. This bibliometric study aimed to develop a new measure, the Digital Impact Factor, as a replacement.
The Digital Impact Factor incorporated modern measures of website authority and reach. This formula was applied to a cross-sectional study of active emergency medicine and critical care blogs and podcasts. For each website, we generated a Digital Impact Factor score based on Ahrefs Domain Rating and the follower count of the websites’ pages from 8 social media platforms. A series of Spearman correlations provided evidence of association by comparing a rank-ordered list to rank lists derived from the Social Media Index over the last 5 years. The Bland-Altman analysis assessed for agreement.
The authors identified 88 relevant websites with a median Ahrefs Domain Rating of 28 (range 0 to 71, maximum 100) and total social media followership count across 8 platforms of 1,828,557. The Domain Rating and individual social media followership scores were normalized based on the highest recorded values to yield the Digital Impact Factor (median 4.57; range 0.02 to 9.50, maximum 10). The correlation between the 2022 Digital Impact Factor and the 2021 Social Media Index was 0.94 (95% confidence interval 0.89 to 0.97; p<.001; n=41 rankings correlated), suggesting that they measure similar constructs. The Bland-Altman plot also demonstrated fair agreement between the 2 scores.
The Digital Impact Factor is a measure of the relative effect of educational blogs and podcasts within emergency medicine and critical care.
Identifying Social Media Competencies for Health Professionals: An International Modified Delphi Study to Determine Consensus for Curricular Design
2022, Annals of Emergency Medicine
Citation Excerpt :
This study was designed to inform the development of a formal social media curriculum for medical professionals. We conducted a multiphase, expert, modified Delphi study, similar to previous studies conducted by members of our team.32,33 We sought expert perspectives framed by Kraiger's Knowledge, Skills, and Attitudes (KSA) framework,34 because it not only relates to social media in general but also asked specific questions about social media platforms.
The use of social media by health professionals is widespread. However, there is a lack of training to support the effective use of these novel platforms that account for the nuances of an effective health and research communication. We sought to identify the competencies needed by health care professionals to develop an effective social media presence as a medical professional, with the goal of building a social media curriculum.
We conducted a modified Delphi study, utilizing Kraiger’s Knowledge, Skills, and Attitudes framework to identify appropriate items for inclusion in a social media curriculum targeted at health care professionals. Experts in this space were defined as health care professionals who had delivered workshops, published papers, or developed prominent social media tools/accounts. They were recruited through a multimodal campaign to complete a series of 3 survey rounds designed to build consensus. In keeping with prior studies, a threshold of 80% endorsement was used for inclusion in the final list of items.
Ninety-eight participants met the expert criteria and were invited to participate in the study. Of the 98 participants, 92 (94%) experts completed the first round; of the 92 experts who completed the first round, 83 (90%) completed the second round; and of the 83 experts who completed the second round, 81 (98%) completed the third round of the Delphi study. Eighteen new items were suggested in the first survey and incorporated into the study. A total of 46 items met the 80% inclusion threshold.
We identified 46 items that were believed to be important for health care professionals using social media. This list should inform the development of curricular activities and objectives.
Systematic online academic resource (SOAR) review: Pediatric respiratory infectious disease
2024, AEM Education and Training
Quality Indicators of Online Cancer Communities from the Perspective of Cancer-Affected People
2023, Proceedings of the Annual Hawaii International Conference on System Sciences
An Evaluation of Emergency Medicine Core Content Covered by Podcasts
2023, Western Journal of Emergency Medicine
Defining new roles and competencies for administrative staff and faculty in the age of competency-based medical education
2023, Medical Teacher

View all citing articles on Scopus

: Please see page 397 for the Editor’s Capsule Summary of this article.

: Supervising editor: Anna Olson, MD

: Author contributions: BT, TMC, JLS, and ML conceived the study and designed the trial. BT, QSP, and WKM supervised and collected data. BT and JLS conducted the data analysis and managed the data. BT drafted the article and all authors contributed substantially to its revision. BT takes responsibility for the paper as a whole.

: Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article as per ICMJE conflict of interest guidelines (see www.icmje.org). Dr. Thoma is the editor in chief of BoringEM, an associate editor at Academic Life in EM, and a social media editor at the Canadian Journal of Emergency Medicine, and has previously contributed to Life in the Fast Lane. Dr. Chan is the managing editor of BoringEM, an associate editor at Academic Life in EM, and a social media editor at the Canadian Journal of Emergency Medicine. Dr. Milne is the editor in chief or the Skeptic's Guide to Emergency Medicine podcast. Dr. Lin is the editor in chief at Academic Life in EM and a deputy editor for DynaMed (a subscription-based online clinical reference).

: A feedback survey is available with each research article published on the Web at www.annemergmed.com.

: A podcast for this article is available at www.annemergmed.com.

View full text

Education/original researchEmergency Medicine and Critical Care Blogs and Podcasts: Establishing an International Consensus on Quality

Study objective

Methods

Results

Conclusion

Introduction

Editor’s Capsule Summary

Section snippets

Materials and Methods

Results

Limitations

Discussion

Surg Neurol

Free open access meducation (FOAM): the rise of emergency medicine and critical care blogs and podcasts (2002-2013)

Emerg Med J

Free open access medical education (FOAM) for the emergency physician

Emerg Med Australas

The use of free online educational resources by Canadian emergency medicine residents and program directors

CJEM

A survey of the current utilization of asynchronous education among emergency medicine residents in the United States

Acad Med

The transom: you can’t spill mustard on a blog

J Sch Publ

Social media: a review and tutorial of applications in medicine and health care

J Med Internet Res

The Google effect: Googling, blogging, wikis and the flattening of expertise

Libri

Academic blogging: academic practice and academic identity

London Rev Educ

Podcasting in medical education: can we turn this toy into an effective learning tool?

Adv Health Sci Educ Theory Pract

Education/original research
Emergency Medicine and Critical Care Blogs and Podcasts: Establishing an International Consensus on Quality