Multi-institution validation of an emergency ultrasound image rating scale - A pilot study

Samuel H.F. Lam, John Bailitz, David Blehar, Brent A. Becker, Beatrice Hoffmann, Andrew S. Liteplo, Kumar Rajan, Michael Lambert

Research output: Contribution to journalArticle

Abstract

Background As bedside ultrasound (BUS) is being increasingly taught and incorporated into emergency medicine practice, measurement of BUS competency is becoming more important. The commonly adopted experiential approach to BUS competency has never been validated on a large scale, and has some limitations by design. Objective Our aim was to introduce and report preliminary testing of a novel emergency BUS image rating scale (URS). Methods Gallbladder BUS was selected as the test case. Twenty anonymous BUS image sets (still images and clips) were forwarded electronically to 16 reviewers (13 attendings, 3 fellows) at six training sites across the United States. Each reviewer rated the BUS sets using the pilot URS that consisted of three components, with numerical values assigned to each of the following aspects: Landmarks, Image Quality, and Annotations. Reviewers also decided whether or not each BUS set would be "Clinically Useful." Kendall τs were calculated as a measure of concordance among the reviewers. Results Among the 13 attendings, image review experience ranged from 2-15 years, 5-300 scans per week, and averaged 7.8 years and 60 images. Kendall τs for each aspect of the URS were: Landmarks: 0.55; Image Quality: 0.57; Annotation: 0.26; Total Score: 0.63, and Clinical Usefulness: 0.45. All URS elements correlated significantly with clinical usefulness (p < 0.001). The correlation coefficient between each attending reviewer and the entire group ranged from 0.48-0.69, and was independent of image review experience beyond fellowship training. Conclusion Our novel URS had moderate-to-good inter-rater agreement in this pilot study. Based on these results, the URS will be modified for use in future investigations.

Original languageEnglish (US)
Pages (from-to)32-39
Number of pages8
JournalJournal of Emergency Medicine
Volume49
Issue number1
DOIs
StatePublished - Jan 1 2015
Externally publishedYes

Fingerprint

Emergency Medicine
Gallbladder
Surgical Instruments
Emergencies

Keywords

  • bedside ultrasound
  • competency
  • image rating scale
  • ultrasound education

ASJC Scopus subject areas

  • Emergency Medicine

Cite this

Lam, S. H. F., Bailitz, J., Blehar, D., Becker, B. A., Hoffmann, B., Liteplo, A. S., ... Lambert, M. (2015). Multi-institution validation of an emergency ultrasound image rating scale - A pilot study. Journal of Emergency Medicine, 49(1), 32-39. https://doi.org/10.1016/j.jemermed.2015.01.010

Multi-institution validation of an emergency ultrasound image rating scale - A pilot study. / Lam, Samuel H.F.; Bailitz, John; Blehar, David; Becker, Brent A.; Hoffmann, Beatrice; Liteplo, Andrew S.; Rajan, Kumar; Lambert, Michael.

In: Journal of Emergency Medicine, Vol. 49, No. 1, 01.01.2015, p. 32-39.

Research output: Contribution to journalArticle

Lam, SHF, Bailitz, J, Blehar, D, Becker, BA, Hoffmann, B, Liteplo, AS, Rajan, K & Lambert, M 2015, 'Multi-institution validation of an emergency ultrasound image rating scale - A pilot study', Journal of Emergency Medicine, vol. 49, no. 1, pp. 32-39. https://doi.org/10.1016/j.jemermed.2015.01.010
Lam, Samuel H.F. ; Bailitz, John ; Blehar, David ; Becker, Brent A. ; Hoffmann, Beatrice ; Liteplo, Andrew S. ; Rajan, Kumar ; Lambert, Michael. / Multi-institution validation of an emergency ultrasound image rating scale - A pilot study. In: Journal of Emergency Medicine. 2015 ; Vol. 49, No. 1. pp. 32-39.
@article{f53cbd11ff8b475a93c81eb7051419ed,
title = "Multi-institution validation of an emergency ultrasound image rating scale - A pilot study",
abstract = "Background As bedside ultrasound (BUS) is being increasingly taught and incorporated into emergency medicine practice, measurement of BUS competency is becoming more important. The commonly adopted experiential approach to BUS competency has never been validated on a large scale, and has some limitations by design. Objective Our aim was to introduce and report preliminary testing of a novel emergency BUS image rating scale (URS). Methods Gallbladder BUS was selected as the test case. Twenty anonymous BUS image sets (still images and clips) were forwarded electronically to 16 reviewers (13 attendings, 3 fellows) at six training sites across the United States. Each reviewer rated the BUS sets using the pilot URS that consisted of three components, with numerical values assigned to each of the following aspects: Landmarks, Image Quality, and Annotations. Reviewers also decided whether or not each BUS set would be {"}Clinically Useful.{"} Kendall τs were calculated as a measure of concordance among the reviewers. Results Among the 13 attendings, image review experience ranged from 2-15 years, 5-300 scans per week, and averaged 7.8 years and 60 images. Kendall τs for each aspect of the URS were: Landmarks: 0.55; Image Quality: 0.57; Annotation: 0.26; Total Score: 0.63, and Clinical Usefulness: 0.45. All URS elements correlated significantly with clinical usefulness (p < 0.001). The correlation coefficient between each attending reviewer and the entire group ranged from 0.48-0.69, and was independent of image review experience beyond fellowship training. Conclusion Our novel URS had moderate-to-good inter-rater agreement in this pilot study. Based on these results, the URS will be modified for use in future investigations.",
keywords = "bedside ultrasound, competency, image rating scale, ultrasound education",
author = "Lam, {Samuel H.F.} and John Bailitz and David Blehar and Becker, {Brent A.} and Beatrice Hoffmann and Liteplo, {Andrew S.} and Kumar Rajan and Michael Lambert",
year = "2015",
month = "1",
day = "1",
doi = "10.1016/j.jemermed.2015.01.010",
language = "English (US)",
volume = "49",
pages = "32--39",
journal = "Journal of Emergency Medicine",
issn = "0736-4679",
publisher = "Elsevier USA",
number = "1",

}

TY - JOUR

T1 - Multi-institution validation of an emergency ultrasound image rating scale - A pilot study

AU - Lam, Samuel H.F.

AU - Bailitz, John

AU - Blehar, David

AU - Becker, Brent A.

AU - Hoffmann, Beatrice

AU - Liteplo, Andrew S.

AU - Rajan, Kumar

AU - Lambert, Michael

PY - 2015/1/1

Y1 - 2015/1/1

N2 - Background As bedside ultrasound (BUS) is being increasingly taught and incorporated into emergency medicine practice, measurement of BUS competency is becoming more important. The commonly adopted experiential approach to BUS competency has never been validated on a large scale, and has some limitations by design. Objective Our aim was to introduce and report preliminary testing of a novel emergency BUS image rating scale (URS). Methods Gallbladder BUS was selected as the test case. Twenty anonymous BUS image sets (still images and clips) were forwarded electronically to 16 reviewers (13 attendings, 3 fellows) at six training sites across the United States. Each reviewer rated the BUS sets using the pilot URS that consisted of three components, with numerical values assigned to each of the following aspects: Landmarks, Image Quality, and Annotations. Reviewers also decided whether or not each BUS set would be "Clinically Useful." Kendall τs were calculated as a measure of concordance among the reviewers. Results Among the 13 attendings, image review experience ranged from 2-15 years, 5-300 scans per week, and averaged 7.8 years and 60 images. Kendall τs for each aspect of the URS were: Landmarks: 0.55; Image Quality: 0.57; Annotation: 0.26; Total Score: 0.63, and Clinical Usefulness: 0.45. All URS elements correlated significantly with clinical usefulness (p < 0.001). The correlation coefficient between each attending reviewer and the entire group ranged from 0.48-0.69, and was independent of image review experience beyond fellowship training. Conclusion Our novel URS had moderate-to-good inter-rater agreement in this pilot study. Based on these results, the URS will be modified for use in future investigations.

AB - Background As bedside ultrasound (BUS) is being increasingly taught and incorporated into emergency medicine practice, measurement of BUS competency is becoming more important. The commonly adopted experiential approach to BUS competency has never been validated on a large scale, and has some limitations by design. Objective Our aim was to introduce and report preliminary testing of a novel emergency BUS image rating scale (URS). Methods Gallbladder BUS was selected as the test case. Twenty anonymous BUS image sets (still images and clips) were forwarded electronically to 16 reviewers (13 attendings, 3 fellows) at six training sites across the United States. Each reviewer rated the BUS sets using the pilot URS that consisted of three components, with numerical values assigned to each of the following aspects: Landmarks, Image Quality, and Annotations. Reviewers also decided whether or not each BUS set would be "Clinically Useful." Kendall τs were calculated as a measure of concordance among the reviewers. Results Among the 13 attendings, image review experience ranged from 2-15 years, 5-300 scans per week, and averaged 7.8 years and 60 images. Kendall τs for each aspect of the URS were: Landmarks: 0.55; Image Quality: 0.57; Annotation: 0.26; Total Score: 0.63, and Clinical Usefulness: 0.45. All URS elements correlated significantly with clinical usefulness (p < 0.001). The correlation coefficient between each attending reviewer and the entire group ranged from 0.48-0.69, and was independent of image review experience beyond fellowship training. Conclusion Our novel URS had moderate-to-good inter-rater agreement in this pilot study. Based on these results, the URS will be modified for use in future investigations.

KW - bedside ultrasound

KW - competency

KW - image rating scale

KW - ultrasound education

UR - http://www.scopus.com/inward/record.url?scp=84933280542&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84933280542&partnerID=8YFLogxK

U2 - 10.1016/j.jemermed.2015.01.010

DO - 10.1016/j.jemermed.2015.01.010

M3 - Article

VL - 49

SP - 32

EP - 39

JO - Journal of Emergency Medicine

JF - Journal of Emergency Medicine

SN - 0736-4679

IS - 1

ER -