Comparing diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought

Jonathan S. Ilgen, Judith L. Bowen, Lucas A. McIntyre, Kenny V. Banh, David Barnes, Wendy C. Coates, Jeffrey Druck, Megan L. Fix, Diane Rimple, Lalena M. Yarris, Kevin W. Eva

Research output: Contribution to journalArticle

26 Citations (Scopus)

Abstract

PURPOSE: Although decades of research have yielded considerable insight into physicians' clinical reasoning processes, assessing these processes remains challenging; thus, the authors sought to compare diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought. METHOD: This 2011-2012 multicenter randomized study of 393 clinicians (medical students, postgraduate trainees, and faculty) measured diagnostic accuracy on clinical vignettes under two conditions: one encouraged participants to give their first impression (FI), and the other led participants through a directed search (DS) for the correct diagnosis. The authors compared accuracy, feasibility, reliability, and relation to United States Medical Licensing Exam (USMLE) scores under each condition. RESULTS: A 2 (instructional condition) × 2 (vignette complexity) × 3 (experience level) analysis of variance revealed no difference in accuracy as a function of instructional condition (F[1,379] = 2.44, P = .12), but demonstrated the expected main effects of vignette complexity (F[1,379] = 965.2, P < .001) and experience (F[2,379] = 39.6, P < .001). Pearson correlations revealed greater associations between assessment scores and USMLE performance in the FI condition than in the DS condition (P < .001). Spearman-Brown calculations consistently indicated that alpha ≥ 0.75 could be achieved more efficiently under the FI condition relative to the DS condition. CONCLUSIONS: Instructions to trust one's first impres-sions result in similar performance when compared with instructions to consider clinical information in a systematic fashion, but have greater utility when used for the purposes of assessment.

Original languageEnglish (US)
Pages (from-to)1545-1551
Number of pages7
JournalAcademic Medicine
Volume88
Issue number10
DOIs
StatePublished - Jan 1 2013

Fingerprint

diagnostic
instruction
performance
analysis of variance
trainee
medical student
experience
physician

ASJC Scopus subject areas

  • Education

Cite this

Comparing diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought. / Ilgen, Jonathan S.; Bowen, Judith L.; McIntyre, Lucas A.; Banh, Kenny V.; Barnes, David; Coates, Wendy C.; Druck, Jeffrey; Fix, Megan L.; Rimple, Diane; Yarris, Lalena M.; Eva, Kevin W.

In: Academic Medicine, Vol. 88, No. 10, 01.01.2013, p. 1545-1551.

Research output: Contribution to journalArticle

Ilgen, Jonathan S. ; Bowen, Judith L. ; McIntyre, Lucas A. ; Banh, Kenny V. ; Barnes, David ; Coates, Wendy C. ; Druck, Jeffrey ; Fix, Megan L. ; Rimple, Diane ; Yarris, Lalena M. ; Eva, Kevin W. / Comparing diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought. In: Academic Medicine. 2013 ; Vol. 88, No. 10. pp. 1545-1551.
@article{3b5f7ad16aa844a0adea97a43826c6de,
title = "Comparing diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought",
abstract = "PURPOSE: Although decades of research have yielded considerable insight into physicians' clinical reasoning processes, assessing these processes remains challenging; thus, the authors sought to compare diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought. METHOD: This 2011-2012 multicenter randomized study of 393 clinicians (medical students, postgraduate trainees, and faculty) measured diagnostic accuracy on clinical vignettes under two conditions: one encouraged participants to give their first impression (FI), and the other led participants through a directed search (DS) for the correct diagnosis. The authors compared accuracy, feasibility, reliability, and relation to United States Medical Licensing Exam (USMLE) scores under each condition. RESULTS: A 2 (instructional condition) × 2 (vignette complexity) × 3 (experience level) analysis of variance revealed no difference in accuracy as a function of instructional condition (F[1,379] = 2.44, P = .12), but demonstrated the expected main effects of vignette complexity (F[1,379] = 965.2, P < .001) and experience (F[2,379] = 39.6, P < .001). Pearson correlations revealed greater associations between assessment scores and USMLE performance in the FI condition than in the DS condition (P < .001). Spearman-Brown calculations consistently indicated that alpha ≥ 0.75 could be achieved more efficiently under the FI condition relative to the DS condition. CONCLUSIONS: Instructions to trust one's first impres-sions result in similar performance when compared with instructions to consider clinical information in a systematic fashion, but have greater utility when used for the purposes of assessment.",
author = "Ilgen, {Jonathan S.} and Bowen, {Judith L.} and McIntyre, {Lucas A.} and Banh, {Kenny V.} and David Barnes and Coates, {Wendy C.} and Jeffrey Druck and Fix, {Megan L.} and Diane Rimple and Yarris, {Lalena M.} and Eva, {Kevin W.}",
year = "2013",
month = "1",
day = "1",
doi = "10.1097/ACM.0b013e3182a31c1e",
language = "English (US)",
volume = "88",
pages = "1545--1551",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",
number = "10",

}

TY - JOUR

T1 - Comparing diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought

AU - Ilgen, Jonathan S.

AU - Bowen, Judith L.

AU - McIntyre, Lucas A.

AU - Banh, Kenny V.

AU - Barnes, David

AU - Coates, Wendy C.

AU - Druck, Jeffrey

AU - Fix, Megan L.

AU - Rimple, Diane

AU - Yarris, Lalena M.

AU - Eva, Kevin W.

PY - 2013/1/1

Y1 - 2013/1/1

N2 - PURPOSE: Although decades of research have yielded considerable insight into physicians' clinical reasoning processes, assessing these processes remains challenging; thus, the authors sought to compare diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought. METHOD: This 2011-2012 multicenter randomized study of 393 clinicians (medical students, postgraduate trainees, and faculty) measured diagnostic accuracy on clinical vignettes under two conditions: one encouraged participants to give their first impression (FI), and the other led participants through a directed search (DS) for the correct diagnosis. The authors compared accuracy, feasibility, reliability, and relation to United States Medical Licensing Exam (USMLE) scores under each condition. RESULTS: A 2 (instructional condition) × 2 (vignette complexity) × 3 (experience level) analysis of variance revealed no difference in accuracy as a function of instructional condition (F[1,379] = 2.44, P = .12), but demonstrated the expected main effects of vignette complexity (F[1,379] = 965.2, P < .001) and experience (F[2,379] = 39.6, P < .001). Pearson correlations revealed greater associations between assessment scores and USMLE performance in the FI condition than in the DS condition (P < .001). Spearman-Brown calculations consistently indicated that alpha ≥ 0.75 could be achieved more efficiently under the FI condition relative to the DS condition. CONCLUSIONS: Instructions to trust one's first impres-sions result in similar performance when compared with instructions to consider clinical information in a systematic fashion, but have greater utility when used for the purposes of assessment.

AB - PURPOSE: Although decades of research have yielded considerable insight into physicians' clinical reasoning processes, assessing these processes remains challenging; thus, the authors sought to compare diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought. METHOD: This 2011-2012 multicenter randomized study of 393 clinicians (medical students, postgraduate trainees, and faculty) measured diagnostic accuracy on clinical vignettes under two conditions: one encouraged participants to give their first impression (FI), and the other led participants through a directed search (DS) for the correct diagnosis. The authors compared accuracy, feasibility, reliability, and relation to United States Medical Licensing Exam (USMLE) scores under each condition. RESULTS: A 2 (instructional condition) × 2 (vignette complexity) × 3 (experience level) analysis of variance revealed no difference in accuracy as a function of instructional condition (F[1,379] = 2.44, P = .12), but demonstrated the expected main effects of vignette complexity (F[1,379] = 965.2, P < .001) and experience (F[2,379] = 39.6, P < .001). Pearson correlations revealed greater associations between assessment scores and USMLE performance in the FI condition than in the DS condition (P < .001). Spearman-Brown calculations consistently indicated that alpha ≥ 0.75 could be achieved more efficiently under the FI condition relative to the DS condition. CONCLUSIONS: Instructions to trust one's first impres-sions result in similar performance when compared with instructions to consider clinical information in a systematic fashion, but have greater utility when used for the purposes of assessment.

UR - http://www.scopus.com/inward/record.url?scp=84885184993&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84885184993&partnerID=8YFLogxK

U2 - 10.1097/ACM.0b013e3182a31c1e

DO - 10.1097/ACM.0b013e3182a31c1e

M3 - Article

C2 - 23969355

AN - SCOPUS:84885184993

VL - 88

SP - 1545

EP - 1551

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

IS - 10

ER -