Measuring the quality of physician practice by using clinical vignettes: A prospective validation study

John W. Peabody, Jeff Luck, Peter Glassman, Sharad Jain, Joyce Hansen, Maureen Spell, Martin Lee

Research output: Contribution to journalArticle

434 Citations (Scopus)

Abstract

Background: Worldwide efforts are under way to improve the quality of clinical practice. Most quality measurements, however, are poorly validated, expensive, and difficult to compare among sites. Objective: To validate whether vignettes accurately measure the quality of clinical practice by using a comparison with standardized patients (the gold standard method), and to determine whether vignettes are a more or less accurate method than medical record abstraction. Design: Prospective, multisite study. Setting: Outpatient primary care clinics in 2 Veterans Affairs medical centers and 2 large, private medical centers. Participants: 144 of 163 eligible physicians agreed to participate, and, of these, 116 were randomly selected to see standardized patients, to complete vignettes, or both. Measurements: Scores, expressed as the percentage of explicit quality criteria correctly completed, were obtained by using 3 methods. Results: Among all physicians, the quality of clinical practice as measured by the standardized patients was 73% correct (95% CI, 72.1% to 73.4%). By using exactly the same criteria, physicians scored 68% (CI, 67.9% to 68.9%) when measured by the vignettes but only 63% (CI, 62.7% to 64.0%) when assessed by medical record abstraction. These findings were consistent across all diseases and were independent of case complexity or physician training level. Vignettes also accurately measured unnecessary care. Finally, vignettes seem to capture the range in the quality of clinical practice among physicians within a site. Limitations: Despite finding variation in the quality of clinical practice, we did not determine whether poorer quality translated into worse health status for patients. In addition, the quality scores are based on measurements from 1 patient-provider interaction. As with all other scoring criteria, vignette criteria must be regularly updated. Conclusions: Vignettes are a valid tool for measuring the quality of clinical practice. They can be used for diverse clinical settings, diseases, physician types, and situations in which case-mix variation is a concern. They are inexpensive and easy to use. Vignettes are particularly useful for comparing quality among and within sites and may be useful for longitudinal evaluations of interventions intended to change clinical practice.

Original languageEnglish (US)
JournalAnnals of Internal Medicine
Volume141
Issue number10
DOIs
StatePublished - Nov 16 2004
Externally publishedYes

Fingerprint

Validation Studies
Prospective Studies
Physicians
Medical Records
Diagnosis-Related Groups
Veterans
Ambulatory Care
Health Status
Primary Health Care

ASJC Scopus subject areas

  • Internal Medicine

Cite this

Measuring the quality of physician practice by using clinical vignettes : A prospective validation study. / Peabody, John W.; Luck, Jeff; Glassman, Peter; Jain, Sharad; Hansen, Joyce; Spell, Maureen; Lee, Martin.

In: Annals of Internal Medicine, Vol. 141, No. 10, 16.11.2004.

Research output: Contribution to journalArticle

Peabody, John W. ; Luck, Jeff ; Glassman, Peter ; Jain, Sharad ; Hansen, Joyce ; Spell, Maureen ; Lee, Martin. / Measuring the quality of physician practice by using clinical vignettes : A prospective validation study. In: Annals of Internal Medicine. 2004 ; Vol. 141, No. 10.
@article{d747ecb6a6c44fdfa3f88b843b233b11,
title = "Measuring the quality of physician practice by using clinical vignettes: A prospective validation study",
abstract = "Background: Worldwide efforts are under way to improve the quality of clinical practice. Most quality measurements, however, are poorly validated, expensive, and difficult to compare among sites. Objective: To validate whether vignettes accurately measure the quality of clinical practice by using a comparison with standardized patients (the gold standard method), and to determine whether vignettes are a more or less accurate method than medical record abstraction. Design: Prospective, multisite study. Setting: Outpatient primary care clinics in 2 Veterans Affairs medical centers and 2 large, private medical centers. Participants: 144 of 163 eligible physicians agreed to participate, and, of these, 116 were randomly selected to see standardized patients, to complete vignettes, or both. Measurements: Scores, expressed as the percentage of explicit quality criteria correctly completed, were obtained by using 3 methods. Results: Among all physicians, the quality of clinical practice as measured by the standardized patients was 73{\%} correct (95{\%} CI, 72.1{\%} to 73.4{\%}). By using exactly the same criteria, physicians scored 68{\%} (CI, 67.9{\%} to 68.9{\%}) when measured by the vignettes but only 63{\%} (CI, 62.7{\%} to 64.0{\%}) when assessed by medical record abstraction. These findings were consistent across all diseases and were independent of case complexity or physician training level. Vignettes also accurately measured unnecessary care. Finally, vignettes seem to capture the range in the quality of clinical practice among physicians within a site. Limitations: Despite finding variation in the quality of clinical practice, we did not determine whether poorer quality translated into worse health status for patients. In addition, the quality scores are based on measurements from 1 patient-provider interaction. As with all other scoring criteria, vignette criteria must be regularly updated. Conclusions: Vignettes are a valid tool for measuring the quality of clinical practice. They can be used for diverse clinical settings, diseases, physician types, and situations in which case-mix variation is a concern. They are inexpensive and easy to use. Vignettes are particularly useful for comparing quality among and within sites and may be useful for longitudinal evaluations of interventions intended to change clinical practice.",
author = "Peabody, {John W.} and Jeff Luck and Peter Glassman and Sharad Jain and Joyce Hansen and Maureen Spell and Martin Lee",
year = "2004",
month = "11",
day = "16",
doi = "10.7326/0003-4819-141-10-200411160-00008",
language = "English (US)",
volume = "141",
journal = "Annals of Internal Medicine",
issn = "0003-4819",
publisher = "American College of Physicians",
number = "10",

}

TY - JOUR

T1 - Measuring the quality of physician practice by using clinical vignettes

T2 - A prospective validation study

AU - Peabody, John W.

AU - Luck, Jeff

AU - Glassman, Peter

AU - Jain, Sharad

AU - Hansen, Joyce

AU - Spell, Maureen

AU - Lee, Martin

PY - 2004/11/16

Y1 - 2004/11/16

N2 - Background: Worldwide efforts are under way to improve the quality of clinical practice. Most quality measurements, however, are poorly validated, expensive, and difficult to compare among sites. Objective: To validate whether vignettes accurately measure the quality of clinical practice by using a comparison with standardized patients (the gold standard method), and to determine whether vignettes are a more or less accurate method than medical record abstraction. Design: Prospective, multisite study. Setting: Outpatient primary care clinics in 2 Veterans Affairs medical centers and 2 large, private medical centers. Participants: 144 of 163 eligible physicians agreed to participate, and, of these, 116 were randomly selected to see standardized patients, to complete vignettes, or both. Measurements: Scores, expressed as the percentage of explicit quality criteria correctly completed, were obtained by using 3 methods. Results: Among all physicians, the quality of clinical practice as measured by the standardized patients was 73% correct (95% CI, 72.1% to 73.4%). By using exactly the same criteria, physicians scored 68% (CI, 67.9% to 68.9%) when measured by the vignettes but only 63% (CI, 62.7% to 64.0%) when assessed by medical record abstraction. These findings were consistent across all diseases and were independent of case complexity or physician training level. Vignettes also accurately measured unnecessary care. Finally, vignettes seem to capture the range in the quality of clinical practice among physicians within a site. Limitations: Despite finding variation in the quality of clinical practice, we did not determine whether poorer quality translated into worse health status for patients. In addition, the quality scores are based on measurements from 1 patient-provider interaction. As with all other scoring criteria, vignette criteria must be regularly updated. Conclusions: Vignettes are a valid tool for measuring the quality of clinical practice. They can be used for diverse clinical settings, diseases, physician types, and situations in which case-mix variation is a concern. They are inexpensive and easy to use. Vignettes are particularly useful for comparing quality among and within sites and may be useful for longitudinal evaluations of interventions intended to change clinical practice.

AB - Background: Worldwide efforts are under way to improve the quality of clinical practice. Most quality measurements, however, are poorly validated, expensive, and difficult to compare among sites. Objective: To validate whether vignettes accurately measure the quality of clinical practice by using a comparison with standardized patients (the gold standard method), and to determine whether vignettes are a more or less accurate method than medical record abstraction. Design: Prospective, multisite study. Setting: Outpatient primary care clinics in 2 Veterans Affairs medical centers and 2 large, private medical centers. Participants: 144 of 163 eligible physicians agreed to participate, and, of these, 116 were randomly selected to see standardized patients, to complete vignettes, or both. Measurements: Scores, expressed as the percentage of explicit quality criteria correctly completed, were obtained by using 3 methods. Results: Among all physicians, the quality of clinical practice as measured by the standardized patients was 73% correct (95% CI, 72.1% to 73.4%). By using exactly the same criteria, physicians scored 68% (CI, 67.9% to 68.9%) when measured by the vignettes but only 63% (CI, 62.7% to 64.0%) when assessed by medical record abstraction. These findings were consistent across all diseases and were independent of case complexity or physician training level. Vignettes also accurately measured unnecessary care. Finally, vignettes seem to capture the range in the quality of clinical practice among physicians within a site. Limitations: Despite finding variation in the quality of clinical practice, we did not determine whether poorer quality translated into worse health status for patients. In addition, the quality scores are based on measurements from 1 patient-provider interaction. As with all other scoring criteria, vignette criteria must be regularly updated. Conclusions: Vignettes are a valid tool for measuring the quality of clinical practice. They can be used for diverse clinical settings, diseases, physician types, and situations in which case-mix variation is a concern. They are inexpensive and easy to use. Vignettes are particularly useful for comparing quality among and within sites and may be useful for longitudinal evaluations of interventions intended to change clinical practice.

UR - http://www.scopus.com/inward/record.url?scp=19644400934&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=19644400934&partnerID=8YFLogxK

U2 - 10.7326/0003-4819-141-10-200411160-00008

DO - 10.7326/0003-4819-141-10-200411160-00008

M3 - Article

C2 - 15545677

AN - SCOPUS:19644400934

VL - 141

JO - Annals of Internal Medicine

JF - Annals of Internal Medicine

SN - 0003-4819

IS - 10

ER -