A new instrument to measure appropriateness of services in primary care

David H. Thom, Richard L Kravitz, Steven Kelly-Reif, Ronnie V. Sprinkle, Joseph R. Hopkins, Lisa V. Rubenstein

Research output: Contribution to journalArticle

9 Citations (Scopus)

Abstract

Objective. To develop a new instrument for judging the appropriateness of three key services (new prescription, diagnostic test, and referral) as delivered in primary care outpatient visits. Design. Candidate items were generated by a seven-member expert panel, using a five-step nominal technique, for each of three service categories in primary care: new prescriptions, diagnostic tests, and referrals. Expert panelists and a convenience sample of 95 community-based primary care physicians ranked items for (i) importance and (ii) feasibility of ascertaining from a typical office chart record. Resulting items were used to construct a measure of appropriateness using principals of structured implicit review. Two physician reviewers used this measure to judge the appropriateness of 421 services from 160 outpatient visits. Setting. Primary care practices in a staff model health maintenance organization and a large preferred provider network. Measures. Inter-rater agreement was measured using intraclass correlation coefficient (ICC) and kappa statistic. Results. For overall appropriateness, the ICC and kappa were 0.52 and 0.44 for new medication, 0.35 and 0.32 for diagnostic test, and 0.40 and 0.41 for referral, respectively. Only 3% of services were judged to be inappropriate by either reviewer. The proportion of services judged to be less than definitely appropriate by one or both reviewers was 56% for new medication, 31% for diagnostic test, and 22% for referral. Conclusions. This new measure of appropriateness of primary care services has fair inter-rater agreement for new medications and referrals, similar to appropriateness measures of other general services, but poor agreement for diagnostic tests. It may be useful as a tool to assess the appropriateness of common primary care services in studies of health care quality, but is not suitable for evaluating performance of individual physicians.

Original languageEnglish (US)
Pages (from-to)133-140
Number of pages8
JournalInternational Journal for Quality in Health Care
Volume16
Issue number2
DOIs
StatePublished - Apr 2004

Fingerprint

Routine Diagnostic Tests
Primary Health Care
Referral and Consultation
diagnostic
medication
Prescriptions
physician
Physicians
Health Maintenance Organizations
Quality of Health Care
Primary Care Physicians
Ambulatory Care
expert
outpatient care
Outpatients
candidacy
statistics
health care
staff
organization

Keywords

  • Diagnostic tests
  • Prescription
  • Process assessment
  • Referral
  • Utilization review

ASJC Scopus subject areas

  • Nursing(all)
  • Health(social science)
  • Health Professions(all)
  • Public Health, Environmental and Occupational Health

Cite this

A new instrument to measure appropriateness of services in primary care. / Thom, David H.; Kravitz, Richard L; Kelly-Reif, Steven; Sprinkle, Ronnie V.; Hopkins, Joseph R.; Rubenstein, Lisa V.

In: International Journal for Quality in Health Care, Vol. 16, No. 2, 04.2004, p. 133-140.

Research output: Contribution to journalArticle

Thom, David H. ; Kravitz, Richard L ; Kelly-Reif, Steven ; Sprinkle, Ronnie V. ; Hopkins, Joseph R. ; Rubenstein, Lisa V. / A new instrument to measure appropriateness of services in primary care. In: International Journal for Quality in Health Care. 2004 ; Vol. 16, No. 2. pp. 133-140.
@article{bcb3f173a60f4cb68addf41f7011d4f2,
title = "A new instrument to measure appropriateness of services in primary care",
abstract = "Objective. To develop a new instrument for judging the appropriateness of three key services (new prescription, diagnostic test, and referral) as delivered in primary care outpatient visits. Design. Candidate items were generated by a seven-member expert panel, using a five-step nominal technique, for each of three service categories in primary care: new prescriptions, diagnostic tests, and referrals. Expert panelists and a convenience sample of 95 community-based primary care physicians ranked items for (i) importance and (ii) feasibility of ascertaining from a typical office chart record. Resulting items were used to construct a measure of appropriateness using principals of structured implicit review. Two physician reviewers used this measure to judge the appropriateness of 421 services from 160 outpatient visits. Setting. Primary care practices in a staff model health maintenance organization and a large preferred provider network. Measures. Inter-rater agreement was measured using intraclass correlation coefficient (ICC) and kappa statistic. Results. For overall appropriateness, the ICC and kappa were 0.52 and 0.44 for new medication, 0.35 and 0.32 for diagnostic test, and 0.40 and 0.41 for referral, respectively. Only 3{\%} of services were judged to be inappropriate by either reviewer. The proportion of services judged to be less than definitely appropriate by one or both reviewers was 56{\%} for new medication, 31{\%} for diagnostic test, and 22{\%} for referral. Conclusions. This new measure of appropriateness of primary care services has fair inter-rater agreement for new medications and referrals, similar to appropriateness measures of other general services, but poor agreement for diagnostic tests. It may be useful as a tool to assess the appropriateness of common primary care services in studies of health care quality, but is not suitable for evaluating performance of individual physicians.",
keywords = "Diagnostic tests, Prescription, Process assessment, Referral, Utilization review",
author = "Thom, {David H.} and Kravitz, {Richard L} and Steven Kelly-Reif and Sprinkle, {Ronnie V.} and Hopkins, {Joseph R.} and Rubenstein, {Lisa V.}",
year = "2004",
month = "4",
doi = "10.1093/intqhc/mzh029",
language = "English (US)",
volume = "16",
pages = "133--140",
journal = "International Journal for Quality in Health Care",
issn = "1353-4505",
publisher = "Oxford University Press",
number = "2",

}

TY - JOUR

T1 - A new instrument to measure appropriateness of services in primary care

AU - Thom, David H.

AU - Kravitz, Richard L

AU - Kelly-Reif, Steven

AU - Sprinkle, Ronnie V.

AU - Hopkins, Joseph R.

AU - Rubenstein, Lisa V.

PY - 2004/4

Y1 - 2004/4

N2 - Objective. To develop a new instrument for judging the appropriateness of three key services (new prescription, diagnostic test, and referral) as delivered in primary care outpatient visits. Design. Candidate items were generated by a seven-member expert panel, using a five-step nominal technique, for each of three service categories in primary care: new prescriptions, diagnostic tests, and referrals. Expert panelists and a convenience sample of 95 community-based primary care physicians ranked items for (i) importance and (ii) feasibility of ascertaining from a typical office chart record. Resulting items were used to construct a measure of appropriateness using principals of structured implicit review. Two physician reviewers used this measure to judge the appropriateness of 421 services from 160 outpatient visits. Setting. Primary care practices in a staff model health maintenance organization and a large preferred provider network. Measures. Inter-rater agreement was measured using intraclass correlation coefficient (ICC) and kappa statistic. Results. For overall appropriateness, the ICC and kappa were 0.52 and 0.44 for new medication, 0.35 and 0.32 for diagnostic test, and 0.40 and 0.41 for referral, respectively. Only 3% of services were judged to be inappropriate by either reviewer. The proportion of services judged to be less than definitely appropriate by one or both reviewers was 56% for new medication, 31% for diagnostic test, and 22% for referral. Conclusions. This new measure of appropriateness of primary care services has fair inter-rater agreement for new medications and referrals, similar to appropriateness measures of other general services, but poor agreement for diagnostic tests. It may be useful as a tool to assess the appropriateness of common primary care services in studies of health care quality, but is not suitable for evaluating performance of individual physicians.

AB - Objective. To develop a new instrument for judging the appropriateness of three key services (new prescription, diagnostic test, and referral) as delivered in primary care outpatient visits. Design. Candidate items were generated by a seven-member expert panel, using a five-step nominal technique, for each of three service categories in primary care: new prescriptions, diagnostic tests, and referrals. Expert panelists and a convenience sample of 95 community-based primary care physicians ranked items for (i) importance and (ii) feasibility of ascertaining from a typical office chart record. Resulting items were used to construct a measure of appropriateness using principals of structured implicit review. Two physician reviewers used this measure to judge the appropriateness of 421 services from 160 outpatient visits. Setting. Primary care practices in a staff model health maintenance organization and a large preferred provider network. Measures. Inter-rater agreement was measured using intraclass correlation coefficient (ICC) and kappa statistic. Results. For overall appropriateness, the ICC and kappa were 0.52 and 0.44 for new medication, 0.35 and 0.32 for diagnostic test, and 0.40 and 0.41 for referral, respectively. Only 3% of services were judged to be inappropriate by either reviewer. The proportion of services judged to be less than definitely appropriate by one or both reviewers was 56% for new medication, 31% for diagnostic test, and 22% for referral. Conclusions. This new measure of appropriateness of primary care services has fair inter-rater agreement for new medications and referrals, similar to appropriateness measures of other general services, but poor agreement for diagnostic tests. It may be useful as a tool to assess the appropriateness of common primary care services in studies of health care quality, but is not suitable for evaluating performance of individual physicians.

KW - Diagnostic tests

KW - Prescription

KW - Process assessment

KW - Referral

KW - Utilization review

UR - http://www.scopus.com/inward/record.url?scp=1842865667&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=1842865667&partnerID=8YFLogxK

U2 - 10.1093/intqhc/mzh029

DO - 10.1093/intqhc/mzh029

M3 - Article

VL - 16

SP - 133

EP - 140

JO - International Journal for Quality in Health Care

JF - International Journal for Quality in Health Care

SN - 1353-4505

IS - 2

ER -