Designing an abstraction instrument: Lessons from efforts to validate the AHRQ Patient Safety indicators

Background: The U.S. Agency for Healthcare Research and Quality (AHRQ) and other organizations have developed quality indicators based on hospital administrative data. Characteristics of effective abstraction instruments were identified for determining both the positive predictive value (PPV) of Patient Safety Indicators (PSIs) and the extent to which hospitals and clinicians could have prevented adverse events. Methods: Through an iterative process involving nurse abstractors, physicians, and nurses with quality improvement experience, and health services researchers, 25 abstraction instruments were designed for 12 AHRQ provider-level morbidity PSIs. Data were analyzed from 13 of these instruments, and data are being collected using several more. Findings: Common problems in designing the instruments included avoiding uninformative questions and premature termination of the abstraction process, anticipating misinterpretation of questions, allowing an appropriate range of response options; using clear terminology, optimizing the flow of the abstraction process, balancing the utility of data against abstractor burden, and recognizing the needs of end users, such as hospitals and quality improvement professionals and researchers, for the abstracted information. Conclusions: Designing medical record abstraction instruments for quality improvement research involves several potential pitfalls. Understanding how we addressed these challenges might help both investigators and users of outcome indicators to appreciate the strengths and limitations of outcome-based quality indicators and tools designed to validate or investigate such indicators within provider organizations.