Detecting the effects of physician training in self-care interviewing skills: Coding of standardized patient (SP) visit recordings versus SP post-visit ratings

Objective: To compare how coder ratings of standardized patient (SP) visit recordings and SP ratings of the visits detect primary care physician (PCP) training in self-efficacy enhancing interviewing techniques (SEE IT). Methods: Analyses of data from 50 PCPs who participated in a randomized controlled trial of SEE IT training, which led to increased SEE IT use during three SP visits 1-3 months post-intervention. Untrained SPs rated SEE IT use post-visit. Subsequently, three trained coders generated a consensus SEE IT rating from visit audio recordings. SPs and coders were blinded to provider study arm, and coders to SP ratings. Results: SP and coder ratings were correlated (r = 0.62). In detecting the intervention effect, the areas under the receiver operating characteristic curve were 0.80 (95% CI 0.74-0.87) and 0.76 (95% CI 0.69-0.84) for consensus coder and SP ratings, respectively (difference 0.04, 95% CI -0.04-0.11; z = 1.04, p = 0.30). Conclusion: SP ratings were not significantly different from coder ratings of SP visit recordings in detecting PCP SEE IT training. Practice implications: If similar findings are observed in larger studies, it would suggest a greater role for SP ratings in detecting provider interviewing skills training, given the relative simplicity, low cost, and non-intrusiveness of the approach.