Reliability of Peer Review of Abstracts Submitted to Academic Family Medicine Meetings

Joshua J. Fenton, Hazel Tapp, Netra M. Thakur, Andrea Pfeifle

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

Purpose: To assess the reliability of peer review of abstracts submitted to academic family medicine meetings in North America. Methods: We analyzed reviewer ratings of abstracts submitted: 1) as oral presentations to the North American Primary Care Research Group (NAPCRG) meeting from 2016 to 2019, as well as 2019 poster session or workshop submissions; and 2) in 12 categories to the Society of Teachers of Family Medicine (STFM) Spring 2018 meeting. In each category and year, we used a multi-level mixed model to estimate the abstract-level intraclass correlation coefficient (ICC) and the reliability of initial review (using the abstract-level ICC and the number of reviewers per abstract). Results: We analyzed review data for 1554 NAPCRG oral presentation abstracts, 418 NAPCRG poster or workshop abstracts, and 1145 STFM abstracts. Across all years, abstract-level ICCs for NAPCRG oral presentations were below 0.20 (range, 0.10 in 2019 to 0.18 in 2016) and were even lower for posters and workshops (range, 0.00-0.10). After accounting for the number of reviewers per abstract, reliabilities of initial review for NAPCRG oral presentations ranged from 0.24 in 2019 to 0.30 in 2016 and 0.00 to 0.18 for posters and workshops in 2019. Across 12 STFM submission categories, the median abstract-level ICC was 0.21 (range, 0.12-0.50) and the median reliability was 0.42 (range, 0.25-0.78). Conclusions: For abstracts submitted to North American academic family medicine meetings, inter-reviewer agreement is often low, compromising initial review reliability. For many submission categories, program committees should supplement initial review with independent postreview assessments.

Original languageEnglish (US)
Pages (from-to)986-991
Number of pages6
JournalJournal of the American Board of Family Medicine
Volume33
Issue number6
DOIs
StatePublished - Nov 2020
Externally publishedYes

Keywords

  • Abstracting and Indexing
  • Biostatistics
  • Faculty
  • Observer Variation
  • Peer Review
  • Primary Health Care

ASJC Scopus subject areas

  • Public Health, Environmental and Occupational Health
  • Family Practice

Fingerprint Dive into the research topics of 'Reliability of Peer Review of Abstracts Submitted to Academic Family Medicine Meetings'. Together they form a unique fingerprint.

Cite this