Editorial peer reviewers' recommendations at a general medical journal: Are they reliable and do editors care?

Richard L Kravitz, Peter Franks, Mitchell D. Feldman, Martha Gerrity, Cindy Byrne, William M. Tierney

Research output: Contribution to journalArticle

76 Citations (Scopus)

Abstract

Background: Editorial peer review is universally used but little studied. We examined the relationship between external reviewers' recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine (JGIM). Methodology/Principal Findings: We examined reviewer recommendations and editors' decisions at JGIM between 2004 and 2008. For manuscripts undergoing peer review, we calculated chance-corrected agreement among reviewers on recommendations to reject versus accept or revise. Using mixed effects logistic regression models, we estimated intra-class correlation coefficients (ICC) at the reviewer and manuscript level. Finally, we examined the probability of rejection in relation to reviewer agreement and disagreement. The 2264 manuscripts sent for external review during the study period received 5881 reviews provided by 2916 reviewers; 28% of reviews recommended rejection. Chance corrected agreement (kappa statistic) on rejection among reviewers was 0.11 (p<.01). In mixed effects models adjusting for study year and manuscript type, the reviewer-level ICC was 0.23 (95% confidence interval [CI], 0.19-0.29) and the manuscript-level ICC was 0.17 (95% CI, 0.12-0.22). The editors' overall rejection rate was 48%: 88% when all reviewers for a manuscript agreed on rejection (7% of manuscripts) and 20% when all reviewers agreed that the manuscript should not be rejected (48% of manuscripts) (p<0.01). Conclusions/Significance: Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations. Efforts are needed to improve the reliability of the peer-review process while helping editors understand the limitations of reviewers' recommendations.

Original languageEnglish (US)
Article numbere10072
JournalPLoS One
Volume5
Issue number4
DOIs
StatePublished - 2010

Fingerprint

Manuscripts
peers
medicine
Peer Review
confidence interval
Internal Medicine
statistics
Logistic Models
Confidence Intervals
Logistics
Statistics
Rejection (Psychology)
Weights and Measures
methodology

ASJC Scopus subject areas

  • Agricultural and Biological Sciences(all)
  • Biochemistry, Genetics and Molecular Biology(all)
  • Medicine(all)

Cite this

Editorial peer reviewers' recommendations at a general medical journal : Are they reliable and do editors care? / Kravitz, Richard L; Franks, Peter; Feldman, Mitchell D.; Gerrity, Martha; Byrne, Cindy; Tierney, William M.

In: PLoS One, Vol. 5, No. 4, e10072, 2010.

Research output: Contribution to journalArticle

Kravitz, Richard L ; Franks, Peter ; Feldman, Mitchell D. ; Gerrity, Martha ; Byrne, Cindy ; Tierney, William M. / Editorial peer reviewers' recommendations at a general medical journal : Are they reliable and do editors care?. In: PLoS One. 2010 ; Vol. 5, No. 4.
@article{b4c3640827dd498fa607ce70fc2bade7,
title = "Editorial peer reviewers' recommendations at a general medical journal: Are they reliable and do editors care?",
abstract = "Background: Editorial peer review is universally used but little studied. We examined the relationship between external reviewers' recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine (JGIM). Methodology/Principal Findings: We examined reviewer recommendations and editors' decisions at JGIM between 2004 and 2008. For manuscripts undergoing peer review, we calculated chance-corrected agreement among reviewers on recommendations to reject versus accept or revise. Using mixed effects logistic regression models, we estimated intra-class correlation coefficients (ICC) at the reviewer and manuscript level. Finally, we examined the probability of rejection in relation to reviewer agreement and disagreement. The 2264 manuscripts sent for external review during the study period received 5881 reviews provided by 2916 reviewers; 28{\%} of reviews recommended rejection. Chance corrected agreement (kappa statistic) on rejection among reviewers was 0.11 (p<.01). In mixed effects models adjusting for study year and manuscript type, the reviewer-level ICC was 0.23 (95{\%} confidence interval [CI], 0.19-0.29) and the manuscript-level ICC was 0.17 (95{\%} CI, 0.12-0.22). The editors' overall rejection rate was 48{\%}: 88{\%} when all reviewers for a manuscript agreed on rejection (7{\%} of manuscripts) and 20{\%} when all reviewers agreed that the manuscript should not be rejected (48{\%} of manuscripts) (p<0.01). Conclusions/Significance: Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations. Efforts are needed to improve the reliability of the peer-review process while helping editors understand the limitations of reviewers' recommendations.",
author = "Kravitz, {Richard L} and Peter Franks and Feldman, {Mitchell D.} and Martha Gerrity and Cindy Byrne and Tierney, {William M.}",
year = "2010",
doi = "10.1371/journal.pone.0010072",
language = "English (US)",
volume = "5",
journal = "PLoS One",
issn = "1932-6203",
publisher = "Public Library of Science",
number = "4",

}

TY - JOUR

T1 - Editorial peer reviewers' recommendations at a general medical journal

T2 - Are they reliable and do editors care?

AU - Kravitz, Richard L

AU - Franks, Peter

AU - Feldman, Mitchell D.

AU - Gerrity, Martha

AU - Byrne, Cindy

AU - Tierney, William M.

PY - 2010

Y1 - 2010

N2 - Background: Editorial peer review is universally used but little studied. We examined the relationship between external reviewers' recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine (JGIM). Methodology/Principal Findings: We examined reviewer recommendations and editors' decisions at JGIM between 2004 and 2008. For manuscripts undergoing peer review, we calculated chance-corrected agreement among reviewers on recommendations to reject versus accept or revise. Using mixed effects logistic regression models, we estimated intra-class correlation coefficients (ICC) at the reviewer and manuscript level. Finally, we examined the probability of rejection in relation to reviewer agreement and disagreement. The 2264 manuscripts sent for external review during the study period received 5881 reviews provided by 2916 reviewers; 28% of reviews recommended rejection. Chance corrected agreement (kappa statistic) on rejection among reviewers was 0.11 (p<.01). In mixed effects models adjusting for study year and manuscript type, the reviewer-level ICC was 0.23 (95% confidence interval [CI], 0.19-0.29) and the manuscript-level ICC was 0.17 (95% CI, 0.12-0.22). The editors' overall rejection rate was 48%: 88% when all reviewers for a manuscript agreed on rejection (7% of manuscripts) and 20% when all reviewers agreed that the manuscript should not be rejected (48% of manuscripts) (p<0.01). Conclusions/Significance: Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations. Efforts are needed to improve the reliability of the peer-review process while helping editors understand the limitations of reviewers' recommendations.

AB - Background: Editorial peer review is universally used but little studied. We examined the relationship between external reviewers' recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine (JGIM). Methodology/Principal Findings: We examined reviewer recommendations and editors' decisions at JGIM between 2004 and 2008. For manuscripts undergoing peer review, we calculated chance-corrected agreement among reviewers on recommendations to reject versus accept or revise. Using mixed effects logistic regression models, we estimated intra-class correlation coefficients (ICC) at the reviewer and manuscript level. Finally, we examined the probability of rejection in relation to reviewer agreement and disagreement. The 2264 manuscripts sent for external review during the study period received 5881 reviews provided by 2916 reviewers; 28% of reviews recommended rejection. Chance corrected agreement (kappa statistic) on rejection among reviewers was 0.11 (p<.01). In mixed effects models adjusting for study year and manuscript type, the reviewer-level ICC was 0.23 (95% confidence interval [CI], 0.19-0.29) and the manuscript-level ICC was 0.17 (95% CI, 0.12-0.22). The editors' overall rejection rate was 48%: 88% when all reviewers for a manuscript agreed on rejection (7% of manuscripts) and 20% when all reviewers agreed that the manuscript should not be rejected (48% of manuscripts) (p<0.01). Conclusions/Significance: Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations. Efforts are needed to improve the reliability of the peer-review process while helping editors understand the limitations of reviewers' recommendations.

UR - http://www.scopus.com/inward/record.url?scp=77956323567&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77956323567&partnerID=8YFLogxK

U2 - 10.1371/journal.pone.0010072

DO - 10.1371/journal.pone.0010072

M3 - Article

C2 - 20386704

AN - SCOPUS:77956323567

VL - 5

JO - PLoS One

JF - PLoS One

SN - 1932-6203

IS - 4

M1 - e10072

ER -