Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster E in Poster Session E: Thursday, August 6, 10:30 am – 12:15 pm, Kimmel Center, Shorin & Rosenthal Rooms

Bayesian calibration for conference reviews

Harrison Ritz¹; ¹Queen's University

Presenter: Harrison Ritz

Selecting among high-quality submissions is a challenge for universities, funders, and conferences. A major difficulty in these evaluations is accounting for idiosyncrasies in reviewers' assessments. Classical psychometric methods from Item Response Theory (IRT) can help mitigate these reviewer effects; however, these methods are rarely used in peer review. Here, we leverage Bayesian IRT to reanalyze openly available reviews from CCN 2023. Our results suggest that IRT can support fairer assessment of conference papers and other high-stakes evaluations.

Topic Area: Methods, Tools, Theory & Neural Coding