The next TCS+ talk, and last of the Fall season, will take place this coming Wednesday, December 4th at 1:00 PM Eastern Time (10:00 AM Pacific Time, 19:00 Central European Time, 18:00 UTC). Nihar Shah from CMU will speak about “Battling Demons in Peer Review” (abstract below).
Please make sure you reserve a spot for your group to join us live by signing up on the online form. As usual, for more information about the TCS+ online seminar series and the upcoming talks, or to suggest a possible topic or speaker, please see the website.
Abstract: Peer review is the backbone of scholarly research. It is however faced with a number of challenges (or “demons”) which cause unfairness to authors, and degrade the overall quality of the process. This talk will present principled and practical approaches to battle these demons in peer review:
- Subjectivity: How to ensure that all papers are judged by the same yardstick?
- Mis-calibration: How to use ratings in presence of arbitrary or adversarial mis-calibration?
- Bias: How to rigorously test for existence of (gender/fame/race/…) biases in peer review?
- Strategic behavior: How to insulate peer review from strategic behavior of author-reviewers?
- Noise: How to assign reviewers to papers to simultaneously ensure fair and accurate evaluations in the presence of review noise?
The work uses tools from social choice theory, statistics and learning theory, information theory, game theory and decision theory. No prior knowledge on these topics will be assumed.
Nihar B. Shah is an Assistant Professor in the Machine Learning and Computer Science departments at Carnegie Mellon University (CMU). His research interests include statistics, machine learning, information theory, and game theory, with a focus on applications to learning from people. He is a recipient of the the 2017 David J. Sakrison memorial prize from EECS Berkeley for a “truly outstanding and innovative PhD thesis”, the Microsoft Research PhD Fellowship 2014-16, the Berkeley Fellowship 2011-13, the IEEE Data Storage Best Paper and Best Student Paper Awards for the years 2011/2012, and the SVC Aiya Medal 2010, and has supervised the Best Student Paper at AAMAS 2019.