About the OpenRank Experiment

This experiment is based on the isotonic mechanism introduced in two papers [1, 2]. This mechanism takes as input the ranking provided by the authors and review ratings and outputs modified review ratings that are consistent with the author-provided ranking. Under certain assumptions, the authors would be better off truthfully reporting the rankings or partial rankings to the best of their knowledge if the modified review ratings are used to inform decision-making in an appropriate manner, and therefore, the modified review ratings would be more accurate than the raw ratings.

Conference-Specific Details

ICML 2023

To be 100% clear, this year the modified review ratings will not be used in decision-making processes. The purpose of this experiment is to assess the actual effectiveness of this mechanism. Our analysis will be based on the ranking data, as well as the review ratings (numeric only) and final decisions obtained from OpenReview, with all personal identifying information removed. Our goal is to understand how reliable the author-provided rankings or pairwise comparisons are, to investigate if the modified ratings accurately reflect the quality of the submissions, and specifically, to investigate if a significant discrepancy between the modified and original ratings suggests inadequate review quality.

The ultimate goal of this experiment is to assess the possibility of combining authors’ own opinions and reviewers’ ratings and comments for peer review in future conferences. As the number of submissions explodes while the number of experienced reviewers is limited, relying on reviewers alone for peer review becomes increasingly challenging in large machine learning conferences. On the other hand, the authors often have their own information about their submission quality that can be complementary to that of the reviewers, and the question is, of course, how to truthfully elicit information from the authors. The isotonic mechanism is an initiative to incorporate author-assisted information into peer review. Potential improvements and alternatives are certainly possible.

Privacy and confidentiality are at the heart of the design of this experiment. We have taken the following strict steps to preserve them:

  1. The rankings will not be shared with co-authors, reviewers, ACs, SACs, or PCs. Your responses will not affect the review process in any sense.
  2. Only the SHA-256 hashed values, but not the original values, of both the submission IDs and author IDs will be preserved for statistical analysis. The experiment team will not analyze the data until the review process of ICML 2023 is done.
  3. Only aggregated statistics will be released for academic purposes only, with explicit approval by the ICML 2023 PCs.
  4. All data collected from this experiment except for the aggregated statistics published in the paper(s) will be completely deleted by December 31, 2024.

This experiment was designed by Jiayao Zhang, Natalie Collina, Aaron Roth, Xiao-Li Meng, and Weijie Su. Please do not hesitate to reach out to us if you have any questions or concerns.