The Library
PeerNomination : a novel peer selection algorithm to handle strategic and noisy assessments
Tools
Lev, Omer, Mattei, Nicholas, Turrini, Paolo and Zhydkov, Stanislav (2023) PeerNomination : a novel peer selection algorithm to handle strategic and noisy assessments. Artificial Intelligence, 316 . 103843. doi:10.1016/j.artint.2022.103843 ISSN 0004-3702.
|
PDF
WRAP-PeerNomination-a-novel-peer-selection-algorithm-to-handle-strategic-and-noisy-assessments-Zhydkov-2023.pdf - Published Version - Requires a PDF viewer. Available under License Creative Commons Attribution 4.0. Download (2417Kb) | Preview |
Official URL: https://doi.org/10.1016/j.artint.2022.103843
Abstract
In peer selection a group of agents must choose a subset of themselves, as winners for, e.g., peer-reviewed grants or prizes. We take a Condorcet view of this aggregation problem, assuming that there is an objective ground-truth ordering over the agents. We study agents that have a noisy perception of this ground truth and give assessments that, even when truthful, can be inaccurate. Our goal is to select the best set of agents according to the underlying ground truth by looking at the potentially unreliable assessments of the peers. Besides being potentially unreliable, we also allow agents to be self-interested, attempting to influence the outcome of the decision in their favour. Hence, we are focused on tackling the problem of impartial (or strategyproof) peer selection – how do we prevent agents from manipulating their reviews while still selecting the most deserving individuals, all in the presence of noisy evaluations? We propose a novel impartial peer selection algorithm, PeerNomination, that aims to fulfil the above desiderata. We provide a comprehensive theoretical analysis of the recall of PeerNomination and prove various properties, including impartiality and monotonicity. We also provide empirical results based on computer simulations to show its effectiveness compared to the state-of-the-art impartial peer selection algorithms. We then investigate the robustness of PeerNomination to various levels of noise in the reviews. In order to maintain good performance under such conditions, we extend PeerNomination by using weights for reviewers which, informally, capture some notion of reliability of the reviewer. We show, theoretically, that the new algorithm preserves strategyproofness and, empirically, that the weights help identify the noisy reviewers and hence to increase selection performance.
Item Type: | Journal Article | ||||||||
---|---|---|---|---|---|---|---|---|---|
Subjects: | Q Science > Q Science (General) Q Science > QA Mathematics Q Science > QA Mathematics > QA76 Electronic computers. Computer science. Computer software T Technology > TK Electrical engineering. Electronics Nuclear engineering |
||||||||
Divisions: | Faculty of Science, Engineering and Medicine > Science > Computer Science Faculty of Science, Engineering and Medicine > Science > Mathematics |
||||||||
SWORD Depositor: | Library Publications Router | ||||||||
Library of Congress Subject Headings (LCSH): | Peer review -- Simulation methods, Artificial intelligence , Pattern recognition systems , Pattern perception , Machine learning | ||||||||
Journal or Publication Title: | Artificial Intelligence | ||||||||
Publisher: | Elsevier BV | ||||||||
ISSN: | 0004-3702 | ||||||||
Official Date: | March 2023 | ||||||||
Dates: |
|
||||||||
Volume: | 316 | ||||||||
Number of Pages: | 23 | ||||||||
Article Number: | 103843 | ||||||||
DOI: | 10.1016/j.artint.2022.103843 | ||||||||
Status: | Peer Reviewed | ||||||||
Publication Status: | Published | ||||||||
Access rights to Published version: | Open Access (Creative Commons) | ||||||||
Date of first compliant deposit: | 31 January 2023 | ||||||||
Date of first compliant Open Access: | 31 January 2023 |
Request changes or add full text files to a record
Repository staff actions (login required)
View Item |
Downloads
Downloads per month over past year