The Library
Calibration with confidence : a principled method for panel assessment
Tools
Tools
Tools
MacKay, Robert S., Kenna, R., Low, R. J. and Parker, S. (2017) Calibration with confidence : a principled method for panel assessment. Royal Society Open Science , 4 (2). 160760. doi:10.1098/rsos.160760 ISSN 2054-5703.
|
PDF
WRAP_160760.full.pd.pdf - Published Version - Requires a PDF viewer. Available under License Creative Commons Attribution 4.0. Download (1039Kb) | Preview |
Official URL: http://doi.org/10.1098/rsos.160760
Abstract
Frequently, a set of objects has to be evaluated by a panel of assessors, but not every object is assessed by every assessor. A problem facing such panels is how to take into account different standards among panel members and varying levels of confidence in their scores. Here, a mathematically based algorithm is developed to calibrate the scores of such assessors, addressing both of these issues. The algorithm is based on the connectivity of the graph of assessors and objects evaluated, incorporating declared confidences as weights on its edges. If the graph is sufficiently well connected, relative standards can be inferred by comparing how assessors rate objects they assess in common, weighted by the levels of confidence of each assessment. By removing these biases, ‘true’ values are inferred for all the objects. Reliability estimates for the resulting values are obtained. The algorithm is tested in two case studies: one by computer simulation and another based on realistic evaluation data. The process is compared to the simple averaging procedure in widespread use, and to Fisher's additive incomplete block analysis. It is anticipated that the algorithm will prove useful in a wide variety of situations such as evaluation of the quality of research submitted to national assessment exercises; appraisal of grant proposals submitted to funding panels; ranking of job applicants; and judgement of performances on degree courses wherein candidates can choose from lists of options.
| Item Type: | Journal Article | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Subjects: | Q Science > QA Mathematics | ||||||||
| Divisions: | Faculty of Science, Engineering and Medicine > Science > Mathematics | ||||||||
| Library of Congress Subject Headings (LCSH): | Algorithms, Educational tests and measurements, Peer review of research grant proposals, Employee selection | ||||||||
| Journal or Publication Title: | Royal Society Open Science | ||||||||
| Publisher: | The Royal Society Publishing | ||||||||
| ISSN: | 2054-5703 | ||||||||
| Official Date: | 8 February 2017 | ||||||||
| Dates: |
|
||||||||
| Volume: | 4 | ||||||||
| Number: | 2 | ||||||||
| Article Number: | 160760 | ||||||||
| DOI: | 10.1098/rsos.160760 | ||||||||
| Status: | Peer Reviewed | ||||||||
| Publication Status: | Published | ||||||||
| Access rights to Published version: | Open Access (Creative Commons) | ||||||||
| Date of first compliant deposit: | 2 March 2017 | ||||||||
| Date of first compliant Open Access: | 2 March 2017 | ||||||||
| Funder: | Economic and Social Research Council (Great Britain) (ESRC), Seventh Framework Programme (European Commission) (FP7) | ||||||||
| Grant number: | ES/K002201/1, ES/N012550/1 (ESRC), FP7-PEOPLE-2013-IRSES Programme (2014-2018) (FP7) |
Request changes or add full text files to a record
Repository staff actions (login required)
 |
View Item |
Downloads
Downloads per month over past year
