Skip to content Skip to navigation
University of Warwick
  • Study
  • |
  • Research
  • |
  • Business
  • |
  • Alumni
  • |
  • News
  • |
  • About

University of Warwick
Publications service & WRAP

Highlight your research

  • WRAP
    • Home
    • Search WRAP
    • Browse by Warwick Author
    • Browse WRAP by Year
    • Browse WRAP by Subject
    • Browse WRAP by Department
    • Browse WRAP by Funder
    • Browse Theses by Department
  • Publications Service
    • Home
    • Search Publications Service
    • Browse by Warwick Author
    • Browse Publications service by Year
    • Browse Publications service by Subject
    • Browse Publications service by Department
    • Browse Publications service by Funder
  • Help & Advice
University of Warwick

The Library

  • Login
  • Admin

Does benchmarking of rating scales improve ratings of search performance given by specialist search dog handlers?

Tools
- Tools
+ Tools

Clark, Corinna C. A. and Rooney, Nicola J. (2021) Does benchmarking of rating scales improve ratings of search performance given by specialist search dog handlers? Frontiers In Veterinary Science, 8 . 545398. doi:10.3389/fvets.2021.545398 ISSN 2297-1769.

[img]
Preview
PDF
WRAP-does-benchmarking-rating-scales-improve-ratings-search-performance-given-specialist-search-dog-handlers-Clark-2021.pdf - Published Version - Requires a PDF viewer.
Available under License Creative Commons Attribution 4.0.

Download (269Kb) | Preview
Official URL: http://dx.doi.org/10.3389/fvets.2021.545398

Request Changes to record.

Abstract

Rating scales are widely used to rate working dog behavior and performance. Whilst behaviour scales have been extensively validated, instruments used to rate ability have usually been designed by training and practitioner organizations, and often little consideration has been given to how seemingly insignificant aspects of the scale design might alter the validity of the results obtained. Here we illustrate how manipulating one aspect of rating scale design, the provision of verbal benchmarks or labels (as opposed to just a numerical scale), can affect the ability of observers to distinguish between differing levels of search dog performance in an operational environment. Previous studies have found evidence for range restriction (using only part of the scale) in raters' use of the scales and variability between raters in their understanding of the traits used to measures performance. As provision of verbal benchmarks has been shown to help raters in a variety of disciplines to select appropriate scale categories (or scores), it may be predicted that inclusion of verbal benchmarks will bring raters' conceptualization of the traits closer together, increasing agreement between raters, as well as improving the ability of observers to distinguish between differing levels of search dog performance and reduce range restriction. To test the value of verbal benchmarking we compared inter-rater reliability, raters' ability to discriminate between different levels of search dog performance, and their use of the whole scale before and after being presented with benchmarked scales for the same traits. Raters scored the performance of two separate types of explosives search dog (High Assurance Search (HAS) and Vehicle Search (VS) dogs), from short (~30 s) video clips, using 11 previously validated traits. Taking each trait in turn, for the first five clips raters were asked to give a score from 1, representing the lowest amount of the trait evident to 5, representing the highest. Raters were given a list of adjective-based benchmarks (e.g., very low, low, intermediate, high, very high) and scored a further five clips for each trait. For certain traits, the reliability of scoring improved when benchmarks were provided (e.g., Motivation and Independence), indicating that their inclusion may potentially reduce ambivalence in scoring, ambiguity of meanings, and cognitive difficulty for raters. However, this effect was not universal, with the ratings of some traits remaining unchanged (e.g., Control), or even reducing in reliability (e.g., Distraction). There were also some differences between VS and HAS (e.g., Confidence reliability increased for VS raters and decreased for HAS raters). There were few improvements in the spread of scores across the range, but some indication of more favorable scoring. This was a small study of operational handlers and trainers utilizing training video footage from realistic operational environments, and there are potential cofounding effects. We discuss possible causal factors, including issues specific to raters and possible deficiencies in the chosen benchmarks, and suggest ways to further improve the effectiveness of rating scales. This study illustrates why it is vitally important to validate all aspects of rating scale design, even if they may seem inconsequential, as relatively small changes to the amount and type of information provided to raters can have both positive and negative impacts on the data obtained.

Item Type: Journal Article
Subjects: S Agriculture > SF Animal culture
Divisions: Faculty of Science, Engineering and Medicine > Science > Life Sciences (2010- )
Library of Congress Subject Headings (LCSH): Search dogs -- Training, Search dogs -- Training -- Vocational guidance, Dogs -- Behavior therapy , Detector dogs -- Training
Journal or Publication Title: Frontiers In Veterinary Science
Publisher: Frontiers Research Foundation
ISSN: 2297-1769
Official Date: 2 February 2021
Dates:
DateEvent
2 February 2021Published
8 January 2021Accepted
Volume: 8
Article Number: 545398
DOI: 10.3389/fvets.2021.545398
Status: Peer Reviewed
Publication Status: Published
Access rights to Published version: Open Access (Creative Commons)
Date of first compliant deposit: 3 February 2021
Date of first compliant Open Access: 4 February 2021
RIOXX Funder/Project Grant:
Project/Grant IDRIOXX Funder NameFunder ID
DFAS RK7283Ministry of Defencehttp://dx.doi.org/10.13039/501100006083

Request changes or add full text files to a record

Repository staff actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics

twitter

Email us: wrap@warwick.ac.uk
Contact Details
About Us