Harsch, Claudia and Martin, Guido. (2012) Adapting CEF-descriptors for rating purposes: Validation by a combined rater training and scale revision approach. Assessing Writing, Vol. 17 (No. 4). pp. 228-250. ISSN 1075-2935

Abstract

We explore how a local rating scale can be based on the Common European Framework CEF-proficiency scales. As part of the scale validation (Alderson, 1991; Lumley, 2002), we examine which adaptations are needed to turn CEF-proficiency descriptors into a rating scale for a local context, and to establish a practicable method to revise the initial scale-draft while simultaneously training raters for trialling the rating scale. We propose a data-driven approach focusing on the descriptors, involving raters in the scale revision as part of the scale validation process (Barkaoui, 2010a; Cohen, 1994). Rating data were analysed during a two-month period, employing 19 writing tasks and 13 raters; individual descriptors and raters were monitored for consistency. Our approach facilitated identifying causes of low agreement, let it be raters or scale wording, which in turn allowed targeted feedback to raters and revisions of scale-wording where indicated. We conclude with a summary of necessary adaptations of CEF-descriptors for rating purposes, and we outline further steps to validate the revised rating scale. We aim to contribute to improve rater training and rating scale development based on the CEF, since the literature for ‘real’ test development contexts seems rather scarce (Barkaoui, 2010b; Knoch, 2011a,b).

Request changes to a record

Actions (login required)

View Item