Dahlia Janan, and Wray, David, 1950- (2012) Guidance on the principles of language accessibility in National Curriculum Assessments : research background. Coventry: Office of Qualifications and Examinations Regulation..

Preview

Text WRAP_Wray_2012-10-12-principles-of-language-accessibility-in-ncas.pdf - Published Version Download (2403Kb) | Preview

Abstract

This review accompanies the document, which describes the principles which should guide the development of clear assessment questions. The purpose of the review is to present and discuss in detail the research underpinning these principles. It begins from the standpoint that National Curriculum assessments, indeed any assessments, should be: - appropriate to the age of the pupils - an effective measure of their abilities, skills and concept development - fair to all irrespective of gender, language, religion, ethnic or social origin or disability. (Ofqual, 2011) The Regulatory Framework for National Assessments: National Curriculum and Early Years Foundation Stage (Ofqual, 2011) sets out a number of common criteria which apply to all aspects of the development and implementation of National Assessments. One of these criteria refers to the need for assessment procedures to minimise bias: “The assessment should minimise bias, differentiating only on the basis of each learner’s ability to meet National Curriculum requirements” (Section 5.39, page 16). The Framework goes on to argue that: “Minimising bias is about ensuring that an assessment does not produce unreasonably adverse outcomes for particular groups of learners” (Annex 1, page 29). This criterion reinforces the guiding principle that any form of assessment should provide information about the knowledge and understanding of relevant content material. That is to say that the means through which this knowledge and understanding is examined, the design of the assessment and the language used should as far as possible be transparent, and should not influence adversely the performance of those being assessed. There is clearly a large number of ways in which any given assessment task can be presented and in which questions can be asked. Some of these ways will make the task more accessible – that is, easier to complete successfully – and some will get in the way of successful completion. Section 26 of the Fair Access by Design (Ofqual, 2010) document lists a number of guiding principles for improving the accessibility of assessment questions, although the research basis for these principles is not made completely clear in that document. The aim of the current review is to examine the research background more closely in order to provide a more substantial basis for a renewed set of principles to underpin the concept of language accessibility. In the review, each section will be prefaced by a statement of the principles outlined in Guidance on the Principles of Language Accessibility in National Curriculum Assessments and then the research evidence underpinning these principles will be reviewed.

Item Type:	Report
Subjects:	L Education > LC Special aspects of education
Divisions:	Faculty of Social Sciences > Institute of Education
Library of Congress Subject Headings (LCSH):	High schools -- Great Britain -- Examinations, Elementary schools -- Great Britain -- Examinations, Examinations -- Questions, Examinations -- Language
Publisher:	Office of Qualifications and Examinations Regulation
Place of Publication:	Coventry
Date:	September 2012
Number:	Ofqual/12/5217
Number of Pages:	62
Status:	Not Peer Reviewed
Publication Status:	Published
Access rights to Published version:	Open Access
References:	Abedi, J., Kao, J., Leon, S., Mastergeorge, A., Sullivan, L., Herman, J. and Pope, R. (2010) Accessibility of Segmented Reading Comprehension Passages for Students with Disabilities. Applied Measurement in Education, 23: 2, 168–186. Abedi, J., Leon, S. and Kao, J. C. (2008) Examining Differential Item Functioning in Reading Assessments for Students with Disabilities (CRESST Report 744). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing. Abedi, J., Leon, S. and Mirocha, J. (2003) Impact of Student Language Background on Content-based Performance: Analyses of Extant Data (CSE Technical Report 603). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing. Abedi, J., Lord, C., Hofstetter, C. and Baker, E. (2000) Impact of Accommodation Strategies on English Language Learners’ Test Performance. Educational Measurements: Issues and Practice, 19: 3, 16–26. Ahmed, A. and Pollitt, A. (2007) Improving the Quality of Contextualized Questions: and Experimental Investigation of Focus. Assessment in Education: Principles, Policy and Practice, 14: 2, 201–232. Anderson, R., Wilkinson, I. and Mason, M. (1987) Do Errors on Classroom Reading Tasks Slow Growth in Reading? Center for the Study of Reading Technical Report No. 404. Champaign: University of Illinois at Urbana-Champaign. Assessment Performance Unit (1985) A Review of Monitoring in Mathematics 1978–82. Slough: NFER. Bailey, A. (2000) Language Analysis of Standardized Achievement Tests: Considerations in the Assessment of English Language Learners. In Abedi, J., Bailey, A., Butler, F., Castellon-Wellington, M., Leon, S. and Mirocha, J. (eds) The Validity of Administering Large-scale Content Assessments to English Language Learners: an Investigation from Three Perspectives (CSE Report 663) 85–105. Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing. Barshi, I. and Healy, A. (2002) The Effects of Mental Representation on Performance in a Navigation Task. Memory and Cognition, 30, 1189–1203. Beck, I., McKeown, M., Sinatra, G. and Loxterman, J. (1991) Revising Social Studies Text from a Text-processing Perspective: Evidence of Improved Comprehensibility. Reading Research Quarterly, 26, 251–276. Bügel, K. and Buunk, B. (1996) Sex Differences in Foreign Language Text Comprehension: The Role of Interests Knowledge. The Modern Language Journal, 80: 1, 15–31. Butler, F. and Stevens, R. (1997) Accommodation Strategies for English Language Learners on Large-Scale Assessments: Student Characteristics and Other Considerations (CSE Technical Report 448). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing. Chall, J. (1976) The Great Debate: Ten Years Later, with a Modest Proposal for Reading Stages. Paper presented at the Conference on Theory and Practice of Beginning Reading Instruction, University of Pittsburgh, Learning Research and Development Center, April 1976. Chall, J. S. and Conard, S. (1991) Should Textbooks Challenge Students? The Case for Easier or Harder Books. New York: Teachers College Press. Chall, J. S. and Dale, E. (1995) Readability revisited: the New Dale-Chall Readability Formula. Cambridge, Mass.: Brookline Books. Chan, D., Schmitt, N., DeShon, R., Clause, C. and Delbridge, K. (1997) Reactions to Cognitive Ability Tests: the Relationships between Race, Test Performance, Face valIdity Perceptions, and Test Taking Motivation. Journal of Applied Psychology, 82, 300–310. Chapman, L.J. (1987) Reading from 5–11 Years. London: Open University Press. Chi, M. T. H. (1985) Interactive Roles of Knowledge and Strategies in the Development of Organized Sorting and Recall. In Chipman, S. F. Segal, J. W. and Glaser, R. (eds), Thinking and Learning Skills (Vol. 2, pp. 457–484). Hillsdale, NJ: Erlbaum. Clay, M. (1977) Reading: The Patterning of Complex Behaviour. Portsmouth, NH: Heinemann. Cole, N. and Zieky, M. (2001) The New Faces of Fairness. Journal of Educational Measurement, 38: 4, 369–382. Connor, U. (1996) Contrastive Rhetoric: Cross-cultural Aspects on Second-language Writing. Cambridge: Cambridge University Press. Crisp, V. (2011) Exploring Features That Affect the Difficulty and Functioning of Science Exam Questions for Those with Reading Difficulties. Irish Educational Studies, 30: 3, 323–343. Crisp, V. and Sweiry, E. (2006) Can a Picture Ruin a Thousand Words? The Effects of Visual Resources in Exam Questions. Educational Research, 48: 2: 139–154. Dale, E. and Chall, J.S. (1948) A Formula for Predicting Readability. Educational Research Bulletin, 11–20, 27, 28, 37–54. Department of Education and Science (1975) A Language for Life (the Bullock Report). London: HMSO. Dorans, N. J. and Kulick, E. M. (1983) Assessing Unexpected Differential Item Performance of Female Candidates on SAT and TSWE Forms Administered in December 1977: an Application of the Standardization Approach (ETS Technical Report RR-83-9). Princeton, NJ: ETS. Downing, J. and Leong, C. (1982) Psychology of Reading. New York: Macmillan. Ecalle, J., Bouchafa, H., Potocki, A. and Magnan, A. (2011) Comprehension of Written Sentences as a Core Component of Children’s Reading Comprehension. Journal of Research in Reading. Article first published online: 6 May 2011. Available at http://onlinelibrary.wiley.com/doi/10.1111/j.1467-9817.2011.01491.x/full (accessed 28/8/12). Edwards, R. and Gibbon, V. (1973) Words Your Children Use: A Survey of the Words Used by Children in Infants’ Schools with the Resultant Graded Vocabulary. London: Burke. Eyles, P., Skelly, J. and Lou Schmuck, M. (2003) Evaluating Patient Choice of Typeface Style and Font Size for Written Health Information in an Outpatient Setting. Clinical Effectiveness in Nursing, 7: 2, 94–98. Fang, Z. (2006) The Language Demands of Science Reading in Middle School. International Journal of Science Education, 28: 5, 491–520. Feely, M., Rubin, G. S., Ekstrom, K. and Perera, S. (2005) Investigation into Font Characteristics for Optimum Reading Fluency in Readers with Sight Problems. International Congress Series, 1282, 530–533. Flesch, R. (1948) A New Readability Yardstick. Journal of Applied Psychology, 32, 221–233. Fisher-Hoch, H., Hughes, S. and Bramley, T. (1997) What Makes GCSE Exam Questions Difficult? Outcomes of Manipulating Difficulty of GCSE Questions. Paper presented at the British Educational Research Association Annual Conference, September 11–14, University of York. Francis, W. and Kucera, H. (1979) Brown Corpus Manual. Providence, Rhode Island: Brown University. Available at http://icame.uib.no/brown/bcm.html (accessed 16/8/12). Fry, E. (1977) Elementary Reading Instruction. New York: McGraw-Hill. Fulcher, G. (1989) Cohesion and Coherence in Theory and Reading Research. Journal of Research in Reading, 12: 2, 146–163. Gambrell, L., Wilson, R. and Gantt, W. (1981) Classroom Observations of Task-attending Behaviors of Good and Poor Readers. Journal of Educational Research, 74, 400–404. Gierl, M. (2005) Using Dimensionality-based DIF Analyses to Identify and Interpret Constructs That Elicit Group Differences. Educational Measurement: Issues and Practice, 24, 3–14. Gierl, M., Bisanz, J., Bisanz, G. and Boughton, K. (2003) Identifying Content and Cognitive Skills That Produce Gender Differences in Mathematics: a Demonstration of the Multidimensionality-based DIF Analysis Paradigm. Journal of Educational Measurement, 40: 4, 281–306. Gilliland, J. (1975) Readability. London: Hodder and Stoughton. Graesser, A., McNamara, D. and Louwerse, M. (2003) What Do Readers Need to Learn in Order to Process Coherence Relations in Narrative and Expository Text. In Sweet, A. P. and Snow, C. E. (eds) Rethinking Reading Comprehension (pp. 82–98). New York: Guilford. Gunning, R. (1952) The Technique of Clear Writing. New York, NY: McGraw-Hill International Book Co. Guthrie, J. and Cox, K. (2001) Classroom Conditions for Motivation and Engagement in Reading. Educational Psychology Review, 13: 3, 283–302. Guthrie, J. T., Solomon, A. and Rinehart, J. M. (1997) Literacy Issues in Focus: Engagement in Reading for Young Adolescents. Journal of Adolescent and Adult Literacy, 40: 6, 438–446. Halliday, M., and Hasan, R. (1976) Cohesion in English. New York: Longman. Hamilton, L. (1998) Gender Differences on High School Science Achievement Tests: Do Format and Content Matter? Educational Evaluation and Policy Analysis, 20: 3, 179–195. Harris, T. L. and Hodges, R. (1995) The Literacy Dictionary: the Vocabulary of Reading and Writing. Newark, Del.: International Reading Association. Harrison, C. (1984) Readability in the classroom. Cambridge: Cambridge University Press. Hughes, L. E. and Wilkins, A. J. (2000) Typography in Children’s Reading Schemes May Be Suboptimal: Evidence from Measures of Reading Rate. Journal of Research in Reading, 23: 3, 314–324. Janan, D., Wray, D. and Pope, M. (2010) Paradigms in Readability Research. International Journal of Arts and Sciences, 3: 17, 19–29. Johnson, K. (1998) Readability: Measuring the Reading Age of Books and Other Reading Matter. Available at www.timetabler.com/reading.html (accessed 16/8/12). Johnston, P. H. (1984) Prior Knowledge and Reading Comprehension Test Bias. Reading Research Quarterly, 19: 3, 219–239. Just, M. and Carpenter, P. A. (1987) The Psychology of Reading and Language Comprehension. Newton, MA: Allyn and Bacon. Just, M. A., Carpenter, P. A. and Wooley, J. D. (1982) Paradigms and Processes in Reading Comprehension. Journal of Experimental Psychology, 111: 2, 228–238. Kieffer, M., Lesaux, N., Rivera, M. and Francis, D. (2009) Accommodations for English Language Learners Taking Large-scale Assessments: a Meta-analysis on Effectiveness and Validity. Review of Educational Research, 79: 3, 1168–1201. Kintsch, Walter (1974) The Representation of Meaning in Memory. Hillsdale: Lawrence Erlbaum. Kintsch, W. and Keenan, J. (1973) Reading Rate and Retention as a Function of the Number of Propositions in the Base Structure of Sentences. Cognitive Psychology, 5: 257–274. Kletzien, S. (1991) Strategy Use by Good and Poor Comprehenders Reading Expository Text of Differing Levels. Reading Research Quarterly, 26, 67–86. Louwerse, M. M. and Graesser, A. C. (2004) Coherence in Discourse. In Strazny, P. (ed.) Encyclopedia of Linguistics. Chicago: Fitzroy Dearborn. Lund, O. (1999) Knowledge Construction in Typography: The Case of Legibility Research and the Legibility of Sans Serif Typefaces. Thesis submitted for the degree of Doctor of Philosophy. Reading: The University of Reading, Department of Typography and Graphic Communication. McLaughlin, G. Harry (May, 1969) SMOG Grading — a New Readability Formula. Journal of Reading, 12 (8), 639–646. McNally, J. and Murray, W. (1968) Key Words to Literacy and the Teaching of Reading: a Basic Word List for Developing Early Reading and Writing Skills. London, Schoolmaster Publishing. Maccoby, E. (1990) Gender and Relationships: a Developmental Account. American Psychologist, 45: 4, 513–20. Millard, E. (1997) Differently Literate: Boys, Girls and the Schooling of Literacy. London: Falmer Press. Milone, M. (2008) The Development of ATOS: The Renaissance Readability Formula. Wisconsin Rapids, WI: Renaissance Learning. Morgan, P., Farkas, G. and Hibel, J. (2008) Matthew Effects for Whom? Learning Disability Quarterly, 31: 4, 187–198. Mullis, I. V. S., Martin, M. O., Gonzalez, E. J. and Kennedy, A. M. (2003) PIRLS 2001 International Report: IEA’s Study of Reading Literacy Achievement in Primary School. Chestnut Hill, MA: Boston College. Nation, P. and Waring, R. (1997) Vocabulary Size, Text Coverage and Word Lists. In Schmitt, N. and McCarthy, M. (eds) Vocabulary: Description, Acquisition and Pedagogy (6–19). Cambridge: Cambridge University Press. Available at www.lextutor.ca/research/nation_waring_97.html (accessed 16/8/12). Newbold, N. and Gillam, L. (2010) The Linguistics of Readability: The Next Step for Word Processing. Proceedings of the NAACL HLT 2010 Workshop on Computational Linguistics and Writing (June), 65–72. Available at http://www.aclweb.org/anthology/W/W10/W10-0409.pdf (accessed 28/8/12). Nickson, M. and Green, S. (1996) A Study of the Effects of Context in the Assessment of the Mathematical Learning of 10/11 Year Olds. Paper presented to the 1996 British Educational Research Association. Available at www.cambridgeassessment.org.uk/ca/ (accessed 16/8/12). O’Reilly, T. and McNamara, D. (2007) Reversing the Reverse Cohesion Effect: Good Texts Can Be Better for Strategic, High-knowledge Readers. Discourse Processes, 43, 121–152. Ofqual (2010) Fair Access by Design. Coventry: Ofqual. Ofqual (2011) Regulatory Framework for National Assessments: National Curriculum and Early Years Foundation Stage. Coventry: Ofqual. Ofqual (2012) Guidance on the Principles of Language Accessibility in National Curriculum Assessments. Coventry: Ofqual. Ofqual (2012) Guidance on Monitoring Access to National Curriculum Assessments. Coventry: Ofqual. Ofqual (2012) Monitoring Access to National Curriculum Assessments: Research Background. Coventry: Ofqual. Ozuru, Y., Dempsey, K. and McNamara, D. (2009) Prior Knowledge, Reading Skill, and Text Cohesion in the Comprehension of Science Texts. Learning and Instruction, 19, 228–242. Perera, K. (1980) The Assessment of Linguistic Difficulty in Reading Material, Educational Review, 32: 2, 151–161. Pillai, K. G., Katsikeas, C. S. and Presib, C. (2012) Print Advertising: Type Size Effects. Journal of Business Research. Volume 65, Issue 6 June 2012, pages 865-868. Pollitt, A., Hutchinson, C., Entwhistle, N. and de Luca, C. (1985) What Makes Examination Questions Difficult? Edinburgh: Scottish Academic Press. Rasinski, T. V. (1990) The Effects of Cued Phrase Boundaries on Reading Performance: a Review. Kent, Ohio: Kent State University, Department of Teacher Development and Curriculum Studies. Reid, J. (1972) Children’s Comprehension of Syntactic Features Found in Extension Readers. In Reid, J. (ed) Reading: Problems and Practices. London: Ward Lock. Robinson, J. (2010) The Effects of Test Translation on Young English Learners’ Mathematics Performance. Educational Researcher, 39: 8, 582–590. Roussos, L. and Stout, W. (1996) A Multidimensionality-based DIF Analysis Paradigm. Applied Psychological Measurement, 20, 355–371. Saarnio, D. A., Oka, E. R. and Paris, S. G. (1990) Developmental Predictors of Children’s Reading Comprehension. In Carr, T. H. and Levy, B. A. (eds), Reading and Its Development: Components Skills Approaches (57–79). San Diego, CA: Academic Press. Schagen, I. and Sainsbury, M. (1996) Multilevel Analysis of the Key Stage 1 National Curriculum Assessment Data in 1995. Oxford Review of Education, 22: 3, 265–272. Schmit, M. and Ryan, A. (1992) Test-taking Dispositions: a Missing Link? Journal of Applied Psychology, 77, 629–637. Senter, R.J. and Smith, E. A. (November, 1967) Automated Readability Index. Wright-Patterson Air Force Base. p. iii. AMRL-TR-6620. Available online at www.dtic.mil/cgi-bin/GetTRDoc?AD=AD0667273 (accessed 28/8/12). Shorrocks-Taylor, D., Curry, J., Swinnerton, B. and Nelson, N. (2003) National Curriculum Mathematics Tests in England at Key Stage 2: Weights and Measures? Oxford Review of Education, 29: 1, 51–66. Shuard, H. and Rothery, A. (1984) Children Reading Mathematics. London: John Murray. Sonnleitner, P. (2008) Using the LLTM to Evaluate an Item Generating System for Reading Comprehension. Psychology Science Quarterly, 50, 345–362. Spache, G. (1953) A New Readability Formula for Primary-Grade Reading Materials. The Elementary School Journal, 53 (7), 410–413. Spache, G. (1974) Good Reading for Poor Readers. Champaign, IL: Garrard. Stanovich, K. (1986) Matthew Effects in Reading: Some Consequences of Individual Differences in the Acquisition of Literacy. Reading Research Quarterly, 21: 4, 360–407. Stone, E., Cook, L., Laitusis, C. C. and Cline, F. (2010) Using Differential Item Functioning to Investigate the Impact of Testing Accommodations on an English-Language Arts Assessment for Students Who Are Blind or Visually Impaired. Applied Measurement in Education, 23: 2, 132–152. Stricker. L. and Emmerich, W. (1999) Possible Determinants of Differential Item Functioning: Familiarity, Interest, and Emotional Reaction. Journal of Educational Measurement, 36: 4, 347–366. Thompson, C., and Shapiro, L. (2007) Complexity in Treatment of Syntactic Deficits. American Journal of Speech-Language Pathology, 18, 30–42. Thompson, S. J., Thurlow, M. L. and Malouf, D. (2004) Creating Better Tests for Everyone Through Universally Designed Assessments. Journal of Applied Testing Technology. 6: 1. Available at http://www.testpublishers.org/assets/documents/volume%206%20issue%201%20Creating%20%20better%20tests.pdf (accessed 28/8/12). Tinker, M. A. (1963) Legibility of Print. Iowa: Iowa State University Press. Waller, R. H. W. (1991) Designing Reports and Presentations. Redhill: Monotype Desktop Solutions. Wellington, J. and Osborne, J. (2001) Language and Literacy in Science Education. Buckingham: Open University Press. White, A. (2004) Thinking in Type: The Practical Philosophy of Typography. New York: Allworth Press. Wilkins, A., Cleave, R., Grayson, N. and Wilson, L. (2009) Typography for Children May Be Inappropriately Designed. Journal of Research in Reading, 32: 4, 402–412. Zumbo, B. (2007) Three Generations of DIF Analyses: Considering Where It Has Been, Where It Is Now, and Where It Is Going. Language Assessment Quarterly, 4: 2, 223–233.
URI:	http://wrap.warwick.ac.uk/id/eprint/51269

Request changes to a record

Actions (login required)

View Item