Introduction
The Office of Qualifications and Examinations Regulation (Ofqual) is the regulator of qualifications (other than degrees), examinations and assessments in England, and of vocational qualifications in Northern Ireland. We aim to make sure that learners get the results their work merits and that the qualifications they receive count, both now and in the future.
In 2010, Awarding Organisations (AOs) the Qualifications and Curriculum Development Agency (QCDA) and organisations representing disabled learners indicated they would welcome guidance on the language accessibility of, and modifications to, assessment papers. It was suggested that Ofqual should take a lead in providing such guidance.
We have in the past contributed to guidance such as Fair access by design (2010). Fair access by design focuses almost exclusively on sentence difficulty and legibility aspects of assessments. New research suggests, however, that readability problems can be caused by other textual factors. Factors to do with textual cohesion and coherence have been especially neglected in the past. We are therefore consulting on guidance on what should represent good practice in this area.
Subject to the views expressed in the consultation, we will revise and publish the guidance. This guidance must be considered in the context of the form of the assessment and the knowledge, skills and understanding being assessed.
We propose that the guidance could be used by test development agencies (for National Curriculum Assessments) and by awarding organisations (for qualifications). However, its use would be optional. We recognise that some of the approaches explored in the consultation, such as Computer Adaptive Tests, would not be suitable or desirable for all assessments.
The Guidance might also be useful for those who train subject experts and others who prepare the assessments.
The present consultation is a result of work on the accessibility of assessments carried out by Ofqual since 2009. In 2009, Ofqual carried out a research project on the Current Practices and operational aspects of paper modification in England 2009/10 leading to a publication on The future of language accessibility in the UK.
About this consultation
This consultation on language accessibility considers National Curriculum Assessment (NCA) tests in sections 1 and 2 and qualifications in sections 3, 4 and 5.
We wish to reach, and receive responses from as many stakeholder groups as possible. These include, but are not limited to: test development agencies (TDAs), principal examiners from awarding organisations who write and quality assure assessment items; awarding organisations, learners and organisations and individuals who represent their interests, speech and language therapists, diversity and inclusion specialists, and other stakeholders representing the diverse equality strands. Our aim is to build up strong relationships with our stakeholders which will last after this consultation has ended.
It is disappointing that this consultation assumes the continued provision of paper based modifications to tests is fair and complies with the Equalities Act (2010). Many European countries provide a digital version of the paper test that enables greater access for a range of students with disabilities. Also the DDA and the Equalities Act specifically require that provision is re-evaluated in the light of technological developments.
Response 1: We think that a move towards digital versions of tests is warranted from a traditional equal opportunities perspective. Many children will nowadays be more familiar with making responses via a screen than they will via paper (Facebook, online gaming, texting etc.) So if we want to give all children the best opportunity to show what they really know in testing situations, then the medium of the testing is important. Some learners do much better using screens so they should perhaps be allowed that choice in tests. (David Wray)
Response 2: Please read Consultation Section 5 and its background report written by Michael Birdsall for Ofqual. Section 5 is exactly about awarding organisations using Computer adaptive Testing as a way of mitigating bias in assessments . This would be our long term perspective. (Fatima Carvalho)
Response 3: I think that there are a number of issues here that need to be disentangled.
Unfortunately, Birdsall’s paper focuses on computer adaptive tests. These represent only one particular type of computer-based assessment. In a computer adaptive test the pupil is given one or two questions of ‘average’ difficulty (or a bit below). Then, depending on whether the pupil gets the questions right or wrong the computer presents harder or easier questions until they find the ‘level of difficulty’ that is appropriate for that pupil.
Computer adaptive tests raise a number of issues. I wrote about some of these problems in my comments last March, but these do not seem to have been noted.
I think it is a great pity that Birdsall’s paper went onto the Consultation website as it stood. It conflates computer adaptive tests with all other types of computer-based assessment. It is, I think, irrelevant to the criticisms raised by this contributor.
The contributor is suggesting that there should be ‘a digital version of the paper test that enables greater access for a range of students with disabilities’. This is a very different thing from a computer adaptive test. A digital version of the paper test would present the original paper-based version of each question on screen. This would allow pupils to enlarge the type face and perhaps to change the background colour and contrast. This might improve access for some pupils. It might also be possible to provide an aural recording of the test questions to improve access for pupils with limited reading skills. All this would be worth considering although it could create new problems, for example with mathematical diagrams that would be the wrong size or with questions that spread over two pages. But at a basic level providing on-screen access to PDFs of the question papers would be quite straightforward, and might be helpful to some pupils. However, it has nothing to do with computer adaptive testing.
David addresses a different point when he observes that ‘Many children will nowadays be more familiar with making responses via a screen than they will via paper.’ This is true, but it is not what the teacher who made the comment was suggesting. ‘Making responses via a screen’ sounds straightforward, but it is really quite a complex issue. In the English reading test many of the questions do require a textual response, so for at least some of these pupils might be able to type rather than write their answers – although this would not always be possible. But in mathematics the assessment is very often as much in the working as in the answer. So, for example, even quite a simple problem requiring a multiplication might carry two marks, one for selecting the correct numbers and attempting to multiply them, and one for doing the multiplication and getting the right answer. But ’showing your working’ on screen can be very difficult. For example, consider the question:
Tom has 37 boxes of books with 25 books in each box.
How many books does he have? (2 marks)
The first mark here is for indicating that the correct computation is to multiply (not add, divide or subtract) the numbers 37 and 25, and for showing an appropriate method. This method might be a conventional layout, with the 25 written beneath 37, then two lines of working, then a line showing the sum. This is the way that many adults would do it, but pupils might adopt a different, but still correct (in fact more efficient), strategy. The test developer would want to allow pupils to lay out their working as they wished. Setting the numbers one beneath the other, lining up the units, tens and hundreds, is very tricky on-screen: it is much easier to do working like this on paper. But if the pupil is just required to type in their final answer, and only this is marked, then the marker will not be able to judge whether they have used a correct method even if they have got the final answer wrong.
I have given just one rather simplistic example of what can happen if you ask pupils to respond on screen to questions which were designed to be answered on paper. Any complex, multi-step question is likely to present serious problems. If we want pupils to respond on-screen then the questions must be designed from the beginning with this in mind. We cannot simply put print-based questions up on screen and ask pupils to type in their responses. I do not think that we should suggest to the consultation forum that this would be a good approach.
But in any case, the questions in Annex 4 of the consultation document relate only to computer adaptive tests, not to any other form of computer-based assessment. Furthermore, they seem to be addressed exclusively to test developers, not to teachers and others who are working with pupils with special educational or assessment needs. This being the case I do not see how they can be considered relevant to the point raised by the teacher who made this comment. This teacher suggested that there should be ‘a digital version of the paper test’, not a completely different type of assessment such as computer adaptive test. (Tandi Clausen-May)
Like or Dislike:
0 
0
Dear Jim,
Please view responses to your comment.Response 1: We think that a move towards digital versions of tests is warranted from a traditional equal opportunities perspective. Many children will nowadays be more familiar with making responses via a screen than they will via paper (Facebook, online gaming, texting etc.) So if we want to give all children the best opportunity to show what they really know in testing situations, then the medium of the testing is important. Some learners do much better using screens so they should perhaps be allowed that choice in tests. (David Wray)
Response 2: Please read Consultation Section 5 and its background report written by Michael Birdsall for Ofqual. Section 5 is exactly about awarding organisations using Computer adaptive Testing as a way of mitigating bias in assessments . This would be our long term perspective. (Fatima Carvalho)
Response 3:(Paul Simpson)Interesting point but very little attention has yet been paid to the implications of computer-based assessment on deaf, visually impaired candidates and those with linguistic difficulties – a lot more work and impact assessments need to be done. There is very little guidance currently about this. Unless someone can point me to it?
Response 4: I think that there are a number of issues here that need to be disentangled.
Unfortunately, Birdsall’s paper focuses on computer adaptive tests. These represent only one particular type of computer-based assessment. In a computer adaptive test the pupil is given one or two questions of ‘average’ difficulty (or a bit below). Then, depending on whether the pupil gets the questions right or wrong the computer presents harder or easier questions until they find the ‘level of difficulty’ that is appropriate for that pupil.
Computer adaptive tests raise a number of issues. I wrote about some of these problems in my comments last March, but these do not seem to have been noted.
I think it is a great pity that Birdsall’s paper went onto the Consultation website as it stood. It conflates computer adaptive tests with all other types of computer-based assessment. It is, I think, irrelevant to the criticisms raised by this contributor.
The contributor is suggesting that there should be ‘a digital version of the paper test that enables greater access for a range of students with disabilities’. This is a very different thing from a computer adaptive test. A digital version of the paper test would present the original paper-based version of each question on screen. This would allow pupils to enlarge the type face and perhaps to change the background colour and contrast. This might improve access for some pupils. It might also be possible to provide an aural recording of the test questions to improve access for pupils with limited reading skills. All this would be worth considering although it could create new problems, for example with mathematical diagrams that would be the wrong size or with questions that spread over two pages. But at a basic level providing on-screen access to PDFs of the question papers would be quite straightforward, and might be helpful to some pupils. However, it has nothing to do with computer adaptive testing.
David addresses a different point when he observes that ‘Many children will nowadays be more familiar with making responses via a screen than they will via paper.’ This is true, but it is not what the teacher who made the comment was suggesting. ‘Making responses via a screen’ sounds straightforward, but it is really quite a complex issue. In the English reading test many of the questions do require a textual response, so for at least some of these pupils might be able to type rather than write their answers – although this would not always be possible. But in mathematics the assessment is very often as much in the working as in the answer. So, for example, even quite a simple problem requiring a multiplication might carry two marks, one for selecting the correct numbers and attempting to multiply them, and one for doing the multiplication and getting the right answer. But ’showing your working’ on screen can be very difficult. For example, consider the question:
Tom has 37 boxes of books with 25 books in each box.
How many books does he have? (2 marks)
The first mark here is for indicating that the correct computation is to multiply (not add, divide or subtract) the numbers 37 and 25, and for showing an appropriate method. This method might be a conventional layout, with the 25 written beneath 37, then two lines of working, then a line showing the sum. This is the way that many adults would do it, but pupils might adopt a different, but still correct (in fact more efficient), strategy. The test developer would want to allow pupils to lay out their working as they wished. Setting the numbers one beneath the other, lining up the units, tens and hundreds, is very tricky on-screen: it is much easier to do working like this on paper. But if the pupil is just required to type in their final answer, and only this is marked, then the marker will not be able to judge whether they have used a correct method even if they have got the final answer wrong.
I have given just one rather simplistic example of what can happen if you ask pupils to respond on screen to questions which were designed to be answered on paper. Any complex, multi-step question is likely to present serious problems. If we want pupils to respond on-screen then the questions must be designed from the beginning with this in mind. We cannot simply put print-based questions up on screen and ask pupils to type in their responses. I do not think that we should suggest to the consultation forum that this would be a good approach.
But in any case, the questions in Annex 4 of the consultation document relate only to computer adaptive tests, not to any other form of computer-based assessment. Furthermore, they seem to be addressed exclusively to test developers, not to teachers and others who are working with pupils with special educational or assessment needs. This being the case I do not see how they can be considered relevant to the point raised by the teacher who made this comment. This teacher suggested that there should be ‘a digital version of the paper test’, not a completely different type of assessment such as computer adaptive test. (Tandi Clausen-May)
R
Like or Dislike:
0 
0