Standards in e-assessment

Technology is second nature to today’s learners. They must be allowed to embrace its potential and maximise the new opportunities it provides for them to demonstrate their achievements. The use of ICT in assessment raises issues of central concern to the regulator, but we must seek ways of dealing with the challenge rather than hiding from it.

One of the requirements of the Apprenticeship, Skills, Children and Learning Act is that Ofqual should ‘have regard to the desirability of facilitating innovation in connection with the provision of regulated qualifications’. As a regulator I want to recognise the ever-increasing impact that technology is having on the lives of learners and reflect it in the delivery and awarding of qualifications. Ofqual aims to facilitate innovation, including e-assessment, across the whole range of regulated qualifications while maintaining the integrity, reliability and validity of the assessment.

We must start by being realistic about what we mean by e-assessment and the current position regarding its use. For our purposes e-assessment is where the learner responds to questions or tasks on a computer. It does not necessarily mean that the computer carries out the assessment. Although computers can be used to make some assessments, the day when computers can be used to assess such matters as the quality of essays is very distant. Nor do we use the term ‘e-assessment’to mean the use of technology as a tool to support other aspects of the assessment process, such as on-screen marking or the standardisation of markers.

E-assessment is not new, having been used for many years in some vocational and professional qualifications in the UK and even more widely in the USA. However, it is still quite rare in large-scale, high-profile qualifications in the UK – particularly those used by schools such as the GCE and GCSE. There has been some progress but it has been slow. Our records show that in the GCSE and A levels in 2008, for example, only 421 centres used any form of e-assessment and there are only a few specifications that rely heavily upon it. Among the GCSEs that include an element of e-assessment are AQA’s science A, CCEA’s moving image arts, Edexcel’s construction and the built environment while WJEC offer e-assessments in AS level Applied Business and AS level Applied ICT.

The challenge for us is to enable the development of qualifications that make the best possible use of e-assessment while ensuring that standards are maintained. This must include, where appropriate, the comparability of standards between e-assessment and traditional pen-and­paper examinations.

New qualifications have the opportunity to use e-assessment from the start: they have no pen-and-paper versions against which to be compared. However, where the aim is to move from traditional assessment to e-assessment there will inevitably be a transition period during which some learners will be using the e-assessments while others are still using pen and paper. Ofqual needs to be able to assure users that the results of such assessments are comparable and that they need not be concerned about the mode of assessment.

There has been some research on the comparability of the different modes of candidates’ responses to questions and tasks. In many studies, particularly those using ‘simple’item types such as multiple-choice or short-answer questions, comparable scoring can be achieved. Even where there have been statistically significant differences, the effect has tended to be small. In other words, there may be a real difference but it is only very small.

We recognise that it is very hard for awarding organisations to demonstrate comparability between the modes in advance of carrying out the assessment, so we have reached a consensus with them on the technical issues involved and how to tackle them. We hope this will allow scope for innovative approaches to assessment. The consensus statement places the emphasis on making the on-screen assessment and the paper-based assessment as similar as possible. The type of question should be the same, the contexts should be as similar as possible and the amount of time available should be the same. In other words, the only substantive difference between the tests under consideration should be the mode of delivery and response.

Inevitably some candidates will be more familiar than others with the computer and its software. However, it appears that these problems can be overcome provided candidates have sufficient opportunity to acquaint themselves with the interface by taking a practice test or similar activity.

There must also be clear mechanisms to deal with cheating, which is often quoted as an argument against e-assessment. Care must be taken to ensure that a candidate cannot copy from a neighbour’s screen, make inappropriate use of the internet or exchange information if the assessment runs over a number of days. On the other hand there are techniques to combat cheating that are much easier to operate in e-assessment. Test versions can be created with different (but equivalent) questions, the order of questions can be changed and multiple choice options can appear in random order. These are issues that can be addressed, not insurmountable obstacles to e-assessment.

One question that needs to be considered is the number and length of ‘test windows’ – the time during which candidates can take their assessment. At present many test centres, and particularly schools, may not have the technology infrastructure available for all candidates, for a large examination to take an e-assessment at the same time. Is it better to have a longer time period or a number of single-day windows?

Perhaps the ultimate form of e-assessment is ‘on-demand’ testing where each assessment is unique to the candidate, tailored to his or her needs and available whenever the centre decides. Again developments in on-demand testing are more advanced in vocational and occupational assessments than in general qualifications.

Many of the organisations we have consulted suggest that on-demand testing is an essential approach to e-assessment. Where successful it gives the greatest flexibility to the learner and the centre to undertake the assessment when it is most appropriate. We have therefore commissioned researchers to develop some principles to underpin developments in this area¹. They have put forward principles designed to ensure the maintenance of examination standards, to improve accessibility for candidates, to monitor any changes in the burden of assessment and to ensure clear understanding of the processes by all involved. We are launching a project to develop and consult on a regulatory framework that facilitates on-demand testing using these principles as a basis.

We are committed to an ongoing dialogue with all those interested in developing e-assessment and on-demand testing. As part of this we have commissioned a number of pieces of work on various aspects and we have published, and will continue to publish, our findings. Some of the issues raised will be difficult but we are determined to move forward, recognising that there will be setbacks along the way. Learners will be making use of an ever-widening range of technologies and it is the responsibility of the assessment community to devise ways of assessing their knowledge, understanding and skills using the same media as they use in their schools, colleges and places of work.

As part of our communication strategy we will use e-Futures² a dedicated website on e-assessment, to help us reach all those interested in advancements. eFutures helps users move from basic awareness to strategic planning in e-assessment, offering advice and information on the essentials of e-assessment, e-testing and e-portfolios. It incorporates an online toolkit³ providing access to advice and examples of good practice that have brought plaudits from around the country.

Given our brief to encourage innovation I think it is necessary for the regulator to encourage the awarding organisations to plan for a future where there are technological solutions to current problems. We need to evaluate the obstacles to progress in developing new qualifications using e-assessment and the introduction of e-assessment to current qualifications. The agreements we have reached with the awarding organisations over comparability should make it easier for them to develop assessments that meet the requirements, even in a transition period. Awarding organisations must, however, be held to account for equality of access. So they will need to be sensitive to the needs and resources of the centres – whether they are schools, colleges or workplaces – to avoid placing undue burdens upon them. Above all, we must ensure that the needs of the learners are paramount. Their curriculum must not be distorted merely to make it easier to assess on a computer. Whatever its nature, assessment must cope with all the richness and diversity of the teaching and learning experience.

Ofqual expects that the awarding organisations will wish to be innovative and encourages them to be so by setting challenging performance standards and targets for them to achieve, subject to the readiness of the centres of learning to deliver. Some awarding organisations are and may continue to make faster progress than others. But if the market philosophy holds then the market will generate competition and ensure that all reach our expected standard more quickly than some of the awarding organisations – and some observers – would think possible.