Unfinished business – an update

Hundreds of thousands of learners have achieved qualifications and national curriculum assessment results since my last report in May. I would like to offer my congratulations to all these successful candidates and their teachers.

We are all familiar with the high profile given in August to the A level and GCSE results, followed in September by the reporting of the national curriculum key stage tests. What is frequently overlooked is the achievement of those learners who each day gain thousands of vocational and occupational qualifications. No fewer than five million vocational and occupational qualifications – known as VQs to distinguish them from GQs or general qualifications – were awarded in the 12 months to June. It is a record number but the lack of a single ‘results day’ means they receive much less attention from the media.

Vocational and occupational qualifications are valued by thousands of learners and employers in all walks of life. They are an important part of many people’s career development, giving recognition for the knowledge and skills acquired in the course of employment and providing an incentive to move forward. Not only do these qualifications open doors and lead to better opportunities for individuals but they also make a vital contribution to the success of the nation.

This is why VQ Day in June was an important opportunity to recognise the achievements of the people who achieved these diverse and wide-ranging qualifications. Celebrations were held in Belfast, Cardiff, Edinburgh and London, at which VQ Heroes – VQ Learners and Young Learners of the year – for each country were announced. Reading about the challenges many of these successful learners faced during their courses is a salutary lesson for us all.

I was delighted to have the opportunity to visit two colleges on VQ Day: North Warwickshire & Hinckley College in Nuneaton and Warwickshire College in Leamington Spa. Ofqual also attended the VQ Day event for England where a total of 18 regional VQ Heroes collected their certificates and the national award winners were announced.

In Belfast our Ofqual team took part in a ceremony at the Waterfront Hall where Sir Reg Empey, Northern Ireland’s Employment and Learning Minister, announced the Northern Ireland Learner and Young Learner of the Year.

The next VQ Day will be on Wednesday 23 June 2010. Let us all put the date in our diaries.

I should also like to pay tribute to the learners who achieved their Diplomas this year. Diplomas focus on applied learning and bring together attainments across a range of qualifications. The Diploma is not just a new qualification; it is a new approach in 14–19 qualifications, a ‘meta-qualification’. To obtain a Diploma, learners must have:

• undertaken the principal learning qualification in the area covered by the Diploma and a project relevant to the theme of the Diploma
• achieved passes in functional skills qualifications in English, mathematics and ICT at the appropriate level
• completed a range of qualifications showing their achievement in areas of additional and specialist learning
• undertaken a significant period of work experience
• developed personal, learning and thinking skills that are needed for success in learning and life.

Those learners who expected to complete their Diploma this year but did not manage to do so should not be too disheartened; all the qualifications they did achieve remain valid and show what they have achieved. And what is more, when they do complete the remaining ones – and I urge them to persevere – they will be eligible for the Diploma, whether it takes another week, month, year or two or even more.

The fact that the Diploma is such a new approach raises significant issues for Ofqual. I discuss these in more detail in Section 2.

Public confidence in qualifications, examinations and assessments is essential. In my May 2009 report I noted some outstanding issues that needed to be addressed in order to safeguard this confidence. In this section I want to bring you up to date with what we have done to ensure that the qualifications and assessments were of high quality and delivered accurately and on time.

Ofqual has been monitoring both the quality of the assessment processes and the robustness of the delivery systems. I am pleased to report that 99.9 per cent of the key stage 2 national curriculum assessment results were delivered on time. Our monitoring of the quality of the marking process indicates that it was satisfactory. It was particularly important to ensure that markers were consistent this year following the discontinuation of the borderlining procedure, which had been used for some years to re-check scripts falling close to the cut-off score for a level. However, changes to the methods by which markers were checked for accuracy led to a larger than expected number of markers being rejected, particularly in English and science. We are investigating the cause of the high level of rejections as part of our qualitative research into 2009 assessment procedures to ensure that the approach is truly improving the quality of the assessment.

A new approach to checking the accuracy of markers is being used in the pilot assessments of the single level tests¹. This involves marking the test papers on the marker’s computer screen, which is a common feature of many examinations but has not been used in key stage assessment before. One advantage is that markers can be checked more frequently. They do not even need to know when the standard of their work is being monitored by senior examiners, making the process more objective and reliable. Experience at GCSE, where this approach has been used for some years, suggests that this leads to more consistent marking. We are discussing with ministers and the Qualifications and Curriculum Development Agency (QCDA) how this might be taken forward in national curriculum assessments.

I also reported a number of issues relating to the GCSE sciences. Bearing in mind the concerns that Ofqual voiced earlier in the year, we have been working with the QCDA and the awarding organisations to ensure that the criteria and specifications for GCSE science are improved. QCDA has undertaken a consultation on revised criteria and new specifications meeting the new requirements will be produced over the next year and published in time for teaching to start in September 2011. At the same time the government’s Science and Learning Expert Group – chaired by Sir Mark Walport – has consulted on more general issues relating to science (including mathematics) education pre-19.

Revisions to the criteria and specifications are important for the future but there is still the question of what to do for those learners who are following the current specifications. We cannot afford to allow these learners to suffer because of the flawed specifications. Ofqual has therefore been working closely with the awarding organisations. We have required them to improve the quality of papers, ensuring that they give all candidates the opportunity to show what they can do and giving better opportunities for more able candidates to demonstrate their talents. In monitoring the 2009 awards we used information about the learners’ achievements at key stage 3 to compare results across the awarding organisations in England. Overall we were satisfied that the results fairly reflected the candidates’ performance.

There are some things that we cannot change in the short term. Many candidates had already taken and been graded on most of the units necessary to achieve the qualification. With only a few units left, it is impossible to make much of a difference to the standard of the award or address failings in those units already taken. The issues vary across the awarding organisations and our aim has been to maintain year-on-year standards while mitigating as far as possible the anomalies we identified in 2008.

We will continue to keep a close eye on the situation over the next few years. We will attend many of the awarding meetings and facilitate discussions with the technical staff of the awarding organisations on how best to evaluate results. We will also gather evidence of the quality of questions from an examination of the candidates’ answers. In this way we will protect the interests of the learners currently involved until the full reforms can be implemented.

This year has also seen the first awards of the new AS qualifications, which are based generally on two instead of three units. Carrying forward standards from year to year is a difficult process at any time and, particularly in the first year of such a major change, the complexity of the task is too great to rely solely on the judgement of subject specialists. There is new content, new assessment approaches and new procedures for aggregating across units. It would be unfair to expect awarders to be entirely consistent in such circumstances.

Our role as regulator is to ensure that standards are maintained from year to year, across specifications in the same subject and across the awarding organisations. To carry forward standards in such circumstances we agreed that the awarding organisations would use an agreed range of technical and statistical evidence to check that the grades awarded under the new system were broadly in line with judgements made about candidates in previous years. These included the relationship between attainment at GCSE and AS level and the performance of the previous year’s candidates. I wrote an open letter to all secondary schools and colleges in March indicating the approach we were taking².

Of course the progress of an individual learner is unpredictable, but for the candidature as a whole these relationships are remarkably stable. In other words, for many years once we knew the results a group of candidates achieved in their GCSEs we could predict with reasonable accuracy what grades the group would get a year later at AS level. This information can then be used by awarders to inform their judgements and by the awarding organisations and Ofqual to check that their draft grades are within the range we might expect. If awarders’ proposals result in more or fewer candidates in a particular grade than the statistical and technical information suggests, further investigations can be requested. The awarding organisations can look again at the judgements of the awarders to satisfy themselves that the grades are as secure as possible.

We used this approach for the first unit awards in February 2009 when awarding organisations were asked to report to Ofqual any unit awards that were outside the proposed tolerances. Broadly speaking the outcomes were acceptable. In the summer the awarding organisations were asked to report outcomes for full qualifications and again highlight any that were outside tolerances. This was carried out on a weekly basis through the summer. There were very few awards outside the tolerances where the regulator needed to ask for a review of the awarders’ recommendations.

On 4 November Ofqual published the regulatory criteria for the full introduction of the functional skills qualifications in English, mathematics and ICT for which teaching begins in September 2010. Over the past two years QCDA has been running a number of pilot projects, encouraging the development and assessment of functional skills qualifications by a number of awarding organisations. Ofqual has carried out a review of the standards and comparability of the assessments. The tests will be used as part of the Diploma and learners will be required to have passed in all three subjects before they can receive their Diploma award. Functional skills tests are not only part of the Diploma but are free-standing qualifications in their own right. It is intended that they should replace other assessments in key and basic skills and be relevant to all learners, whether they are young people in schools, apprentices in industry or mature military personnel on tour in other countries.

The functional skills qualifications do not fit either of the traditional categories of general or vocational qualifications. For example:

• The award is made on a pass/fail basis similar to many vocational qualifications but unlike general qualifications such as GCSEs, AS or A levels, which are awarded across a range of grades.
• The focus on the effective application of the functional skills to achieve a purpose is distinct from the predominantly knowledge-based learning of general qualifications.
• While the functional skills assessments are applied – task-based and set into real life situations – they are not intended to prepare candidates for work in a specific job or skill sector as one would expect in vocational qualifications.

In a pass/fail assessment the decision to be made can be very simply stated:

• ‘Pass’ indicates that the candidate has demonstrated the skills required.
• ‘Fail’ indicates that they have not.

But getting to that decision is far from simple.

To be valid the assessment must allow learners to demonstrate that they can apply their skills in situations such as they might expect to find in life or work. In English, for example, a level 1 pass requires that candidates must be able to write a range of documents, and it is surely obvious that a range cannot be demonstrated by a single piece. At level 2, where a wide range of documents is required, at least three examples of different types of documents must be required if learners are to be given the opportunity to demonstrate their abilities.

In GCSE mathematics it is common to allow some credit for correct working out even if the answer is wrong because of a mistake early in the calculation. In the functional skills assessments, however, learners are required to demonstrate their interpretation skills by considering the appropriateness and accuracy of their results and conclusions. They must therefore be given the opportunity to demonstrate this in practice. An answer that is far too large or too small to be reasonable in the context should be given no credit, even if the learner used the correct method to work it out.

In the ICT functional skills assessments candidates must demonstrate that they can insert, remove, label and store media safely. This clearly must involve a learner’s physical handling of CDs, DVDs or memory sticks, not merely saving to the hard drive of the computer. This may require those preparing candidates to modify their IT practices to provide more hands-on experience.

As I have already noted, it is a requirement of the Diploma that the learner has demonstrated competence in all of the functional skills. So failure in just one of the functional skills assessments can have a dramatic consequence: no matter how well the learner has done in the principal learning, the project, the additional and specialist learning and personal and learning skills, the Diploma cannot be awarded.

Our monitoring of the pilot over the past two years has indicated that the assessment schemes have some way to go before they can be regarded as fully satisfactory. In the light of the findings of our studies, pilot awarding organisations have been asked to take appropriate measures to ensure that the assessments meet the requirements laid out in the standards³.

When it comes to the new qualifications to be tested post September 2010 we will only accredit those in which the functional skills qualifications meet our criteria and are therefore of high quality and fit for purpose.

On the broader front we are half way through our two-year investigation into the reliability of qualifications and assessments. So far we have commissioned a number of projects, three of which were considered by experts at a seminar held at the University of Warwick in October. The three projects explored different approaches to reliability and one is now on our website⁴. Others will be released in the coming months. Further work has been commissioned and the projects will report towards the end of next year.

One suggestion arising from the research was that we should explain what we are doing in non-technical terms for a general audience. As part of this approach it was suggested that we should move away from the use of the word ‘error’ in our discussions of reliability as the word in its ordinary usage does not capture the nuances of the way it is used in assessment.

Unfortunately the discussion was interpreted in some quarters as an attempt at ‘spin’ and brought accusations that Ofqual was shying away from any discussion of mistakes – particularly mistakes by the awarding organisations or ourselves. This could not be further from the truth. We have stated that ‘where mistakes are found, Ofqual will report them’⁵.

However, there has been some confusion over what we are trying to do and I should like to make our position clear.

The word ‘error’ has at least two distinct definitions. The one we use in everyday language:

something incorrectly done through ignorance or inadvertence; a mistake

and one from mathematics:

the difference between an approximate result and the true determination

(Both of these are taken from The Shorter Oxford English Dictionary.)

This latter definition is not talking about a mistake. It is recognising that, as in many aspects of mathematics, absolute precision is impossible (the value of π, for example⁶ The same is true of any form of measurement: however accurate your scales may be, there is always a small amount of imprecision – ‘error’ – in the reading.

The fact that some commentators immediately assumed that we were talking about mistakes serves to highlight why we are looking for a different way of talking about this aspect of error.

Awarding organisations and assessment agencies must do their utmost to eliminate mistakes. It is Ofqual’s job to see that they take all reasonable steps to do so. Mistakes must be identified as early as possible in the process and corrected at once. The awarding organisations’ procedures are designed to guard against anything ‘incorrectly done through ignorance or inadvertence’.

But even when they – and we – have checked and rechecked the assessments to correct any mistakes, there will remain a degree of imprecision that cannot be removed. This imprecision, which is not a mistake but which can affect the reliability of a qualification, arises from factors over which neither the awarding organisation nor the regulator has any control such as the degree to which the assessment reflects two years of classroom teaching or the fluctuations in a candidate’s performance from day to day. Then there is some imprecision that we can try to reduce but cannot eliminate such as the differences between sets of questions the candidates choose to answer.

I hope this sets the record straight.