A way of viewing the driving test is as an exercise in data entry. Every test is recorded with volumes of information. Have a close look at the DL 25 just to see the range of information that it contains.
This data should have 2 characteristics, reliability and validity. The first is concerned with the consistency of the data and the second with its relevance. We are going to look at both of these in some detail,
Reliability:
- Test-retest. This looks at data over time. There will be two ways of looking at this. Repeating the test with the same examiner at the same time of day. Or doing it at different times of day.
Will you keep getting the same results. Clearly at different times of the day you will have a different number of interactions. Rush hour may have 200 interactions with other road users compared with 100 mid morning.
One of the things that make the driving test unique is every driving situation is going to be different in some way. Even if the route is the same, with the same cars, drivers and examiners. Because we are all different from day to day, our decisions and those of other drivers will be different day to day.
To just keep repeating the same route with the same examiner is clearly a non-starter and highly unlikely to happen. But it does make the point that pupils should be able to deal with general situations rather than specific places.
A pupil that passes the first time only has one result but what about our serial failures. What is this telling us. Should we be doing some soul searching. I know my serial failures come from when they have put in for the test without asking me. They then give some reason why they can not put it back.
- Interrater. This looks at different examiners and do they give the same result. They obviously do for our serial failures. We have all heard the comment about a pupils’ chance of passing depending on which examiner they get.
I like to think that the more an examiner is like me, the better the chance the pupil will relax and drive to their potential. My passes tend to come from very good looking, highly intelligent and charming examiners. This is proved by the fact they have passed my pupil. This is also an illustration of bias in the way I see things.
We know the examiners themselves besides all being very good looking, highly intelligent and charming are subject to their own quality control. At some point we have all had the examiners examiner in the back. I would be very surprised if the statistics of how they do their job is not looked at.
- Internal consistency. Will different parts of the test produce the same result. A driving test does not have the time to test all parts of the syllabus of learning to drive. So what will happen is the various test routes will test as much as geographically possible for that centre.
This will give rise to a variety of routes which if nothing else stop local people from being bothered by having the same area used all the time. Also this stops us from learning then teaching a route parrot fashion.
Would your pupil fail if they got a certain manoeuvre. Would a dual carriageway or a narrow road cause your pupil problems. Have they covered enough of the syllabus. Or to put it simply would your pupil pass or fail with a different route and what does that mean for their safety when passed.
Validity:
- Construct: Does this conform to our knowledge and theory of what is being tested (driving safely).
The Highway Code and Driving The Essential Skills will form the core of our knowledge of that. While an individual route may not cover everything. The Theory Test and it’s high pass mark will give it good coverage of The Syllabus.
On the theory side we see constant changes, Probably the biggest change being the introduction of The Theory Test. One of the reasons given for the introduction of this side of the test was to bring us into line with Europe. Personally I would also like to think it is about making things more comprehensive. Training pupils to look for hazards using the Hazard Perception Test can only be a good thing.
- Content. Does this cover all aspects that it should.
Is all of The Syllabus from The DVSA covered. Clearly it can not for a huge range of reasons. What can not be tested on a practical driving test can be covered on The Theory Test. But knowing what to do in bad weather or the dark is not the same as showing what to do.
Because of geographical and time restraints the range of the syllabus can not be covered. But the range of test routes an individual centre has along with the spread of them between examiners should go some of the way to meeting this requirement.
The test candidate will only be asked to do one manoeuvre but it is wise to know all four. We have no way of knowing which one of the four they will be doing. The same will go for all parts of the syllabus.
- Criterion. Are there other valid measures of a person’s ability to drive.
What other ways could you test a person’s ability to drive. It’s a very practical skill for something that could be up to 3.5 tonne MAM and driven at up to 70 mph on a public road. Surely we should only let someone be able to do that if they can demonstrate the skill to do so safely. Hence licencing and testing. Which is then followed by enforcement.
What about the new driver accident rate. This certainly says the test is deficient. But what can we do about it. Should we make the driving test harder, longer, more difficult. Is this socially, politically or even morally acceptable. Would the net effect of all this be turning our pupils into cash machines.
You could argue that car insurance is already very expensive for the young new driver. So they are getting financial penalties for being young and prepared to take risk. But being young is about pushing boundaries and taking risks. Passing a test means we are given some freedom but like all things it comes with some responsibility.
If we did extend the test would this mean that only an affluent section of the population would be able to afford driving lessons. As instructors we should certainly do our best to cover all of the syllabus.
What about all the data that is accumulated. The DVSA publishes all sorts of statistics. What can we use?
Well you can email OMI@dvsa.gov.uk and ask for a fault analysis report of all your tests for the year. You must have had your badge in the car for the examiner to record it. This will cover the financial year 1st April to 31st March. The figures will generally become available in June. So 2019-2020 was available from last June.
Having got your breakdown you can now identify where your pupils are picking up the most faults and start to do something about it. As far as I am aware there is no reason why you could not do this every year.
You can ask for a number of years and just pick the best year. That’s what I have done and a very healthy 76.92% pass rate. I do have worse years and there is no way I would be telling what my worst year was.
This is the problem with interpreting data. The person presenting it, who happens to be me has manipulated it by only selecting my best year. This is the reason for assessing the reliability and validity of data.
The question might be what can we take away from looking at a driving test this way:
- Cover the syllabus as far as you are able. Each individual test route can not cover everything
- But collectively they can cover a large part of it. Or put another way, do not just do test routes
- Make sure your pupil can deal with situations rather than places
- Get a fault analysis of how your pupils perform on their driving tests each year.
© Liam Greaney
