Marking with AI: compare it with our real errors, not with perfection

In an earlier article, I wrote that AI is usually compared with an error-free ideal, when the honest comparison is with how we actually mark. That claim needs evidence. To know what a marking aid is worth, we first need to know what human marking is worth. On that question, the research is old, plentiful and not very flattering to us.

1.Same paper, very different marks

In 1912, Daniel Starch and Edward Elliott sent the same two English exam papers to two hundred American high schools; 142 teachers marked them. When asked, teachers usually said two markers would differ by 10 points at most. In fact, the marks given to the same paper could spread over 35 to 40 points, and for one of the two papers, 22 of the 142 teachers did not give a pass mark (Starch & Elliott, 1912). The following year, the two researchers repeated the experiment in geometry, and the marks were even more scattered than in English. Their conclusion speaks directly to maths teachers: there is no reason to think a mathematics paper can be marked more precisely than any other (Starch & Elliott, 1913).

Almost a century later, Hunter Brimi (2011) put the question to 90 secondary teachers marking the same paper. Among the 73 who marked out of 100, scores ranged from 50 to 96. In letter grades, the same paper received 10 As, 18 Bs, 30 Cs, 9 Ds and 6 Fs.

2.The order of papers and the length of the session matter

I had mentioned fatigue at the bottom of the pile. Klein and El (2003) gave the same 31 compositions by seventh-grade pupils to 60 teachers, in three different orders. As the teachers worked through the stack, grades rose gradually, eventually exceeding those given by specialist markers; between the first and the last papers, the gap was about 10%. The authors link this drift to long, uninterrupted marking sessions.

Order also works another way: trainee teachers marked two average essays lower when they followed five good essays than when they followed five poor ones (Hales & Tokar, 1975, as cited in Meadows & Billington, 2005). The same review, however, reports a study of exam marking that found no important effect of the point at which a script was marked. So the effect is not always the same; but our mark depends partly on what we have just read.

Add what every teacher knows: a criterion applied more generously to the thirtieth paper than to the third, a typo in a question that we only spot once the papers have been handed back. None of this is professional misconduct. It is simply how human markers work.

3.What the first studies on AI show

Against this baseline, large language models are neither perfect nor useless. On 1,710 short answers from pupils aged 5 to 16 in science and history, to be judged right or wrong, GPT-4 reached an agreement of 0.70 (Cohen’s kappa) with teachers, against 0.75 between the teachers themselves (Henkel et al., 2024). On 20 essays by Year 7 and 8 pupils, rated by 37 teachers on ten criteria, the best model tested reached a correlation of 0.74 with the teachers’ scores; but it marked more generously and was weaker at judging the quality of the content (Seßler et al., 2025). For written feedback on pupils’ writing, trained human raters did better than ChatGPT 3.5 in every category but one, criteria-based feedback (Steiss et al., 2024).

My reading of these results: AI is not yet a reliable marker on its own, especially when it comes to judging the substance of a text. But the gap with a human marker is much smaller than it looks when AI is compared with perfection.

4.A second reader, not an automatic marker

That is why I see it as a second pair of eyes. The teacher marks; the tool suggests a mark or a comment in parallel; only the papers where the two disagree are looked at again. Ofqual, the regulator of qualifications and exams in England, describes exactly this set-up (AI marks in parallel with the human marker, and discrepancies flag a response for review) and notes that it avoids anchoring. It also states that using AI as the sole means of determining a student’s mark does not comply with its regulations (Williamson, 2026).

What I expect from it is that the tool catches part of what fatigue or the order of the papers makes us miss. This is a hypothesis: as far as I know, no study has yet measured, in real classrooms, how many marking errors such a second reader prevents.

Two conditions are non-negotiable. The first is data protection. A guidance sheet on Eduvaud, the Vaud education portal, written for upper-secondary teachers, puts it in one line: never send personal or sensitive data through online AI tools; when in doubt, use the tools provided by the canton (IA et protection des données, 2024). In practice: anonymised papers, with no name or detail that could identify the pupil. The second: the decision stays with the teacher. The tool flags; it does not decide.

The right question

So the useful question is not “does AI mark without errors?”. No marker does. It is rather: does a teacher who marks with a second reader make fewer mistakes than a teacher marking alone? That is an empirical question, and it deserves to be tested in our own classrooms, with our own papers and criteria.

Let’s talk

Do you already mark with a second reader, human or not? Have you ever compared your marks with a colleague’s on the same papers? I would be glad to discuss it. My contact details: pablocorreaprieto.ch

Text written with the help of AI, reviewed and approved by the author.