A correct item is not necessarily aligned: three questions for judging an assessment item

In my current research, I ask whether large language models can produce mathematics assessment items for cycle 3 that comply with the Plan d’études romand (PER), the curriculum of French-speaking Switzerland. Before even looking at what these tools produce, we need to agree on what “comply” means. I break it down into three questions, because an item can pass one and fail the other two.

1.Alignment: does the item target the stated objective?

The PER organises each subject into learning objectives, which translate its priority aims (visées prioritaires) for the cycle concerned, together with a progression of learning and essential expectations (attentes fondamentales) (CIIP, n.d.). In cycle 3, the progression is set out school year by school year. An item is aligned if it assesses what the target objective asks for, and not some neighbouring content.

The term comes from assessment research. Norman Webb (1997) defines alignment as the degree to which expectations and assessments agree, and breaks it down into several criteria. The first, categorical concurrence, requires the same categories of content to appear in both the expectations and the assessment. That is the sense in which I use the word here: a content match with a specific PER objective.

Why insist? Because an item can be mathematically correct and still off target. In a recent study of maths word problems generated by language models and rated, among others, by teachers against US standards (grades 3 to 5), most alignment errors came from problems that left out important parts of the target standard (Christ et al., 2026). The problem is then not wrong; it is incomplete with respect to what we wanted to assess.

2.Cognitive demand: what level of thinking does the item really require?

Two items aligned with the same objective can ask very different things of pupils: recalling a fact, applying a procedure, or reasoning and justifying. Webb (1999) distinguishes four depth-of-knowledge levels: recall, skill/concept, strategic thinking and extended thinking. In mathematics education, Stein, Grover and Henningsen (1996) describe demands ranging from memorisation to the use of procedures without or with connections to concepts, up to “doing mathematics”. Their framework was designed for classroom tasks; applying it to assessment items requires an adaptation, and that is a methodological point to make explicit.

Webb actually treats this question within alignment: an assessment is aligned if what it elicits from students is as cognitively demanding as what they are expected to know and do (Webb, 1997). I deliberately keep it separate, because the two can diverge: an item can target the right objective while asking only for recall. McGalliard and Otten (2025) point to exactly this risk: in their view, generative AI so far tends to produce tasks of low cognitive demand.

3.Quality: is the item correct, clear and usable?

An aligned and demanding item can still be badly built. Haladyna, Downing and Rodriguez (2002) brought together 31 guidelines for writing multiple-choice items. Several apply to any item: very clear directions, a single right answer, and no clue that gives the answer away (a word such as “always” or “never”, a grammatical inconsistency, an option that stands out). For a maths item, I add correctness itself: consistent data and a checked solution.

Here, recent models sometimes do better than people expect. In a study in higher education, in programming, GPT-4 produced multiple-choice questions with clear language, a single correct choice and good distractors, which appeared well aligned with the course objectives (Doughty et al., 2024). That result holds for that context; it says nothing about the PER or cycle 3. All the more reason to ask the three questions separately, against our own objectives.

Why three questions rather than one

Asking “is this a good item?” mixes three judgements. An item can be correct but off target, aligned but too easy, demanding but ambiguous. Keeping them apart tells us what to fix: change the target objective, raise the demand or rewrite the question. It is also what makes a serious study possible: you cannot say whether a tool produces “good” items without saying good by what standard.

The protocol and materials of my study will be posted on the OSF project. This article only sets out the framework, with no results.

Let’s talk

Do you design assessments for cycle 3, or work on alignment with the PER? I would be glad to discuss it. My contact details: pablocorreaprieto.ch

Text written with the help of AI, reviewed and approved by the author.