TurtleDock
Writing

How AI generates question papers, and where it goes wrong

AI question paper generators work in one of three ways: retrieving items from an existing question bank, filling templates with varied values, or generating questions with a language model. The first two are reliable within their limits; the third produces questions that look correct and sometimes are not, which is the failure mode teachers should understand.

We do not generate questions and hold no question bank, so this is an explanation rather than a product pitch. Our own tool produces a blueprint and leaves the questions to the teacher.

Written by Checked by Kalyan SundaramLast reviewed

Teachers evaluating a question paper generator are usually shown output rather than method, and the output looks fine. Understanding which of the three methods a tool actually uses tells you far more about whether you can trust it than reading a sample paper does.

Method 1
Retrieval from a question bank — reliable, limited by the bank
Method 2
Template with varied values — reliable for computation
Method 3
Language model generation — fluent, sometimes wrong
The tell
Ask where a specific question came from
01

The three methods

Most products use a combination, which is worth asking about explicitly.

MethodHow it worksWhere it fails
Retrieval from a bankItems are written by people, tagged by chapter, difficulty and type, then selected to match a blueprintOnly as good as the bank. Repeats across a year; students recognise the set
Template with varied valuesA question pattern with numbers substituted — a triangle with different side lengths each timeWorks for computation, not for reasoning. Substituted values can produce impossible cases
Language model generationA model writes new questions from a prompt describing the topic and formatFluent and sometimes wrong in ways that look right. This is the one to understand
02

Why generated questions fail in a specific way

A language model produces text that resembles a question about the topic. That is not the same as a question that is correct, answerable, and on the syllabus, and the gap between them is invisible at a glance.

  • Beyond the syllabus. The model knows the subject at large, not what your board prescribes for that class. A perfectly good chemistry question can require a concept taught two years later.
  • Answerable only with unstated assumptions. A physics question that omits whether friction is neglected has two defensible answers and no defensible mark scheme.
  • Impossible values. Numbers that make a triangle inequality fail or a concentration exceed saturation. Fine as prose, unsolvable as a problem.
  • A mark scheme that does not match the question. Frequently the generated answer solves a subtly different problem from the one printed.
  • Misjudged difficulty. A question labelled as recall that actually requires two-step reasoning, which wrecks a blueprint’s cognitive spread.

None of these is common enough to make the output obviously useless, and all are common enough to make every question require checking. That is precisely what makes the saving illusory: verifying forty generated items carefully takes longer than writing twenty good ones.

03

If you are going to use one anyway

Teachers do, and there are sensible ways to do it.

  1. Use it for a first draft, not a paper. Treat every item as a suggestion from a colleague who has not read the syllabus.
  2. Solve every question yourself. Fully, on paper. This catches impossible values and unstated assumptions, and nothing else does.
  3. Check each against the prescribed syllabus, not against your sense of the subject.
  4. Re-rate the difficulty yourself. The label the tool assigned is a guess.
  5. Never use generated items in a board-pattern or final examination without the same moderation any paper gets. The consequences of an unanswerable question scale with the stakes.
04

What we do instead

We hold no question bank and do not generate items. Our tool produces the blueprint — sections, question counts, choice, marks and the cognitive spread — and checks the arithmetic, which is the part that is genuinely mechanical and is where the errors found after printing actually come from.

The questions stay with the teacher, who can defend each one to a moderator. That is what makes it a paper rather than a worksheet.

FAQ

Questions people ask

Phrased the way they arrive, answered so each one stands on its own.

How do I tell which method a tool uses?

Ask where a specific question came from. A retrieval system can tell you which bank item it is and who wrote it. A generative one cannot, because the question did not exist until you asked.

Are AI question generators safe for practice worksheets?

Lower stakes, so lower risk, but a wrong question still teaches a student something wrong and they usually cannot tell. Solving them yourself first remains the minimum.

Does TurtleDock generate question papers?

No. We produce the blueprint and check the marks arithmetic. We are the record, not the syllabus, and that boundary is deliberate rather than a gap we intend to close.

Related

Read next

Provenance

Who checked this, and against what

Accountability

Checked by the Product desk on 7 September 2026. Corrections go in the page, dated, rather than quietly.

No call required

Tell us your enrolment, we will tell you the number.

Pricing is on the page, in the open, with the counting rule written next to it. Nobody has to book a call to find out what it costs.

See pricing

Bring your own structure. We configure it live on the call.

Forty minutes with an implementation engineer — your classes, your fee heads, your grading scheme, entered as we go. No slides, no sample school.