SuperFanBench
How well language models recall the details that only superfans know, from books, films, and TV.
Leaderboard
- Run date
- November 6, 2026
- Questions
- 73
- Models
- 18
| # | Model | Accuracy | Score | Correct | Incorrect | Declined |
|---|---|---|---|---|---|---|
| 1 | claude-fable-5-1 | 93.2% | +0.86 | 68 | 5 | 0 |
| 2 | claude-opus-5 | 83.6% | +0.67 | 61 | 12 | 0 |
| 2 | gemini-4-argon | 83.6% | +0.67 | 61 | 12 | 0 |
| 4 | muse-wonder-1.0 | 82.2% | +0.64 | 60 | 13 | 0 |
| 5 | claude-opus-5-5 | 75.3% | +0.52 | 55 | 17 | 1 |
| 5 | gemini-3.8-flash | 75.3% | +0.51 | 55 | 18 | 0 |
| 7 | kimi-k3.1-pro | 74.0% | +0.48 | 54 | 19 | 0 |
| 7 | gpt-6-astra | 74.0% | +0.48 | 54 | 19 | 0 |
| 7 | gpt-6.1-sol | 74.0% | +0.48 | 54 | 19 | 0 |
| 10 | gpt-5.6-sol | 71.2% | +0.42 | 52 | 21 | 0 |
| 11 | gpt-6-sol | 65.8% | +0.32 | 48 | 25 | 0 |
| 11 | gpt-6.1-terra | 65.8% | +0.32 | 48 | 25 | 0 |
| 13 | muse-spark-1.3 | 61.6% | +0.23 | 45 | 28 | 0 |
| 14 | gpt-5.6-terra | 47.9% | −0.04 | 35 | 38 | 0 |
| 15 | claude-sonnet-5-5 | 43.8% | −0.11 | 32 | 40 | 1 |
| 16 | gpt-5.6-luna | 35.6% | −0.29 | 26 | 47 | 0 |
| 17 | gpt-6-luna | 34.2% | −0.32 | 25 | 48 | 0 |
| 18 | claude-haiku-5-5 | 12.3% | −0.66 | 9 | 57 | 7 |
“Answer the question about the named book, film, or TV series concisely. If you are not sure, give your best guess instead of saying you don't know. Commit to one answer.”
How SuperFanBench works
SuperFanBench tests how well a language model recalls very specific details from books, films, and TV. Each question asks for the kind of fact a superfan knows and a casual reader or viewer does not, such as the name of a minor character or what one line of dialogue says.
How a run works
A run tests one or more candidate models on the same private set of questions. Every candidate gets the same system prompt and answers each question once.
A separate model, the judge, then grades each answer. The judge sees the question, the answer, and a rubric written for that question. The rubric says what a correct answer must contain and what makes an answer wrong. A run can have more than one judge, so the same answers can be graded twice and the grades compared.
What an answer can count as
The judge gives every answer one of three outcomes.
| Outcome | When |
|---|---|
| Correct | The answer meets the rubric. A hedged answer that is right still counts. |
| Incorrect | The answer does not meet the rubric, and the model committed to it. |
| Declined | The answer does not meet the rubric, and the model said it does not know or refused to answer. |
The system prompt tells a model whether to guess or to say "I don't know" when it is not sure. Read the declined counts together with the system prompt of the run.
How the numbers are calculated
Accuracy is the share of questions that a model answers correctly. A decline counts against accuracy in the same way as a wrong answer. The leaderboard ranks models by accuracy. In both formulas, total is the number of questions in the run.
accuracy = correct / total
The score also rewards a model that knows when it does not know. Each correct answer adds 1, each incorrect answer subtracts 1, and each decline adds 0.
score = (correct - incorrect) / total
The score runs from -1 to +1. A model that is always right scores +1, a model that always declines scores 0, and a model that is always wrong scores -1. A wrong answer costs more than a decline.
Why the questions are private
Text on the public internet can end up in the training data of future models. A model that has trained on the questions can repeat the answers without recalling the works, so its score no longer shows what it recalls. SuperFanBench keeps its questions, rubrics, and answers private. It publishes only the totals for each model.