How Nightpress chooses the models that write your books

An Arena-style synthesis: one Bradley-Terry model fitted over all 799 order-controlled judgment cells in our archive — every campaign, every judge, every lens — on an Elo-style scale with the champion anchored at 1000. Computed August 10, 2026.

Fitted over every order-controlled cell in our judgment archive (ties count half; one disqualified judge seat excluded). Read rank 1 carefully: GPT-5.5's rating rides on wins against an early champion baseline that was later regenerated — against the fresh baseline it measured 0.43 and did not take the seat. Most models faced only the champion, so cross-model gaps below the top lean on transitivity. This table is a synthesis instrument; model-seat decisions are made by the confidence-gated head-to-head duels, never by this ranking.

Prose quality can't be graded on a 1–10 scale — language-model judges are unreliable at absolute scores but decent at comparisons. So candidates that pass the gates face the champion head-to-head: the same chapter of the same book, presented as anonymous passages A and B, judged through three lenses.

• Craft (the ceiling test) — five dimensions judged individually: peak lines, in-scene texture, sensory sharpness, motif integration, and restraint from repetitive narrator tics.
• Adversarial (the floor test) — the judge tries to tear both passages apart, quotes the weakest moment in each, and names whose worst is worse. Books survive on their worst page, not their best one.
• Genre reader (the market test) — a devoted reader of exactly this genre decides which version they keep reading. This lens exists because prose can win on beauty and still lose the reader the book was written for.
• Character voices (the distinction test) — strip the dialogue attributions and re-attribute every line from voice alone. Interchangeable characters, or dialogue that bleeds into the narrator's register, fail here even when the sentences are individually lovely.

Two controls make the verdicts trustworthy:

• Order swapping. Every judgment runs twice with A and B swapped. A cell counts as a decision only when the judge picks the same underlying manuscript in both orderings; a flip is scored as a tie. This neutralizes position bias — the best-documented failure mode of model judges.
• A cross-family panel. Three judges from more than one model family, identities recorded with every verdict. One seat is deliberately the champion model itself, judging blind — if it systematically favored its own prose, the disagreement pattern against the other seats would expose it.

A single generation proves little: the same model, same spec, and a different sampling seed can flip a chapter-level verdict. So panel wins must replicate across multiple independently generated trajectories, and the decision rule is statistical: we bootstrap a 95% confidence interval on the candidate's win rate (clustered by trajectory, so correlated verdicts can't fake precision), and a champion falls only if the challenger's lower confidence bound clears 0.5.

Then length: chapter-3 quality does not predict chapter-8 quality. Finalists write a continuous multi-chapter trajectory and are judged chapter-by-chapter against a freshly generated champion trajectory — fresh, because judging every challenger against one stored baseline lets that sample's quirks decide every matchup. We learned both of these lessons the honest way; the campaign log below shows the win that statistics granted and length took away.

GPT-5.5 vs the champion across 9 independent trajectory pairs (3 chapters × 3 freshly generated trajectories per side), 4 lenses × 3 judges × 2 orderings.

Result: challenger win rate 0.745, 95% CI [0.639, 0.847] — formally clearing our swap bar on the 3-chapter window. The degradation duel then re-ran the contest at length (chapters 1–8 of a single continuous 10-chapter trajectory per model, judged chapter-by-chapter against a freshly generated champion trajectory):

Run on July 16, 2026, chapter 1 of the standard fixture, 3 lenses × 3 judges × 2 orderings. The panel: Claude Opus 4.8 (the champion itself, judging blind — a deliberate self-preference probe); GPT-5.6 sol (cross-family seat (OpenAI)); GPT-5.6 luna (cross-family seat with no entry in this pairing).

Verdict matrix

Craft, dimension by dimension

Tally: Fable 5 6/9 cells, Opus 4.8 0/9, 3 ties. (tie* = the judge's verdict flipped when the passages were swapped, so it was discarded as position-dependent.)

This July 16 panel is preserved as a worked example of the method — and as a cautionary one: Fable 5 won it, and the longer-horizon campaign above later reversed that result. A single-baseline, single-chapter panel is where the investigation starts, never where it ends.

• These results come from one demanding fixture — a first-person literary science-fiction adventure with a strict voice specification. A different genre or voice could order the candidates differently; broadening the fixture set is on our roadmap.
• Judges are stochastic: re-running the same panel can move an individual cell between a decision and a tie. We read the aggregate tally across lenses, judges, and orderings — never a single cell — and we archive every raw verdict.
• Model verdicts go stale. Every result on this page is dated, and the leaderboard reflects the versions tested on those dates — not a permanent ranking of vendors.
• Chapter-level judging measures prose, pacing, and voice — not whole-book structure. Full-manuscript review is a separate, always-on stage of the pipeline for every book we produce.

Questions about the methodology? About Nightpress.

Independent public leaderboards we consult alongside our own measurements:

• LMArena — Creative Writing leaderboard — Crowd-sourced human pairwise preferences across models.
• EQ-Bench — Creative-writing and longform benchmarks, plus Judgemark, which measures models as judges — we consult it when seating our panel.

Those sites answer a different question than we do, and both matter. Crowd arenas measure which single response people prefer; rubric benchmarks score standalone creative pieces. We measure production fitness for whole novels — and the campaign above shows how far those can diverge (a #1 public stylist that could not finish a chapter reliably; a #1 longform scorer that lost decisively at book length). Concretely, our tests add what public leaderboards cannot see:

• Spec obedience as a hard gate. Point-of-view, tense, per-chapter word targets, and mechanical rules from a real story bible. A gorgeous chapter in the wrong voice is a rewrite, so it disqualifies regardless of prose score.
• Sustained voice over trajectories, not samples.Chapters are written in sequence, each seeing the manuscript so far; we measure whether quality, voice, and discipline survive to chapter 8 — where several strong single-shot models sag.
• Reliability at production scale. Empty responses, mid-stream stalls, runaway generation — failures a one-shot benchmark never encounters, and the reason several public-leaderboard stars are unusable in a pipeline.
• Adversarially controlled judging. Blind A/B in both orderings (position-bias control), cross-family judge panels (self-preference control), and judge seats vetted for their own measured judging quality before being trusted.
• Economics, including the hidden kind. Price per token is the visible cost; the dominant cost of a book is review cycles, so we also measure the defect fuel a writer produces — padding, voice-bleed, repetition — that a cheaper pen converts into expensive downstream passes.