Model Scorecard

Generated: 2026-10-05T05:00:54.191Z | Window: 2026-09-05 → 2026-10-05

Cost, quality, and speed per LLM, joined from runstate trails (GitHub), runs.jsonl (token/cost telemetry), and events.jsonl (dispatch tier, stall/escalation counts). Regenerate with npm run scorecard -- --days 30. See #566.

n is a confidence column, not a metric — a row with n=1 is one data point, not a trend. Read cost/delivered and delivery rate alongside n, never alone.

Per model × repo × tier

Totals per model (all repos/tiers)

One row per model, with the gateways it was served through as ↳ sub-rows whenever there is more than one — the fold that makes the model row readable would otherwise hide a gateway costing more or delivering less than the same weights elsewhere.

Billable tokens count uncached input + cache-creation + cache-read + output — cache reads are billed and, on this fleet, are the dominant term (issue #540: 262 uncached vs 13.4M cache-read). That is the same total batch-pilot-2-execution.md §13 publishes.

Batch amortization

Batching exists to pay the CI gate once for N issues (#770), so the headline is issues shipped per gate run. A gate run is one CI gate on one PR: 1 + ci_fix_attempts for a merged PR (batch or full-cycle alike), 0 for a batch that never merged. Shipped members are the merged batch PR’s closing references — a batch blocked at batch-validate and recovered by hand never posts batch-ship, so its trail alone reads it as unshipped.

Per batch

• Tokens per shipped issue EXCLUDES prep; prep is disclosed in its own column (#796). batch-issues-preparation (the classifier agents that pick members) runs in the operator session, not a scheduler dispatch. sched enqueue records the calling session id (CLAUDE_CODE_SESSION_ID) per batch in <sched-dir>/batch-prep.jsonl, and the usage ledger attributes that session’s tokens (subagents included) from the previous enqueue of the same session (max 6h back) to this one. It is an upper bound: unrelated operator work in that window counts too. N/A = no prep session recorded on this host (batches formed before #796 cannot be backfilled; a batch prepared on another host reads N/A).
• A dissolved batch counts every requeued member as evicted — the dissolve requeues all unshipped members as full-cycle runs, so none of them shipped with the batch.
• A hand-recovered batch PR records no ci_fix_attempts and counts as one gate run.
• Wall-clock/shipped issue is amortized throughput for batches, latency for full-cycle. A batch row is Σ batch wall-clock ÷ Σ shipped issues (a 300-min 3-member batch reads 100 min); the full-cycle row is the mean per-issue span. Batch token/cost figures include evicted members’ dispatch spend, while full-cycle averages delivered issues only.
• Tokens by model are per-host. They come from ~/.dossier/sched/<slug>/runs/ (ai-dossier sched stats --batch <id> reads the same logs); a batch run elsewhere reads N/A (not on this host), and a subscription-plan model is tokens without cost.
• Full-cycle Wall-clock/shipped issue is the milestone span; batch is batch-setup → PR merge. Gate wall-clock is PR open → merge, which includes CI and review waits.

Wall-clock per phase (all models)

Median seconds between a phase’s milestone and the previous one, from the trails’ own at= stamps. Wall-clock is the only per-phase measure available: cost AND API-minutes both come from one agent session that usually spans several phases, so neither can be attributed to a phase (see Limitations).

Reconciliation

This is a regenerated snapshot, not the first one — see git history for docs/reports/model-scorecard.md for prior windows. The first-snapshot reconciliation against batch-pilot-2-execution.md §13.3 and model-agnostic-fleet.md ran once, at #566.

Limitations

• A moving version tag folds onto its pin only where someone declared the mapping. glm-latest → glm-5.3 is declared (MODEL_ALIASES in cli/src/runstate-stats.ts, the mapping #566 states) and folds, along with every routed spelling of it. Which pin a -latest tag points at is a fact about the provider’s state, not about the string, so it cannot be derived here and it goes stale the moment the provider ships a new version under the same tag — when that happens, update the value in MODEL_ALIASES rather than reading the row as one version. Undeclared tags (kimi-latest, which has both kimi-k3 and kimi-k3-fast as plausible pins) keep their own row and are named in Data warnings — a guessed alias misattributes cost and quality silently, a missing one only splits a row.
• A context-window variant keeps its own row. claude-opus-5[1m] does not fold into claude-opus-5: the milestone protocol says the suffix should never have been written (gate records the bare model id), but 1M-context is billed differently, so folding it would blend two cost profiles to fix a formatting slip. Read the two rows together when judging quality, separately when judging cost.
• Cost comes from two sources, and the column says which. ~/.dossier/runs.jsonl is authoritative; where a dispatch predates the telemetry fix (#564) and left it null, the figure is recovered from that dispatch’s own agent log under ~/.dossier/sched/<slug>/runs/. Both are on-host only — a dispatch run from another machine has neither, and its row reads N/A because the data is elsewhere, not because it was free.
• Δ vs prev is snapshot-over-snapshot, not a strict 7-day delta. It compares this run against whatever sidecar is on the base branch — two overlapping 30-day windows when the weekly cron produced both, and a longer gap whenever a weekly PR went unmerged. A baseline older than the window raises its own data warning naming the age, so the column is never silently stale; read it as “since the last published snapshot”.
• The window selects ISSUES by last update, not runs by date. The trail read is gh issue list --search "updated:>=<start>", and a matched issue contributes its whole dispatch history — so an old run on an issue merely touched inside the window counts in full. Read the window as “every run on an issue active in the last N days”, not “every run started in the last N days”.
• A <unknown> tier is usually an artifact of that mismatch, not an untiered dispatch. Tier and agent CLI come from events.jsonl events inside the window, while the trail data behind the same row is not time-filtered — so a run whose dispatch event predates the window keeps its outcome columns and loses its tier. The per-model totals fold tiers away and are unaffected; only the per-tier rows split.
• Cost and API-minutes per phase are not separable. A dispatch is usually one continuous agent session covering several phases, so both are recorded per issue, not per phase — the per-phase section above reports wall-clock only, which the milestone at= stamps do carry. For a per-phase breakdown of a single run rather than a median across many, run ai-dossier runstate stats directly.
• Stall/escalation/unverified-exit counts are a per-host gap. They come from ~/.dossier/sched/<project>/events.jsonl, which only exists on the machine that ran the dispatch. A run dispatched from another host reports 0 for these columns here even if it really stalled — fleet-cli-audit.sh documents which hosts exist; this script does not collect across them.
• Cost/delivered averages only delivered issues. Work that was dispatched and then blocked, evicted, or abandoned is not in the denominator or the numerator — see the n vs Delivered columns for how much of a bucket that excludes.

Data warnings

• imboard-ai/ai-dossier: 31 run(s) recorded no model= — their row is bucketed as , and its outcome columns are not attributable to any model
• imboard-ai/ai-dossier: 20 of 150 run(s) are classifier dispatches (classify milestones only) — excluded from the by-model and by-class tables, since they record no model= and never ship
• imboard-ai/ai-dossier: 112 of 130 run(s) (86%) carry no classifier risk= verdict — they are bucketed as , so the class rows describe only the classified remainder
• imboard-ai/imboard-monorepo: 54 run(s) recorded no model= — their row is bucketed as , and its outcome columns are not attributable to any model
• imboard-ai/imboard-monorepo: 80 of 336 run(s) are classifier dispatches (classify milestones only) — excluded from the by-model and by-class tables, since they record no model= and never ship
• imboard-ai/imboard-monorepo: 198 of 256 run(s) (77%) carry no classifier risk= verdict — they are bucketed as , so the class rows describe only the classified remainder

Rendered from docs/reports/model-scorecard.md in the repository. Edit it there.