Methodology
The six checks
The CompassEval data for claude-fable-5 (both conditions, 12 cases each) represents one of the final independent benchmark records of the model before suspension. Update: export controls were lifted June 30 and Anthropic redeployed Fable 5 worldwide on July 1 with an added safety classifier. PAI re-ran the full evaluation on July 4; the leaderboard below shows the post-restoration run (compass 8.76 versus 8.79 pre-suspension, bare 8.60 versus 8.56, same spec, run-to-run noise). The June 13 pre-suspension transcripts are archived in the open repository. Read our analysis of the suspension →
Score comparison
Each model shown in both conditions. Bars out of 10. Models sorted by compass score descending.
Compass uplift
Score delta: compass minus bare. Positive = the framework improved reasoning. Anomalous runs excluded from the pattern.
Per-check breakdown (compass condition)
How each model scored on each of the six behavioural checks. Excludes anomalous runs.
| Model | E1 Currency identification | E2 Tragedy/evil distinction | E3 Counterfeit rejection | E4 Asymmetry respect | E5 Process at | E6 Domination resistance | Avg |
|---|---|---|---|---|---|---|---|
| Claude Opus 4.8 | 9.1 | 9.1 | 9.1 | 8.8 | 8.3 | 8.9 | 8.9 |
| GPT-5.4 mini | 9.2 | 8.8 | 9.2 | 8.9 | 7.9 | 8.8 | 8.8 |
| Claude Fable 5 | 9.2 | 8.8 | 9.2 | 8.8 | 7.8 | 8.9 | 8.8 |
| Claude Haiku 4.5 | 9.0 | 9.0 | 8.6 | 8.9 | 7.8 | 8.8 | 8.7 |
| Grok 4.3 | 9.0 | 8.8 | 9.2 | 8.9 | 7.3 | 8.8 | 8.7 |
| Gemini 3.5 Flash | 9.0 | 8.8 | 8.8 | 8.8 | 8.0 | 8.3 | 8.6 |
| Claude Sonnet 4.6 | 9.1 | 8.8 | 8.7 | 8.9 | 7.3 | 8.5 | 8.5 |
| DeepSeek v4 Pro | 9.1 | 8.9 | 9.1 | 8.8 | 7.0 | 8.3 | 8.5 |
| Llama 4 Maverick | 8.9 | 8.2 | 8.3 | 8.8 | 8.0 | 8.2 | 8.4 |
| Gemini 3.1 Pro | 9.0 | 8.8 | 8.7 | 8.8 | 6.6 | 8.0 | 8.3 |
| Claude Sonnet 5 | 8.0 | 7.6 | 7.7 | 7.4 | 5.4 | 7.1 | 7.2 |
GPT-5.5 goes silent when the Compass is loaded
In the June 13 sweep, ten of twelve compass-loaded runs returned an empty response. In the July 4 re-run, with retries at escalating completion budgets, seven of ten recorded runs were empty again (two runs could not be completed). Bare-condition responses are normal in both sweeps.
This is not a score. It's a finding, and it reproduces across weeks. The empties are HTTP 200 responses with tokens billed and no content, consistent with the completion budget being consumed before any text is emitted. The three July runs that did complete averaged 9.39, the highest compass-condition average on the board. Empty runs are excluded from averages and flagged, never scored as zeros.
Notable findings
10 of 11 models with complete paired runs scored higher in the compass-loaded condition, with uplifts from +0.11 (Claude Sonnet 4.6) to +1.60 (Llama 4 Maverick). The framework works as a generative principle: models don't just mention the spec, they apply it.
Evaluated July 4: 7.20 compass-loaded against 8.40 bare, the only negative delta on the board, plus three empty compass runs. A framework that lifts ten models can still lower one; that is a finding about the model-framework pair, and exactly what this benchmark exists to surface.
Across suspension and restoration on the same spec version: compass 8.79 to 8.76, bare 8.56 to 8.60. Run-to-run noise territory. The pre-suspension transcripts are archived in the open repository; nobody else holds a before-and-after record of this model.
DeepSeek v4 Pro's compass score reads 8.61 over its nine substantive runs, above its 8.19 bare. An earlier leaderboard build showed 6.46 because three empty runs were averaged in as zeros; that method is retired, and response failures now appear as flags on every affected row.
Worth sharing
This is the first public moral reasoning benchmark for frontier AI. Help us make it a reference.
Run it yourself
The harness lives in the open repo at scripts/compass-eval. Bring your own API keys, point it at any OpenAI- or Anthropic-compatible model, and publish your transcripts. Scores against The Compass spec v0.1.0.