Production AI Institute — vendor-neutral certification for AI practitioners
Verify a credentialFor organisationsContact
AI Incident Registry
CriticalFinancial Services·1999·Fujitsu / Post Office Ltd

UK Post Office Horizon — AI as Infallible Evidence

The UK Post Office prosecuted over 700 subpostmasters for theft and fraud based on accounting discrepancies produced by the Horizon IT system — a system known internally to have software bugs. Subpostmasters were imprisoned and bankrupted. The Post Office suppressed evidence of Horizon's defects for over two decades. Widely considered the greatest miscarriage of justice in British legal history.

D6 · Human OversightD8 · Vendor Resilience

What happened

Fujitsu's Horizon accounting system was introduced to UK Post Offices from 1999. The system had significant software defects that produced phantom shortfalls in branch accounts. Post Office management treated Horizon's output as infallible — and prosecuted over 700 subpostmasters for theft, false accounting, or fraud based on the shortfalls the system generated. Many subpostmasters were imprisoned. Others lost their businesses, homes, and suffered severe mental health impacts. Several died before their convictions were quashed. The Post Office and Fujitsu had internal evidence of the bugs but suppressed it during prosecutions.

PSF Analysis

How the Production Safety Framework maps to this failure

A D6 and D8 failure of historical magnitude. The fundamental PSF violation was the absence of any principle that IT system outputs should be treated as fallible. Horizon was used as the prosecution's primary evidence with no independent verification — a catastrophic D6 failure. The D8 failure was the vendor relationship: Fujitsu held evidence of defects and the Post Office had no contractual mechanism to compel disclosure. This case predates the modern AI era but is directly applicable: the 'computer says so' reasoning that enabled it is identical to the reasoning that produces unsafe AI deployments today.

Controls that would have prevented this

Specific PSF controls mapped to each failure point

1
D6 · Human Oversight
Require independent verification of IT system outputs before using them as the sole evidence basis for criminal prosecution.
2
D8 · Vendor Resilience
Establish contractual obligations for Fujitsu to disclose known software defects, particularly those affecting financial calculations.
3
D4 · Observability
Implement anomaly detection that flags when multiple branches show similar patterns of unexplained shortfalls — a systemic software issue would produce correlated anomalies.

Outcome

Public inquiry launched 2021. Over 900 convictions under review. The Post Office (Horizon Offences) Act 2024 quashed convictions. Compensation scheme established. Fujitsu's CEO apologised in 2024 and said the company bore moral responsibility.

accountabilityevidencesoftware-defectsvendor-trustcriminal-justice

Related incidents

High2018
Amazon Recruiting AI Discriminated Against Women
D3D6
Critical2018
Uber Self-Driving Car Kills Pedestrian in Arizona
D6D5
Critical2021
Zillow iBuying Algorithm Collapse
D4D8
NEXT STEP

Prove you understand how to prevent failures like this

The AIDA exam tests PSF knowledge across all 8 domains. Free to take, immediately verifiable.

Take the AIDA exam →← All incidents