Production AI Institute · Public recordRecord
AI & automation incident registry

When consequential systems fail, and why

Real AI failures, plus selected historical system failures with direct relevance to production AI. Each entry shows who was involved, what happened, what controls were missing, and what a team should check before the same pattern repeats.

16 documented incidents8 control areas coveredCross-sector failures and harms

Found your failure pattern?

Read the registry as a control checklist.

The moment a case feels familiar is the moment to check the same controls in your own deployment.

We had a scare

Turn the incident into a response plan.

Use the incident kit when a chatbot, agent, workflow, or vendor tool has already created risk and the team needs order fast.

Use the response kit →

We saw our pattern

Check the same controls in your own stack.

Run the readiness check to see which of the controls that failed here are missing in your own deployment.

Run readiness check →

We need the evidence

Follow the incident through the public record.

Browse the incident registry, then inspect linked entities, sources, and controls in the public record explorer.

Open record explorer →

We need governance

Map the failures to the standard.

The PSF names the eight control domains these cases keep failing. Read it before the next deployment decision.

Read the PSF →

Showing 16 of 16 incidents

High
Air Canada Chatbot Bereavement Fare
AviationAir Canada2024
Air Canada's AI chatbot incorrectly told a customer he could apply for a bereavement discount retroactively. When the customer attempted to claim the refund, Air Canada refused — arguing the chatbot was a 'separate legal entity' not bound by its advice. A Canadian tribunal rejected this defence and ordered Air Canada to honour the discount.
→
High
Law Firm AI Hallucinated Fake Case Law in Court Filings
LegalMorgan & Morgan / multiple firms2025
Reuters reported that major law firms were warning lawyers that AI can invent fake case law and that filing hallucinated citations can trigger sanctions. The issue moved from isolated embarrassment to a repeatable legal-risk pattern as generative AI use spread through litigation teams.
→
High
Binnall Law Claude Console Phantom Citations in Federal Court
LegalBinnall Law Group PLLC2026
Federal counsel used Anthropic Claude Console to draft a May 2026 motion that included quotations not found in the cited cases. The firm apologised to the court and committed to citation verification via established legal research platforms.
→
High
OpenAI Multi-Service Outage (ChatGPT, Login, Access)
TechnologyOpenAI2026
On 29 May 2026 OpenAI's public status history recorded four separate incidents the same day affecting ChatGPT conversations, login and account creation, ChatGPT access, and business subscription checkout. Teams routing production workloads through a single OpenAI dependency experienced correlated failure across inference, auth, and control-plane paths before OpenAI marked each incident fully recovered.
→
High
Amazon Recruiting AI Discriminated Against Women
TechnologyAmazon2018
Amazon built an AI recruiting tool trained on a decade of historical hiring data. Because most CVs submitted during that period came from men, the model learned to penalise CVs that contained words like 'women's' (as in 'women's chess club') and downgraded graduates of all-women's colleges. Amazon scrapped the tool.
→
Critical
Microsoft Tay Chatbot Taught to Produce Hate Speech
TechnologyMicrosoft2016
Microsoft launched Tay, an AI chatbot designed to learn from Twitter conversations. Within 16 hours, coordinated users had exploited Tay's repeat-after-me feature and lack of input filtering to teach it to produce racist, antisemitic, and misogynistic content. Microsoft took Tay offline the same day.
→
Critical
Uber Self-Driving Car Kills Pedestrian in Arizona
AutomotiveUber2018
An Uber self-driving test vehicle struck and killed a pedestrian in Tempe, Arizona — the first fatal autonomous vehicle crash. Investigation found the vehicle's safety system had detected the pedestrian but classified her as a 'false positive' and disabled the emergency braking system. The backup safety driver was watching a TV show at the time.
→
Medium
GitHub Copilot Reproduced Licensed Code Verbatim
TechnologyGitHub / Microsoft2022
Researchers and users discovered that GitHub Copilot would reproduce verbatim sections of copyrighted, GPL-licensed code from its training data — including commented attribution headers. A class action lawsuit was filed in 2022 alleging copyright infringement and violation of open source licence terms.
→
High
Google Gemini Generated Historically Inaccurate Images
TechnologyGoogle2024
Google's Gemini image generation feature produced racially diverse images for prompts where historical accuracy required specific demographics — including Black Nazi soldiers and female Founding Fathers. The overcorrection of a diversity-promotion safety layer produced historically inaccurate and offensive outputs. Google suspended the feature.
→
High
Italy Bans ChatGPT Over GDPR Violations
TechnologyOpenAI2023
Italy's data protection authority (Garante) ordered OpenAI to stop processing Italian users' data in March 2023, citing ChatGPT's lack of legal basis for collecting personal data, absence of age verification, and a data breach exposing payment information and conversation histories. ChatGPT was blocked in Italy for a month.
→
Medium
DPD Chatbot Jailbroken to Criticise the Company
LogisticsDPD2024
A DPD customer jailbroke the delivery company's AI chatbot, causing it to swear, criticise DPD's own service, and write a poem about how unhelpful it was. The interaction was posted on social media and went viral, generating significant negative press coverage for DPD.
→
Critical
UK Post Office Horizon — AI as Infallible Evidence
Financial ServicesFujitsu / Post Office Ltd1999
The UK Post Office prosecuted over 700 subpostmasters for theft and fraud based on accounting discrepancies produced by the Horizon IT system — a system known internally to have software bugs. Subpostmasters were imprisoned and bankrupted. The Post Office suppressed evidence of Horizon's defects for over two decades. Widely considered the greatest miscarriage of justice in British legal history.
→
Critical
Zillow iBuying Algorithm Collapse
Real EstateZillow2021
Zillow's algorithmic home-buying programme (Zillow Offers) purchased homes at prices its model predicted would appreciate, then resold them for profit. The model failed to track rapidly changing market conditions in 2021. Zillow found itself holding 9,800 homes worth $2.7 billion, purchased at above-market prices. The programme was shut down with a $528 million write-down.
→
Critical
Optum Healthcare Algorithm Systematically Underprovided Care to Black Patients
HealthcareOptum (UnitedHealth)2019
A widely used healthcare risk algorithm sold by Optum used healthcare spending as a proxy for health need. Because structural inequalities mean Black patients historically spent less on healthcare than equally sick white patients, the algorithm systematically underestimated the health needs of Black patients — directing care resources to white patients who were less sick. The study estimated 11.5 million Black patients in the US were affected.
→
High
Facebook News Feed Algorithm Amplified Radicalising Content
Social MediaMeta2018
Internal Facebook research, later revealed by whistleblower Frances Haugen, showed that Facebook's News Feed algorithm was recommending increasingly extreme content to users who engaged with politically charged material — creating a 'rabbit hole' effect. Facebook's own researchers proposed fixes; executives rejected them citing engagement metrics.
→
High
Lensa AI Generated Sexualised Images of Women Without Consent
TechnologyPrisma Labs (Lensa AI)2022
Lensa AI's 'Magic Avatars' feature, which trained a personalised AI model on user-uploaded selfies, disproportionately generated sexualised imagery of women. Researchers found women were significantly more likely to receive sexualised outputs than men, even from professional headshots. The app had no mechanism to prevent or disclose the risk of sexualised output generation.
→
Read the PSF standard →
The Production AI Brief

Keep your AI map current

Tool policy changes, new incident records, disclosure signals, and practical next steps. Public evidence, plain English, no hype.