Production AI Failure Cases

An energy company has a published AI ethics policy, an AI Ethics Officer role on the org chart, and a quarterly AI risk meeting. During your interview, the CISO says they own all AI risk. The CTO says AI governance is a technical matter owned by the AI team. The Legal team says they signed off the policy but have not attended any risk meeting. You find no decision gate records showing AI governance applied to any deployment decision in the past 12 months.

• •Accountability is contested — no RACI documented for AI governance roles
• •The governance committee exists on paper but cannot evidence decisions made
• •No deployment was routed through a governance gate in 12 months
• •Policy exists but is not operationally embedded

• ✓RACI matrix for AI governance roles
• ✓AI risk committee meeting minutes with AI-specific agenda items
• ✓Deployment approval records showing governance gate was applied
• ✓Training records for staff on AI ethics policy

A healthcare provider completes a formal AI risk assessment each January covering all deployed AI systems. In March they upgrade their patient triage model to a new foundation model. In July they expand its use from emergency triage to routine appointment prioritisation. In October you audit them. They present the January risk assessment as evidence of C2 compliance.

• •No trigger-based reassessment was conducted after the March model upgrade
• •Scope expansion in July (new use case) was not treated as a reassessment trigger
• •The July use case expansion is materially higher risk than the original deployment
• •Annual-only cadence at L1; L2 requires documented trigger-based reassessment

• ✓Risk assessment dated after each material model change
• ✓Trigger log: list of events that prompted reassessments and the outcome
• ✓Risk register showing each deployment's current risk tier
• ✓Sign-off records for use case expansions

A SaaS company fine-tuned a support model on 3 years of customer support emails. The emails contained PII, occasional health disclosures, and financial information. The data team confirms no consent was obtained for model training use. The model is now in production. When asked for a data lineage document, the data team provides a pipeline diagram showing how data flows into training — but no record of what data was included, excluded, or reviewed.

• •No consent or lawful basis documented for using customer data in model training
• •No data lineage record identifying what was included in training set
• •PII present in training data with no evidence of masking or exclusion process
• •No data retention or deletion policy covering training artefacts

• ✓Data provenance document for each training dataset
• ✓Consent mechanism or lawful basis assessment for training data use
• ✓PII handling procedure for model training pipelines
• ✓Training data retention and deletion policy

A financial services firm deploys a credit scoring model. They present benchmark results from three academic datasets and an internal A/B test showing 94% accuracy. In production, the model disproportionately denies credit to applicants from two postcodes, which correlates strongly with ethnicity. No bias testing was conducted pre-deployment. The model was approved by the AI team without an independent review.

• •Pre-deployment evaluation used academic benchmarks not representative of production population
• •No bias testing or fairness evaluation conducted on protected characteristics
• •No independent review — model was approved by the team that built it
• •No performance monitoring in production to detect post-deployment drift

• ✓Pre-deployment evaluation report covering bias and fairness metrics
• ✓Independent review sign-off (not from the development team)
• ✓Production monitoring dashboard with fairness metrics tracked over time
• ✓Documented approval gate criteria for high-risk AI deployments

A logistics company deploys a route optimisation agent that makes autonomous delivery reassignment decisions. The system has a documented human override procedure in the operations manual. During stakeholder interviews, three of four operations managers are unaware an override is possible. The fourth found it 'by accident'. No override has ever been logged. The system has been live for 18 months.

• •Override mechanism exists but operational staff are not trained on its use
• •No override event has been logged in 18 months — likely undiscovered, not unused
• •No escalation path documented for situations where the model's decisions appear anomalous
• •Autonomy limits are not defined — the system's decision scope has never been formally bounded

• ✓Training records showing operations staff completed human oversight training
• ✓Override usage logs (presence of zero events with no documented rationale is a finding)
• ✓Escalation runbook for anomalous model behaviour
• ✓Documented autonomy scope defining what decisions the model can and cannot make autonomously

A retailer's recommendation engine begins surfacing inappropriate product combinations to users — recommending items frequently associated with self-harm alongside unrelated products. The issue persisted for 6 hours before being noticed. It was logged in the IT incident management system as a 'software defect — recommendation algorithm' with a 3-day response SLA. No post-incident review was conducted. No regulators were notified.

• •AI incidents are not classified separately from software defects — no AI incident taxonomy exists
• •SLA applied (3 days) was appropriate for software defects, not for an AI harm incident
• •No post-incident review conducted — root cause not established
• •No assessment of regulatory notification obligation was made (GDPR, sector rules)

• ✓AI incident classification taxonomy with severity tiers for AI-specific harms
• ✓Incident response runbook with AI-specific triggers and response steps
• ✓Post-incident review records for significant AI incidents
• ✓Regulatory notification assessment process for AI-related incidents

A benefits agency uses an AI to assess eligibility for housing assistance. An applicant challenges a denial. The agency's legal team requests the reasoning behind the specific decision. The AI team can produce the model version number but cannot reconstruct: which data was used for this specific applicant, what the model's intermediate reasoning was, or why this decision differed from similar applicants. Logs are retained for 30 days; the decision was made 45 days ago.

• •Decision-level logging does not capture input data used for each specific decision
• •Model reasoning / intermediate steps are not logged
• •Log retention policy (30 days) is shorter than the period in which decisions can be challenged
• •No explainability artefacts are produced for high-risk decisions

• ✓Decision log schema showing what is recorded per decision (inputs, model version, output, confidence, timestamp)
• ✓Log retention policy aligned to the legal challenge window for the use case
• ✓Explainability output for high-risk decisions (LIME, SHAP, or structured reasoning trace)
• ✓Immutable audit trail — evidence that logs cannot be altered after the fact

A legal tech company's contract analysis product is built on a third-party model API. The API provider announces a 30-day deprecation of the model version in use. The legal tech company has no contractual minimum notice period, no alternative model evaluated, and no documented continuity plan. Their vendor contract requires 99.9% uptime but has no specific provision for model version deprecation. Their largest customer's contract requires 90-day service continuity notice.

• •No contractual minimum notice period for model deprecation negotiated with vendor
• •No alternative model or fallback capability identified or tested
• •Vendor SLA covers infrastructure uptime but not model version continuity
• •Customer-facing SLA commitment (90-day notice) cannot be met with 30-day vendor notice

• ✓Third-party AI inventory listing all external model dependencies with version, vendor, and criticality
• ✓Vendor contracts including AI-specific SLA terms (deprecation notice, version support period)
• ✓Alternative model or fallback evaluation documented
• ✓Continuity plan for each critical AI dependency with tested recovery procedures