Incident Patterns in Production LLM Deployments

Key findings

The most consistent pattern is the concentration of failures in PSF domains 1 and 2. Input governance failures — unvalidated user inputs reaching the model — and output validation failures — model outputs consumed downstream without schema validation — together account for the majority of preventable failures observed across assessments. Both are addressable with straightforward engineering controls.

Incident distribution by PSF domain

The prompt injection problem

Prompt injection — the technique by which adversarial content in user inputs or retrieved documents causes the model to deviate from its intended behaviour — is consistently among the most common failure categories observed, and represents the fastest-growing vector in public reporting.

In the majority of injection cases, the attack vector is retrieved content: documents, emails, or web pages that the system is asked to summarise or process contain embedded instructions. A smaller proportion involve direct user manipulation of the prompt structure.

What resolves incidents fastest

Time-to-resolution correlates most strongly with two factors: whether the system had observable logging (PSF-4) and whether a human escalation path existed (PSF-6). Systems with both in place resolve failures significantly faster — the gap between prepared and unprepared systems is consistently greater than 4× in reported cases.

The intervention pattern most associated with fast resolution is the existence of a named incident owner — a person whose responsibility includes monitoring the system — combined with alerting thresholds configured before deployment. The majority of reported failures lack both conditions at the time of the incident.

Methodology and limitations

This analysis draws on three sources: patterns observed by PAI assessors during CPAP and CPAA portfolio reviews; publicly reported post-mortems and disclosed incidents from AI practitioners and organisations; and practitioner-submitted case notes contributed voluntarily to PAI. All case material is treated as confidential and no identifying information is included.

The proportional breakdowns in this paper reflect relative frequency across the observed corpus, not a statistically random sample. They should be interpreted as directional indicators of where practitioners are most likely to encounter failures — not as precise population estimates. Cases of severe or ongoing failures are likely under-represented, as practitioners are less able to share them.