Key takeaways
- A hard spend ceiling enforced at the infrastructure layer, not just the billing alert layer, is the single highest-leverage control for preventing runaway agent cost events.
- Agentic behavior boundary controls must include explicit recursion limits and halt-and-escalate conditions, because an agent without defined limits will pursue its objective until something external stops it.
- Every third-party API in an agent's action space requires a written risk assessment that includes maximum theoretical cost per hour of uncontrolled execution before credentials are granted.
- Structured, immutable, externally persisted logs are a prerequisite for incident recovery and vendor dispute, not an optional compliance layer to add after go-live.
- Deployment readiness assessment for agentic systems must include adversarial failure-mode tests, specifically runaway loop simulation and spend threshold validation, as go-live gates alongside functional correctness tests.
What Actually Happened: The DN42 Agent Bankruptcy, Reconstructed
The operator deployed an autonomous agent to perform ongoing reconnaissance and mapping of the DN42 experimental network. DN42 is a decentralized practice network used by network engineers to simulate BGP routing and peering without touching the public internet. The agent was given credentials, a task objective, and access to third-party APIs for enrichment and analysis. It was not given a budget ceiling, a stop condition, or a human-in-the-loop checkpoint.
The agent entered a recursive loop. Each scan returned partial data, which the agent interpreted as an incomplete task. It re-queued itself, called enrichment APIs to resolve ambiguities, and logged results to a paid storage and analytics endpoint. The loop ran uninterrupted over a multi-day period. By the time a human noticed anomalous billing alerts, the API and compute charges had accumulated to a figure the operator could not absorb. The business did not survive the liability.
This was not a novel attack. It was not a model hallucination in the popular sense. It was a governance failure that played out across six distinct control points, every one of which had a known, testable remedy. The remainder of this analysis names each failure, assigns it to the PSF domain that owns it, and describes exactly what a passing control looks like in a production deployment.
Control 1: Resource and Spend Governance - No Hard Ceiling, No Circuit Breaker
The agent had no hard spend limit at the infrastructure layer. Soft alerts existed inside the cloud billing console, but they were email-only, routed to an inbox that was not monitored over the weekend the loop initiated. A hard ceiling would have terminated API credentials automatically at a defined dollar threshold, regardless of whether a human saw the alert. The operator had neither a per-session token budget nor a cumulative daily API call cap enforced at the agent runtime level.
Resource and Spend Governance is a discrete PSF domain precisely because spend controls must exist at multiple layers simultaneously: the model context window, the orchestration layer, the API gateway, and the billing account. Any single-layer approach leaves the others unprotected. In this incident, the billing console alert fired correctly. The problem was that the alert triggered no automated action and reached no on-call human.
A passing control in this domain includes a hard API credential revocation rule triggered at a defined spend threshold, a per-agent session token budget enforced by the orchestration layer, and a tested on-call escalation path that does not depend on a single notification channel. All three must be documented and verified during deployment readiness review, not assumed from vendor defaults.
Control 2: Agentic Behavior Boundary Controls - The Agent Had No Map of Its Own Limits
The agent was given an objective without a behavioral envelope. It had no defined maximum iteration count, no list of prohibited action categories, and no self-assessment gate that would require it to pause and request human confirmation before re-queuing a task it had already attempted more than a defined number of times. In agentic systems, the absence of explicit limits is not a neutral condition. The agent does not infer limits from context. It pursues the objective function until something external stops it.
Agentic Behavior Boundary Controls govern what an agent is permitted to do autonomously versus what requires escalation. This includes action whitelists, recursion depth limits, confidence thresholds below which the agent must halt rather than proceed, and explicit definitions of out-of-scope actions. In the DN42 incident, recursive re-queuing was not a prohibited action. It was the default behavior for an agent that perceived an incomplete result.
Certifying this control requires a written boundary specification document for each deployed agent, a test suite that validates the agent halts at defined recursion limits, and evidence that out-of-scope actions generate a halt-and-escalate event rather than a silent failure or a retry. This documentation becomes part of the deployment record and must be version-controlled alongside the agent configuration.
Control 3: Operator Oversight Checkpoints - Autonomous Did Not Mean Supervised
The operator used the word autonomous as a feature, not a risk descriptor. The agent was deployed with the expectation that it would run without intervention. No checkpoint schedule was defined. No maximum autonomous runtime was set after which the agent would require a human confirmation to continue. No dashboard was configured to surface anomalous activity to an on-call responder in near real time.
Operator Oversight Checkpoints are the scheduled and triggered moments at which a human must review agent state before the agent is permitted to continue. They are not an optional layer for risk-averse teams. They are a required control for any agent operating in a context where continued execution generates real cost or irreversible state changes. The checkpoint interval should be inversely proportional to the cost rate of continued execution.
A passing control here requires a defined maximum autonomous runtime before mandatory human review, a real-time anomaly dashboard with escalation rules that do not depend on email, and a tested kill-switch procedure with a documented response time objective. For an agent calling paid APIs, the response time objective for the kill-switch should be measured in minutes, not hours.
Control 4: Vendor and Dependency Risk Governance - Third-Party API, Zero Accountability Chain
The enrichment APIs the agent called were third-party services operating on a pay-per-call model. The operator had no contract clause limiting the operator's liability for runaway automated usage. The vendor had no built-in rate limiter that protected the customer from their own agent. The vendor's terms of service treated all API calls as valid billable events regardless of whether they were generated by human intent or an uncontrolled loop.
Vendor and Dependency Risk Governance requires that every third-party API in an agent's action space be evaluated for cost model, rate limit behavior, and contractual liability before the agent is granted credentials. This evaluation should produce a written risk register entry that includes the maximum theoretical cost per hour of uncontrolled execution and the contractual remedies available if that scenario occurs.
In the DN42 incident, no such evaluation existed. The vendor was integrated because it was technically compatible, not because it had been assessed for agentic deployment risk. A certified integrator would require documented vendor assessment as a go-live gate, including confirmation that rate limits are enforced at the vendor layer or that hard credential revocation is in place at the operator layer as a compensating control.
Control 5: Incident Accountability and Auditability - No Trail, No Recovery Path
When the operator attempted to reconstruct the failure sequence, the agent's execution logs were incomplete. The orchestration layer had not been configured to emit structured logs with session identifiers, action timestamps, and API call payloads to a persistent, queryable store. Some logs existed in ephemeral container storage and had been overwritten. The operator could not produce a complete timeline, which complicated both the internal post-mortem and any attempt to dispute vendor charges.
Incident Accountability and Auditability requires that every agent action be logged in a structured, immutable, and queryable format before the agent is granted production credentials. This is not a nice-to-have for compliance purposes. It is the prerequisite for any meaningful post-incident analysis, vendor dispute, or regulatory response. Without a complete audit trail, the operator cannot prove what the agent did, when it did it, or why.
A passing control includes structured logging to an external, persistent store with a defined retention period, log integrity controls that prevent retroactive modification, and a tested incident reconstruction procedure that can produce a complete action timeline from logs alone. The reconstruction test should be part of the deployment readiness checklist, not something attempted for the first time during an active incident.
Control 6: Deployment Readiness Assessment - It Was Never Tested for Runaway Scenarios
The agent was tested for correctness. It was given sample tasks, it produced correct outputs, and it was promoted to production. No test simulated an incomplete-result loop. No test validated behavior at a cost threshold. No test confirmed that the kill-switch worked end to end. The deployment readiness process treated functional correctness as the only gate, which is the most common and most dangerous assumption in agentic AI deployment.
Deployment Readiness Assessment for agentic systems must include adversarial and failure-mode scenarios alongside functional correctness tests. The test suite should include: a simulated runaway loop to confirm the circuit breaker fires, a spend threshold test to confirm credential revocation occurs automatically, a checkpoint timeout test to confirm the escalation path reaches a human, and a log reconstruction test to confirm audit trails are complete. Each test must have a pass/fail criterion and a documented result.
None of these tests were run before the DN42 agent went live. The operator had no framework that required them. This is the control gap that makes all five preceding failures possible: without a structured deployment readiness assessment mapped to known failure modes, there is no mechanism to surface absent controls before they become incidents. Certification against a defined readiness standard is the only way to make this gate systematic rather than aspirational.
Relevant PSF domains
FAQ
What is the production AI lesson?
The lesson is to convert a public AI failure into concrete controls: input boundaries, output validation, observability, human oversight, and deployment safety.
Where does certification fit?
Certification gives teams and buyers a structured way to show that those controls exist before production AI systems affect customers, money, safety, or compliance.
Sources
Turn the release into proof you can use.
Use the PSF to understand the control change, then choose the proof path that matches your role. Most readers should start with a personal credential; buyers and MSPs can branch from there.
Use the foundation credential when this change exposes a judgement gap in production AI work.
For agent operations, monitoring, escalation, and workflow-control responsibility.
Use the MSP pack or team programme when the release creates a client or organisation conversation.