PSF Domain 7: Security

AI systems have a fundamentally different threat model than conventional software. Prompt injection is not a SQL injection analogue — it is a new attack class. Model supply chain attacks are documented and active. RAG pipelines introduce retrieval corpus poisoning as a new attack surface. This guide maps the AI-specific threat model and the controls that close each vector.

The D7 gap: traditional security frameworks (OWASP Top 10, ISO 27001) do not cover AI-specific attack vectors. Prompt injection, indirect injection via retrieved content, and model supply chain attacks are not addressed by conventional application security tooling. Every major framework is Gap or Partial on D7.

Attack surface: User inputs, form fields, customer messages

Attacker embeds instructions in user input that override system prompt. Target: make the AI take unintended actions, exfiltrate context window contents, or break guardrails.

• Input sanitisation and instruction-following boundary enforcement
• Privileged prompt / user prompt architectural separation
• Output monitoring for signs of instruction override

Attack surface: Retrieved documents, web content, database records, email/calendar data

Attacker embeds instructions in data the AI retrieves (a poisoned document, a webpage, a malicious email). When the AI processes the data, it executes the embedded instructions. Particularly dangerous in RAG pipelines and web-browsing agents.

• Treat all retrieved content as untrusted data, not instructions
• Structured output contracts reduce the attack surface (model constrained to a schema cannot execute free-form injected instructions)
• Monitor for anomalous tool calls or actions immediately after retrieval steps

Attack surface: Public-facing AI inference endpoints

Attacker queries the model repeatedly to reconstruct training data or determine whether specific data was in the training set. Relevant when models are fine-tuned on proprietary or sensitive data.

• Rate limiting and anomaly detection on inference endpoints
• Differential privacy techniques during fine-tuning where training data is sensitive
• Avoid fine-tuning on data you would not want reconstructed

Attack surface: Model weights, fine-tuning pipelines, third-party model repos

Attacker inserts a backdoor into model weights or a training dataset. The backdoored model behaves normally on most inputs but triggers on a specific pattern. HuggingFace models have been found to contain embedded pickle exploits.

• Only use models from audited, trusted sources with cryptographic integrity verification
• Scan HuggingFace and other community models for malicious code (safetensors format preferred over pickle)
• Never run unverified model weights in production without sandbox evaluation

Attack surface: Vector databases, document ingestion pipelines

Attacker introduces malicious documents into the retrieval corpus. When retrieved, these documents influence AI outputs towards attacker goals. If the corpus includes user-supplied content, any user can attack any other user via poisoned documents.

• Document provenance tracking — know what is in your retrieval corpus
• Ingestion-time content scanning for injection payloads
• Segregate user-supplied content from trusted internal documents in the retrieval index

Attack surface: AI systems processing images, audio, or structured data

Attacker crafts inputs that appear normal to humans but cause the AI to produce incorrect outputs. Particularly relevant in visual AI and audio processing systems.

• Adversarial robustness testing before deployment
• Ensemble approaches reduce single-model adversarial vulnerability
• Input preprocessing pipelines that reduce adversarial perturbations

Architectural approaches that reduce injection impact (none eliminates it completely):

• Instruction hierarchy: some models support explicit privileged prompt / user message separation that reduces cross-tier instruction propagation
• Structured output enforcement: if the model is constrained to output a specific JSON schema, injected free-text instructions cannot produce arbitrary actions
• Tool call whitelisting: AI agents should have the minimum tool set required — limit blast radius of a successful injection
• Input sandboxing: run untrusted input processing in a separate model context with no access to sensitive tools or data
• Output monitoring: flag anomalous outputs that suggest instruction override (system prompt reproduction, tool calls outside normal scope)

Supply chain security baseline for production AI:

• Only use models from sources with documented security review processes (major providers, audited repos)
• Verify model checksums / SHA256 hashes against published values before loading
• Prefer safetensors format over pickle-based formats for community models
• Run new models in a sandbox environment for 24-48 hours before production evaluation
• Maintain a model SBOM (Software Bill of Materials equivalent) listing every model in production with source, version, and integrity hash

No native security controls. Prompt injection is undefended by default. Tool call permissions are binary. Community hub chains and tools are not vetted and represent a supply chain risk.

Multi-agent architecture amplifies injection risk — a compromised agent can propagate injected instructions to other agents in the crew. No inter-agent trust boundaries provided.

Human-in-the-loop mode provides some injection defence by default. Code execution sandbox is available for tool calls. Still no input sanitisation or context isolation.

Plugin permission model provides function-level access control. Azure-backed deployments benefit from Azure AI Content Safety integration. More security primitives than most frameworks.

Structured output constraint reduces free-form injection execution surface. Tool definitions are typed and bounded. No explicit injection defence but schema enforcement limits attack impact.

Pipeline architecture provides logical separation but no security boundaries. Document ingestion has no content scanning. Retrieval pipelines are vulnerable to corpus poisoning.

Related Guides