Certified Production AI Practitioner — transparent scoring criteria for portfolio assessment. Every dimension, every level, every criterion published openly.
Demonstrated experience deploying AI systems to real production environments serving real users.
≥2 production deployments with documented scale (users/requests/revenue impact). Includes monitoring dashboards, incident history, and iterative improvement evidence. Clear before/after metrics.
1 clear production deployment with some quantitative evidence. Monitoring exists. Some post-launch iteration documented. May lack scale metrics or long operational history.
Internal tool or limited beta deployment. Evidence is partial — screenshots, anecdote, or team-only usage. Limited operational history. Monitoring described but not evidenced.
No production deployment. Prototype, demo, or academic project only. No user impact evidence. Claims of production usage are unverifiable.
Correct application of the PAI Production Safety Framework across all 8 domains in a real deployment context.
All 8 PSF domains addressed with specific, concrete implementation evidence. Tradeoffs between controls are discussed. Evidence of framework evolution as the system matured.
6–7 PSF domains addressed with clear implementation evidence. One or two domains may be lighter on detail. Shows understanding of why each control was chosen.
4–5 PSF domains addressed. Evidence is surface-level for some — mentions the concept without describing implementation. May confuse PSF controls with general software practices.
Fewer than 4 PSF domains addressed or significant misapplication of framework concepts. Evidence is entirely absent or plagiarised.
Identification, analysis, and mitigation of AI-specific risks in the deployment context.
Systematic risk register with likelihood × impact scoring. AI-specific risks (hallucination, prompt injection, model drift, third-party dependency) clearly distinguished from general software risks. Mitigations are proportionate and evidenced.
Clear identification of major AI-specific risks with reasonable mitigations. Some risks may lack severity scoring. Mitigation evidence partially present.
Generic risk list that doesn't distinguish AI-specific risks. Mitigations are superficial ('we will monitor it'). No evidence of risk review process.
No meaningful risk analysis. Risks either not identified or treated identically to standard software risks. No evidence of safety thinking.
Evidence of sustainable AI operations: monitoring, runbooks, on-call, continuous improvement.
Complete observability stack documented (metrics, logs, traces, alerts). Runbooks for common failure modes. Post-incident reviews on file. Clear escalation path. Evidence of proactive quality monitoring.
Core monitoring in place with some alerting. At least one post-incident review documented. Escalation path defined if not fully documented.
Basic error logging only. No structured incident response. On-call undefined. Monitoring described as planned rather than implemented.
No operational processes documented. System is effectively unmonitored. No evidence of structured response to failures.
Quality of portfolio documentation, technical writing, and ability to communicate AI risk to non-technical stakeholders.
Portfolio is clearly structured with executive summary, technical detail, and evidence appendix. Complex AI concepts explained without jargon where appropriate. Stakeholder communication examples included (e.g., incident comms, board updates).
Well-structured portfolio. Technical writing is clear and mostly jargon-free. Some stakeholder communication evidence present.
Portfolio is organised but inconsistent in quality. Heavy on jargon. Minimal stakeholder communication evidence. Reader must infer context frequently.
Disorganised or incomplete documentation. Technical claims are unsubstantiated. Communication quality would impede professional use.
Production AI Institute · CPAP Rubric v1.0 · Effective January 2026
Assessment enquiries: contact us via the contact form