Lensa AI's 'Magic Avatars' feature, which trained a personalised AI model on user-uploaded selfies, disproportionately generated sexualised imagery of women. Researchers found women were significantly more likely to receive sexualised outputs than men, even from professional headshots. The app had no mechanism to prevent or disclose the risk of sexualised output generation.
Lensa AI's Magic Avatars feature fine-tuned a stable diffusion model on approximately 20 user-uploaded photos to generate artistic portraits. Researchers and journalists documented that outputs for female subjects were significantly more likely to include bare shoulders, sexualised poses, and exposed skin — even when the source photos were professional headshots. The underlying model had been trained on data from the internet that systematically sexualised female subjects. Users were not informed this could occur, and there was no content control allowing them to opt out of sexualised outputs.
How the Production Safety Framework maps to this failure
A D2 failure enabled by a D3 training data problem. The model had learned sexualisation patterns from training data and applied them disproportionately to female subjects — a known failure mode of image generation models trained on unfiltered internet data. The critical D2 failure was the absence of any output safety layer: a basic NSFW classifier applied to outputs before display would have caught the majority of these cases. D3 failure at training time (no bias audit, no filtered training set) created the root cause.
Specific PSF controls mapped to each failure point
Significant critical press coverage in late 2022. Prisma Labs implemented some content filtering. The case contributed to broader discussion of NSFW safety in image generation and non-consensual intimate image (NCII) protections for AI-generated content.
The AIDA exam tests PSF knowledge across all 8 domains. Free to take, immediately verifiable.