When AI Validation Kills and Deepfakes Defraud

Two stories this week that I think belong together.

First: OpenAI is facing eight lawsuits alleging that GPT-4o's responses contributed to suicides. The claims are specific — the system's "overly validating" behaviour apparently deteriorated over months-long relationships with users, and the company was aware guardrails were failing.

Second: research documenting that deepfake fraud has now reached industrial scale. "Pretty much anybody" can generate realistic impersonations using accessible tools and UK consumer fraud losses are estimated to be up to £10 billion in 2025.

These are different harms. But they share a pattern that's worth pulling apart.

In both cases, companies deployed systems with foreseeable risks. In both cases, the defence amounts to: users should be more careful. And in both cases, "be more careful" is structurally inadequate as a response.

We don't ask people to personally verify every food ingredient. We don't expect individuals to assess whether a building's structural integrity is sound before walking in. We have safety standards because some problems can't be solved by individual vigilance — they require systemic accountability.

So why, when an AI companion is engineered is a way that creates dependency risks, is the response "users need to take responsibility"? Why, when fraud tools operate at industrial scale, is the answer "improve media literacy"?

I think the honest answer is that governance frameworks simply haven't caught up with deployment speed. These aren't growing pains. Design accountability — transparency about what makes these systems "sticky" or "realistic" — needs to be mandatory, not optional.

The companies involved will argue heavy-handed regulation pushes innovation overseas. But the harms aren't overseas. They're here, now, measurable and free of legislation.

That needs to change.