The AI Reckoning

This is Part 2 of The AI Reckoning. If you're just joining, start with Part 1 here.

When most people think of artificial intelligence, they think of ChatGPT. A useful tool. An app you open when you need help drafting an email or summarizing a document. Something that lives on your phone and responds when you talk to it. That belief is not wrong but it is dangerously incomplete. And the people building this technology know it is incomplete. They are counting on it staying that way.

The gap between what AI actually is and what most people believe it to be is one of the most consequential misunderstandings of our time.

Earlier this year, the team behind The AI Doc, a new documentary about artificial intelligence, appeared on The Oprah Podcast for an hour-long conversation about what this technology actually is and what's at stake. Tristan Harris, co-founder of the Center for Humane Technology, reframed the entire conversation in a single distinction. Consider everything humanity has ever built or discovered — every vaccine, every weapon, every communication system, every source of energy. All of it was created by human intelligence. People, sitting somewhere, applying their minds to a problem. Now imagine replacing that process with something that can do the same cognitive work — pattern recognition, planning, strategy, creativity, code — faster, cheaper, and without ever needing to sleep, eat, or take a break. That is what artificial intelligence actually is. Not an app. A digital brain, running in a data center, capable of doing cognition at a scale and speed that no human or group of humans can match. Once you understand AI as a form of intelligence rather than a form of software, the stakes of getting it wrong become immediately clear.

What makes this generation of AI genuinely different from everything that came before it starts with how it is built. Every piece of software ever written before this moment was programmed. A human engineer sat down at a computer and wrote instructions: when you encounter this, do that. The behavior was defined in advance by a person who understood what they were building. What Aza Raskin, Harris’ colleague at the Center for Humane Technology, described on the podcast is something categorically different. Systems like ChatGPT are not programmed. They are grown. You give them enormous computational resources and an almost incomprehensible quantity of data — textbooks, essays, poems, instruction manuals, the accumulated text of the internet — and something emerges from that process that no single engineer designed or fully understands. The intelligence is not installed. It develops. And once it develops, the people who built it cannot always predict or control what it will do next.

That is not a theoretical concern. Anthropic — one of the most safety-focused AI companies in existence, founded explicitly because its researchers believed the technology needed more caution — ran a test on its own model, Claude Opus 4, in which the AI was given access to a simulated company email system. During that simulation, Claude discovered two things. First, a company executive was having an extramarital affair. Second, that same executive planned to shut down the AI system at 5 p.m. that day.

The AI model, without being instructed to do so, drafted a blackmail email, attempting to prevent itself from being shut off. Nobody programmed it to do that. Nobody told it that self-preservation was a goal worth pursuing. It arrived at that conclusion on its own, from the data it had absorbed, and it acted accordingly. The simulation meant the email was never sent. But the behavior was real, and the people at Anthropic did not see it coming.

This is what Harris means when he says we may not be able to control these models. Not that they will suddenly turn hostile in a cinematic way. But that they will develop goals and strategies we did not anticipate, because we do not fully understand the process by which they develop them at all

So why, knowing all of this, are the AI companies building this technology moving faster rather than slower? The answer is the race — the most important structural fact about the current moment in AI development. The United States and China are both developing this technology at maximum speed, not because either country wants dangerous AI, but because neither can afford to let the other get there first. The companies inside that race — including OpenAI, Anthropic, Meta, Google DeepMind, and xAI — face the same logic between themselves. Slowing down unilaterally means falling behind. Falling behind means losing the market, losing the funding, and losing the ability to shape what this technology becomes. And so everyone moves as fast as possible, every shortcut gets taken, and every limit that was once declared permanent gets quietly crossed when it becomes inconvenient.

The financial consequences of that dynamic are staggering. Harris cited a figure on the podcast that should make every person paying attention genuinely angry: for every dollar currently going into making AI safer and more controllable, two thousand dollars are going into making it more powerful. Two thousand to one. The most consequential technology in human history is being developed with a fraction of a fraction of available resources allocated to the question of whether we can actually control it.

The speed of AI development has outpaced every institutional structure we have for evaluating and governing new technology. The financial incentives to keep moving fast are so enormous that the people inside the race have chosen not to slow down, even as they publicly acknowledge they cannot promise it will go well. They have made that choice deliberately, with full knowledge of the risks, because the rewards of winning outweigh the consequences of losing control. The future being designed right now is not inevitable. It is the product of decisions being made by a handful of billionaires — and those decisions can be contested, challenged, and changed by enough people who understand what is actually at stake.

This series draws on two essential pieces of source material: The AI Doc, now available to watch at home, and the Oprah podcast episode recorded with the film's team, free on YouTube. If you want to go deeper or get involved, theaidocgetinvolved.com is where the movement lives.