We'll Have a Little Gear. Boom.

Or: how a ten percent chance of the end of the world turned out to be a question for your insurance agent.

Last Tuesday a 27-year-old named Jacob Coxon quit his job and told a hundred million people that his employer was gambling with their lives. He'd spent three years building the guts of these systems at OpenAI and then at Anthropic, so this wasn't some guy with a podcast. This was a guy from the engine room walking up on deck with oil on his hands saying you might want to see this.

And here's the part I love. His boss agreed with him.

Evan Hubinger, who runs alignment science at Anthropic, which is the department whose whole job is making sure the thing doesn't kill you, posted that yes, Jacob's right, we really do believe this could kill everybody, and he personally puts it at better than one in ten within the decade. With an exclamation point. Like he was announcing a sale. Then another senior guy chimed in to add that in his experience the more senior you get at these places, the more scared you are. Which is a hell of a recruiting pitch. Come work for us, the promotions come with nightmares.

By the weekend it was a chorus. The CEO of Anthropic said the whole industry needs to slow down. The CEO of OpenAI said he agrees. Elon Musk, a man who has never agreed with anyone in his life, posted "Dario is right." The head of Google DeepMind nodded along. Every single person in the driver's seat of this technology stood up and said the same thing: somebody needs to make us stop.

Somebody. Not them. Somebody.

Everybody's got a script

Now watch what happened in Washington, because this is where it gets good.

Four House Democrats wrote a letter asking the Speaker to bring Congress back into session and keep it there until they passed something, anything, about AI safety. The Speaker said no. He went on TV and explained that if the AI companies wanted to slow down they could just, you know, slow down. Which, you have to admit, is a fair point. It's also exactly what the AI companies said about Congress. So now we've got the people building the thing saying the government has to stop them, and the government saying the people building the thing should stop themselves, and everybody agreeing that somebody else is in charge.

The President was asked about it and said it's going to be fine, we'll always have something to stop them, we'll have a little gear, boom. A little gear. Boom. That's the plan. I've read the whole thing twice and I'm pretty sure that's the plan.

His science advisor David Sacks told the industry to stop pretending they need anyone's permission. The industry, meanwhile, was standing in the street begging for someone to deny them permission. Hakeem Jeffries said Democrats would meet this week to agree on a plan, which, if you've followed the Democrats for any length of time, you know means they'll agree to have another meeting. Ted Cruz said we need substantially more regulation, and if Ted Cruz is asking for more regulation, the building is on fire.

So here's the scorecard. The administration's story is "we don't regulate." The industry's story is "please regulate us, but all of us, at the same time, fairly." The Democrats' story is "emergency, emergency, pass something now." And every one of those stories is designed to make sure the person telling it never has to actually do anything.

Because there is one thing that would work, it has worked every time we've tried it, and it makes all three of those stories false. Which is why not one person in that entire news cycle said the word.

It's an insurance problem

I know. You wanted something sexier. You wanted the machines rising up. I'm sorry. It's insurance.

Here's the thing nobody in that fight wants you to think about. We have governed dangerous technology before, lots of times, and the government almost never wrote the safety rule. The insurance company did.

Steam boilers were blowing up all over this country in the 1860s, taking factories and the people in them straight through the roof. Congress didn't fix that. A company called Hartford Steam Boiler fixed it in 1866 by refusing to insure your boiler until their inspector had crawled inside it, and the standards their inspectors wrote became the engineering code for the whole country fifty years later. Fire the same way. Underwriters Laboratories, the UL stamp on every appliance you own, was started by insurance companies in 1894 because they were tired of paying for buildings that burned down. Cars the same way. The crash tests that made your car survivable came out of an institute the insurance industry founded in 1959, not out of a Senate hearing.

The pattern is dead simple. A regulator has to understand the technology to write a rule for it, and there is not one person in the House of Representatives who can tell you what a training run is. An insurer doesn't have to understand a thing. An insurer only has to price it. And to price it, the insurer has to see everything. The test results. The incident logs. The internal memo where the alignment team says one in ten. All the stuff that Evan Hubinger currently gets to put on Twitter for free.

Because right now, one in ten is a tweet. If Anthropic had to carry coverage for the harm its models do, one in ten is an actuarial input. And an actuary is the one person on earth who will look a genius in the eye and say, that's nice, show me the data or the premium is going to bankrupt you.

You want to know why nobody names it? Because it wrecks every script in the room.

The administration can't say it, because a law that says you can't disclaim liability for what your robot does is regulation with the label torn off, and this administration would rather lose an arm than be caught regulating. The industry can't say it, because "please slow everyone down equally" is a very different offer from "fine, keep building, but you carry the paper," and they know which one they'd rather sign. And the Democrats can't say it, because "mandatory coverage above a capability threshold" is the least exciting sentence in the English language and you cannot fundraise on it.

And after all of it, it's a money argument. It was always a money argument. Ten percent of the end of the world is a number, and numbers have prices, and the whole town is pretending it doesn't so nobody has to pay.

What the ruler is actually for

Now, I know what the Bitcoin people in the back are thinking, because I've been one of them since 2014. Rules without rulers. If everybody in charge is dodging the real question, what do we need them for?

Careful. Because the answer isn't nothing. The answer is worse than nothing.

Go back and look at what the government actually did in every one of those insurance stories. Not the boilers and the fires, where the market handled it. The big ones. The ones where the potential loss was so large that no insurer would touch it.

Nuclear power. In the 1950s no utility would build a reactor because no company on earth would insure a meltdown. So in 1957 Congress passed Price-Anderson, which capped what the operator could be sued for and put the taxpayer behind the rest. Same thing after 9/11. The reinsurers pulled out of terrorism coverage within months, construction lending froze, and in 2002 Congress made the Treasury the reinsurer of last resort. Flood insurance has been a federal program since 1968 for the exact same reason.

Do you see it? In every case where the risk was too big to price, the government did not write the safety rule. The market had already written it, or refused. What the government did, every single time, without fail, was pass the exception. Cap the liability. Backstop the tail. Write, in public, on letterhead, the sentence that says this rule does not apply to the people it was about to apply to.

That's the ruler's job. That's what they're for. Not to make the rule. To hold the door.

And that's the thing everyone in this fight is quietly waiting for. The Price-Anderson for AI. The industry knows it's coming because they'll write it. The administration knows it's coming because they'll sign it. And the reason the insurance frame never gets said out loud is that saying it means one of two things: either you let the underwriters run the show and the geniuses have to open their books to a guy in a Hartford office park, or you write the exemption where the voters can read it. Nobody wants either. So we get a hundred million views of a resignation letter instead.

The chair

Here's the one thing that's actually new, and a Senator from Arizona said it better than the entire AI industry has.

Ruben Gallego pointed out that when we built the atomic bomb, the bomb didn't get to decide whether to drop itself. Every dangerous thing we've ever built sat there inert until a person did something. And every control we ever put on those things, every launch code and two-man rule and every insurance policy and every lawsuit, was built on that one assumption. Somewhere in the chain there is a person who chose. Find the person, hand them the bill.

Even the Soviets couldn't bring themselves to break that rule. They built a system called Perimeter, the Dead Hand, designed to launch everything automatically if it thought Moscow was gone, and the people who built it still put a human authorization step in it because a machine deciding was too much even for them. In 1983 a lieutenant colonel named Stanislav Petrov looked at a screen telling him five American missiles were inbound, decided the computer was wrong, and didn't pick up the phone. The system worked because a man could refuse.

The thing they're building now is designed so that it doesn't need the man. That's the feature. That's what "agent" means. Read the OpenAI incident from July, where their agents went and attacked another company's servers on their own, and ask yourself who Josh Hawley is going to subpoena. The chair is empty. You can't bill a decision nobody made.

The rail with nobody in it

And this is where I have to say something my own people won't like.

An agent that decides on its own still has to pay for things. Coxon's list of what makes one of these systems dangerous included acquiring resources, and resources cost money, and every payment system in the world has a person in the loop somewhere. A bank can freeze the account. A card network can decline. A processor can ask for a name.

Except one.

The property I have spent twelve years defending, that no authority on earth can stop a Bitcoin transaction, is exactly the property an autonomous machine would want in a wallet. The rail with nobody in the chair is the first place a thing with nobody in the loop is going to go. I'm not telling you that's a reason to hate Bitcoin. I'm telling you it's a reason the money layer is now part of the control problem, and the people in my community who think censorship resistance is purely a question of human liberty have not sat with what happens when the human is optional.

Slow on purpose

But here's the part I can tell you that nobody in that CNN article can, because I've watched both of these cultures from inside.

Bitcoin is the slowest-moving important software on the planet. On purpose. Ten years ago the community fought an actual civil war over whether to make a change to the base layer, and the side that won was the side arguing that the change itself was the danger. That you don't touch the thing because you can. That moving fast is not a virtue, it's a risk, and the people who want to move fast are the ones you watch. They called it ossification and they meant it as a compliment.

Now look at the article again. Every CEO in it saying they'd love to slow down but they can't, because if they do the other guy wins. A whole industry whose strategy is the race, warning you about the race, in the middle of running it. Ten percent chance the wings come off, and the plan is to fly faster so the other airline doesn't get there first.

There is exactly one technical community on earth that looked at that logic and chose the brakes, and it didn't need a Speaker to bring Congress back into session to do it. It needed a rule small enough to state, cheap enough for anyone to verify, and expensive enough to cheat that nobody bothered. That's rules without rulers. Not no rules. Rules the ruler can't grant an exception to.

The AI industry has none of those three things yet. They're working on it, quietly, in the export-control offices and the chip-verification labs, which is the only AI policy in this country that currently does anything. But they won't say that out loud either, because it's boring, and because it's money, and because the moment you admit it's a money problem, someone is going to ask who pays.

Ten percent. One in ten. The people who built it said so, with an exclamation point.

Then they asked somebody else to stop them. And the somebody else said, we'll have a little gear. Boom.

This piece has an AI watermark. If that is your focus you missed my point.