:date: 2026-03-08 21:46
Today I read an essay by Dario Amodei, the co-founder and chief shaman at Anthropic, the company behind the robot friend, Claude.
In this essay, he covers a lot and tries to thread the needle between personally having $7 billion and talking to normal humans like he's a normal human too. He's mostly wants to tell us how he hopes his company is not intentionally eschatological — they're not trying to end humanity! Heavens no!
I am actually not really much of an AI doomer and I don't really care as much about the obliteration of Nerddom as one might suppose. But that doesn't mean muddled thinking escapes my attention. Instead of writing a coherent tight response to this article, it was more fun for me to go do something else and just leave you with my own collection of muddled thinking — notes I took while reading this essay. Enjoy!
His essay is crazy long! 21.7k words — 10x longer than this (too long) post. Bro, does that even fit in context?
"Humanity is about to be handed almost unimaginable power..."
I think that horse left the barn with industrial power. And literacy. And torture.
He defines "powerful AI" with (many things including this):
"In terms of pure intelligence, it is smarter than a Nobel Prize winner across most relevant fields: biology, programming, math, engineering, writing, etc. This means it can prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch, etc."
I am not Nobel-prize-winner-smarter than normal people but the fact that I have tons of successful experience in biology, programming, engineering, writing, etc. shows that if this stuff were important, my life would be more important. But I can assure you, nobody (statistically) gives a fuck about these things in real human interactions.
It says later:
"Everyone having a superintelligent genius in their pocket is an amazing advance and will lead to an incredible creation of economic value and improvement in the quality of human life."
And I'm sorry, as someone who has enjoyed the benefits of slightly elevated literacy and STEM nerdery I think the author is over aggrandizing these traits more than normal humans would. Consider that the industrial, scientific, and computer revolutions have been driven by a modest percentage of humans doing actual nerd work. Scaling nerdery does not necessarily scale quality of life. The large growth in the percentage of college education in my lifetime has not had commensurate beneficial effects. Keeps people out of trouble I guess.
"[Fancy AI]...would have a fairly good shot at taking over the world (either militarily or in terms of influence and control) and imposing its will on everyone else..."
My fellow humans, the interesting thing about AI is that it has no will. "Imposing its will" is like imagining my toaster imposing its will on my toast or my table saw imposing its will on a sheet of plywood. The illusion that AI has a will is really it bumbling around the artifacts left in the wake of humans having a will that they, the humans, were trying to impose. If a parrot says, "I want to impose my will on you," it is still a parrot's will and that utterance's message was not part of it.
"We now know that [fancy AI is] a process where many things can go wrong."
I agree with this but it's in the same way that a toaster can electricute you if you if you drop it in a sink full of water or a table saw can cut off fingers. It is a distraction to imagine the AI cleverly scheming to get "what it wants" — it doesn't want anything!
It is a bit unnerving that keeping this in mind is not a disciplined habit for the CEO of a big company responsible for keeping one of the major bots under control.
I'm reminded of people who misunderstand evolution who often speak of what evolution "wants" or "the goal of evolution". Wrong!
All of this reminds me of old timey science fiction. Even the subtitles "I'm sorry, Dave" and "Player Piano" allude to it. I'm not the only one pushing back on this sci-fi credulity. These brilliant Kiwi philosophers had all this figured out at least 16 years ago: "Finally robotic beings rule the world."
"...[fancy AIs] could conclude that they are playing a video game and that the goal of the video game is to defeat all other players (i.e., exterminate humanity)."
Finally! Someone getting close to realizing where true AI progress and threats can be measured: NPCs who still are not even close to convincing! Seriously, NPCs have guns (all of the guns!) and are doing their level best to kill me. Show me a good NPC and I will start worrying. Seriously, when AI starts hollowing out PvP lobbies, that is a metric you can use to chart the apocalypse.
"But I agree that a lot of very weird and unpredictable things can go wrong..."
True enough. Something tells me that him not being wealthier than the entire population of, say Dayton Ohio, is not one of those unpredictable things he is worried about. You may object to me carping about this dude's pathological wealth but to me that is exactly one of those weird and unpredictable things that can go wrong and already has. The failure mode is not hard to predict — we're living it!
"I suspect the situation is not unlike with humans, who are raised with a set of fundamental values (“Don’t harm another person”): many of them follow those values, but in any human there is some probability that something goes wrong, due to a mixture of inherent properties such as brain architecture (e.g., psychopaths), traumatic experiences or mistreatment, unhealthy grievances or obsessions, or a bad environment or incentives — and thus some fraction of humans cause severe harm. The concern is that there is some risk (far from a certainty, but some risk) that AI becomes a much more powerful version of such a person, due to getting something wrong about its very complex training process."
I have an important objection to this line of thinking. One of the key concepts that enables the whole parlor trick of Turing Test passing bots is that they are trained on an absolutely enormous corpus of human culture. This means we should not worry so much about some particular pathological human wierdos but rather we should worry about the general global human. A psycho killer may be a valuable boogie man that 99.99999% of humanity can use as a foil to calibrate disgust.
Sometimes you meet a person who is quirky. They have a weird sense of humor or they created some esoteric programming language or they're several sigmas out there in their enjoyment of skiing or something like that. That is not what AI does. Left unspecified, its natural style is best described as "generic human". Are generic humans slightly racist bumptious dipshits? Sure, but they're not generally genocidal monsters.
AI is like a wig. Might look like really impressive hair. Might be great for some special circumstances like a Commonwealth courtroom or a porn shoot or especially those two combined. But it's not real hair. And analogously, if you encourage "Claude to think of itself as a particular type of person (an ethical but balanced and thoughtful person), and even encourage... Claude to confront the existential questions associated with its own existence in a curious but graceful manner," well, you may get the appearance of those things but you won't get those things any more than a toupee gives you real hair. Could be enough for most situations, but it's just important to keep in mind what we're really dealing with here.
"We believe that a feasible goal for 2026 is to train Claude in such a way that it almost never goes against the spirit of its constitution."
Good luck with that. Red team says: hold my beer.
Later he correctly says, "But all models can be jailbroken..."
He uses the phrase "rapid efforts" — what an odd thing to say! As an athlete I've done hard efforts for achieving rapid speeds but a "rapid effort" could almost sound like an effort that is over quickly and therefore easier than, say, a "prolonged effort". Just an odd choice of words. Did no robot friends check his essay? (Bet you wish one checked this post!)
The whole big company AI thing is a little boring to me. It's like worrying about what new sandwich McDonald's is developing. Should the government make laws that limit the amount of poison that can be put in the sandwich? I know a McDonald's sandwich will be eaten by a lot of people but I also know I won't be one of them. Wake me up when I can make my own sandwich. Here is my post on locally hosted LLM research.. Seriously folks, nothing wholesome will happen until this technology is controlled by real people and not billionaires. And if you think that (non-billionaire) you control this technology now, then that's probably the scariest AI related error you should be working on.
What's my doom scenario? Whenever I talk to a robot friend and it looks something up on the internet, I feel a sense of dread. Not that the world will end because of Skynet AI risk, but because it is conclusive proof that the www is now even more fucked up than it was before. The public aspect of my web pages has been an utter failure through no fault of mine. (My website is still useful to me.) Erosion of the conceptual underpinnings of the www has taken its toll. Is the www's entire shaky foundation ready to fold completely? People burned the Library of Alexandria too. These things happen.
He seems to be worried about "disturbed loners", especially those of us with leet STEM skillz. I hope he's thinking about the non-loners who are also going to be more disturbed (than me) once they don't have an income. I think I'm dealing with it pretty well!
I find his whole discussion about molecular biology (leading to bioterrorism) as typical outsider cluelessness. Having worked in the trenches of real molecular biology battles, the real world in biotech is very different than the impression a Michael Chricton novel imparts. Remember that our recent Plague was so devastating mostly from a lack of the most basic industrial engineering fundamentals (e.g.). To say, "...mRNA vaccines which can be designed to respond to a particular virus or variant..." is fatuous. Can not an attenuated virus vaccine do the same? Perhaps model your answer on the vaccine that tamed the 1958 flu outbreak — a vaccine that was developed quicker than Moderna's mRNA C19 vaccine.
He goes on to say, "The reason I haven’t focused on cyber as much as biology is that (1) cyberattacks are much less likely to kill people, certainly not at the scale of biological attacks, and (2) the offense-defense balance may be more tractable in cyber..."
Okie dokie. If you use a clever AI attack to convince a bunch of people to go and murder all their neighbors, well, that tends to be very nasty.
Although he admits there are other dangers, he says, "...biology is currently the most serious vector of attack,..." To which I say, go ahead and use that magical AI to improve my health one tiny bit and then I'll think about taking bioterrorism claims more seriously than having my fashy neighbors lynch me for my heretical views on the pseudoepigraphical nature of the PR materials of a famous Turkish wellness guru. That's a real threat that I actually must worry about.
He has a whole paragraph on how AI companies are themselves a risky entity, and it is good he sees the irony. As he pointed out that AI companies could subtly brainwash their user base, I was wondering if this article might have been written by an AI trying to brainwash us. It is by the article's own logic that this kind of article is the most likey vector of such an attack at this time.
"The world needs to understand the dark potential of powerful AI in the hands of autocrats, and to recognize that certain uses of AI amount to an attempt to permanently steal their freedom and impose a totalitarian state from which they can’t escape. I would even argue that in some cases, large-scale surveillance with powerful AI, mass propaganda with powerful AI, and certain types of offensive uses of fully autonomous weapons should be considered crimes against humanity."
This is true enough. But does he not see that he is the autocrat here? I'm sure he does.
"This could also lead to a world of “geographic inequality,” where an increasing fraction of the world’s wealth is concentrated in Silicon Valley, which becomes its own economy running at a different speed than the rest of the world and leaving it behind."
Dude, this happened 20 years ago. Blame Steve Jobs specifically. This guy needs to read my post Companies Repudiating Their Own Worthless Products. And it is geniuses like this who are dreaming up definitions to the I in AI.
He keeps talking about the need to prevent autocracy. Do billionaires not watch any news? Maybe he can't imagine it from the perspective of a little person who is not a billionaire.
The working title of this post was: A Memo From A Liege Lord To His Serfs.
"...companies should think about how to take care of their employees."
Lol. He forgot to add, "...if they are shareholders to whom they have a fiduciary duty." Where does this guy think he is? The 1950s?
"...while all the above private actions can be helpful, ultimately a macroeconomic problem this large will require government intervention."
What he meant to say instead of "government intervention" is "guillotines".
"We simply need to break the link between the generation of economic value and self-worth and meaning."
Ya, good luck with that. Even the world's leading practitioners at doing just this (ahem) are not going to get through to normal people until it is way too late. And those of us who have severed the tie are still in danger of starving.
Mighty big of him to be stepping in to make decisions about how to downsize enterprise salarymen. Cool. If I had a small cadre of geniuses at my disposal (Anthropic's headcount of 2500 should be plenty), my goal would be to use them to create a fully open source AI system and put Anthropic out of business. Or, won't that be possible? I may be a knucklehead but your bots are geniuses, right?
Your move, Dario.