Commentary: The AI panic needs some nuclear metaphors
To wrap our minds around the AI threat, there are lessons we can draw from Cold War-era thinking of nuclear weapons, says Andreas Kluth for Bloomberg Opinion.
Visitors stand near a sign for artificial intelligence at an AI robot booth at Security China, an exhibition on public safety and security, in Beijing, China, on Jun 7, 2023. (File photo: Reuters/Florence Lo)
New: You can now listen to articles.
This audio is generated by an AI tool.
Read a summary of this article on FAST.
Get bite-sized news via a new
cards interface. Give it a try.
Click here to return to FAST
Tap here to return to FAST
FAST
BERLIN: Much about the global AI panic – could it really “kill us all by the end of the decade”? – is genuinely new, and thus extra-scary. Then again, the spectre of human-engineered extinction is hardly novel; we have about eight decades of experience in thinking about it.
So what metaphors and lessons can we draw from the field of nuclear weapons as we wrap our minds around what lies ahead in artificial intelligence?
One of the oldest metaphors came from J Robert Oppenheimer, sometimes called “the father of the atomic bomb.” Writing in 1953, when Washington and Moscow were already able to annihilate one another while irradiating the world, he likened America and the Soviet Union “to two scorpions in a bottle, each capable of killing the other, but only at the risk of his own life”.
In game theory, which was then in its heyday, Oppenheimer’s analogy was often paired with the Prisoner’s Dilemma, which analyses the incentives for cooperation or betrayal between two people who are arrested and separately interrogated. Its upshot is that cooperation would be better for both, if they could trust each other; but since they can’t, defection may be the more rational choice.
Guess Word
Crack the word, one row at a time
Buzzword
Create words using the given letters
Mini Sudoku
Tiny puzzle, mighty brain teaser
Mini Crossword
Small grid, big challenge
Word Search
Spot as many words as you can
THE PRISONERS’ DILEMMA
These scorpions, or prisoners, bear a passing resemblance to contestants in today’s AI revolution, whether they’re companies (OpenAI and Anthropic, say) or the two AI superpowers, the United States and China.
Yes, it would be safer, for them and all of humanity, if, as Anthropic’s boss urges, the rivals start cooperating by “pacing the frontier” – voluntarily slowing down to give regulators time to catch up.
But each must assume the other could cheat, so it may be safer to keep racing, even if that leads to dead scorpions or miserable prisoners.
“Whoever wins AI wins,” US President Donald Trump has said, believing that the US is ahead by a nose and seeing no reason to slow down. Beijing, as long as it’s slightly behind, sees no way to ease off without losing. So both hurtle forward.
Believe it or not, Oppenheimer’s metaphor may nonetheless be too optimistic – or not pessimistic enough – to fit today’s context. When rational players go round after round (as opposed to just once) in the Prisoner’s Dilemma, they can reach stability. That’s what happened during the Cold War in the form of deterrence.
A GAME OF CHICKEN
But as leading AI scholars have been making clear all week, nothing about AI, which has advanced by leaps and bounds just in a few months, is stable. A more apt metaphor may be the one used by Thomas Schelling, a titan of nuclear strategy and winner of the Nobel Prize for his work in game theory.
As the Cold War advanced, Schelling imagined a situation in which two adversaries find themselves “in a room full of gasoline,” each holding matches. Neither can credibly threaten to strike his match, since the conflagration would kill them both.
But each can, through provocation or miscalculation, raise the risk of incineration by inadvertent escalation or accident.
Schelling was describing a situation of brinkmanship that is captured in game theory with Chicken, a scenario in which two drivers race toward each other. If one swerves, the other wins. If both swerve, neither wins, but both are safe. If neither swerves, they crash. Translated to today’s AI situation, crash could mean extinction.
Whereas the Prisoner’s Dilemma allows for rationality (even for scorpions, in this mixed metaphor), Chicken describes something more frightening: the loss of control and, in Schelling’s phrase, a “threat that leaves something to chance.”
That seems to describe the AI race, as companies in the US, and presumably China, are creating large-language models and other agents that autonomously manipulate or attack other systems and have already done so.
As if that weren’t scary enough, the industry has also entered the phase of “recursive self-improvement,” as AI agents autonomously build the next generation of even more powerful models. It’s as though, in a gasoline-filled room, the matches were not only making sparks all by themselves but also building new matches, and then sophisticated flame throwers.
THE SWORD OF DAMOCLES
This makes one more metaphor relevant: the sword of Damocles. It was used by US President John F Kennedy in a speech at the United Nations.
“Every inhabitant of this planet must contemplate the day when this planet may no longer be habitable,” he said; “every man, woman and child lives under a nuclear sword of Damocles, hanging by the slenderest of threads, capable of being cut at any moment by accident or miscalculation or by madness.”
That madness resembles visions I’ve had in the past week of AI agents going rogue. If Oppenheimer’s scorpions make us think of retaliation and Schelling’s gasoline-filled room evokes uncontrollable risk, the Damocles sword sums up the panic by positing extinction not for the individual players, but for the species.
The sword of Damocles allows for hope as well as despair. Hope, because in the years after Kennedy’s exhortation the world achieved a form of detente, with treaties for non-proliferation and arms control (an era that is now sadly turning into one of rearmament again).
Today, that would mean collective pacing, plus mutual verification, as well as coordinated regulation by Washington, Beijing and others.
CANNOT BE WON AND MUST NEVER BE FOUGHT
Better yet, the US and China could start playing a different game, called Stag Hunt. It imagines two hunters who must decide whether to cooperate and kill a stag that would feed both of them for a long time. If either defects, the other catches nothing, while the defector only gets a rabbit.
In today’s context, the stag represents the huge benefits that AI could bring to humanity, from curing its diseases to fixing its other problems. The upside of nuclear technology was mainly electricity generation; the bonus of superintelligence could be much larger.
What fills me with awe is the point at which the metaphors and game theory of the Cold War era break down.
The scorpions, prisoners, car racers and other characters in our thought exercises all represented humans with agency interacting with an exogenous technology, namely fission. But what happens when the AI models themselves become players?
AI doesn’t share, at least not intrinsically, our human instinct to avoid catastrophe – a difference euphemistically called the “alignment problem.” That one insight is enough to explain why it’s imperative, as the jargon has it, always to keep a human in the loop, and to go no faster than human speed.
This too is a lesson already learned by leaders during the late Cold War: An out-of-control AI contest, like a nuclear war, cannot be won, and must never be fought.
Source: Bloomberg/ch
Sign up for our newsletters
Get the CNA app
Stay updated with notifications for breaking news and our best stories
Get WhatsApp alerts
Join our channel for the top reads for the day on your preferred chat app
Get bite-sized news via a new
cards interface. Give it a try.
Click here to return to FAST
Tap here to return to FAST
FAST
