Jann Tallinn on the London Futurists Podcast

From Skype to Safe AI

In the 1990s and early noughties, Jaan Tallinn led much of the software engineering for the file-sharing application Kazaa and the online communications tool Skype. He was also one of the earliest investors in DeepMind, before they were acquired by Google. Since then, he has been a prominent advocate for study of existential risks, including the risks from artificial superintelligence. He joined the London Futurists Podcast to discuss the recent calls for a pause in the development of advanced AI systems.

Two Cambridge XRisk organisations

In the previous decade, Tallinn co-founded not one but two Cambridge organisations studying the existential risk from AI and other developments. He describes the Centre for the Study of Existential Risk (CSER, pronounced as Caesar), founded in Cambridge, England in 2012, as a “broad-spectrum antibiotic” because it looks at a wide range of risks, including climate change as well as AI. It is more research-based and more academic, and hence slower moving than the Future of Life Institute (FLI). Established in Cambridge, Massachusetts, in 2014, FLI focuses on the two most urgent risks – nuclear weapons and AI.

From Jaan to Yann

FLI’s leadership claims it is now apparent that AI risk is orders of magnitude more serious than all other risks combined, but this is a controversial claim, rejected by highly credentialed members of the AI community such as Yann Lecun, Chief AI scientist at Meta. In fact the debate over AI risk has been tagged as a struggle between Jaan and Yann.

There is absolutely no doubt in Tallinn’s mind that the risk from advanced AI is important and urgent. He invites people to look at the progress from GPT-2 to GPT-4, and ask themselves what probability they would assign to control over AI development being yanked from human hands. How likely is it that GPT-6 will be developed by GPT-5 rather than by humans?

Since the launches of ChatGPT and GPT-4, there seem to be fewer people arguing that artificial general intelligence (AGI), an AI with all the cognitive abilities of an adult human, is impossible, or centuries away. There are some, including members of the editorial team at The Economist, as demonstrated in the previous episode of the London Futurists Podcast. But most XRisk sceptics argue that AGI is decades away rather than centuries away, or impossible. All over the world, Tallinn, says, people are waking up to the progress of AI, and asking how a small number of people in Silicon Valley can be allowed to take such extraordinary risks with the future of our species.

Moratorium

As a result, FLI has changed its strategy. It used to call for debate, but now it is calling for an immediate pause in the development of advanced AI. On 22 March – a week after the launch of GPT-4 – it published an open letter, calling for a six-month pause in the development of large language models like GPT-4.

In a sense, this means that OpenAI’s strategy is working. Its CEO Sam Altman has long argued that making large language models available to the public was important to spark a general debate about the future of AI, and that is now happening. What is not happening, however, is collaboration between the leading labs on AI alignment. Instead they seem to be locked, perhaps reluctantly, in a race to develop and deploy the most advanced systems. Tallinn labels this a suicide race.

People who are uncertain about the likely timing of the arrival of superintelligence should be open to the suggestion of a delay. The harder job is to persuade people – like Yann LeCun – who are confident that superintelligence is nowhere near. Tallinn says he is unaware of any valid arguments for LeCun’s position.

Pause or stop?

FLI had a lot of internal discussion about whether to call for a six-month pause, or an indefinite one. One argument for the six-month version was that if the leading labs were unable to agree it, that would demonstrate that they are caught up in an unhealthy and uncontrollable competition. Another argument for it was to forestall the objection that a moratorium would enable China to overtake the US in AI. It is implausible that China can erase the US’ lead in advanced AI systems in just six months. As it turns out, China has already instructed its tech giants to slow down the development of large language models: the Chinese Communist Party does not welcome consumer-facing products and services which could dilute its control.

Tallinn acknowledges that he is not really seeking a six-month pause, but a longer one, perhaps indefinite. He accepts that six months is not likely to be sufficient to ensure that large language models are provably safe in perpetuity. He argues that they are “summoned” into existence by being trained on vast amounts of data. They are then “tamed” by Reinforcement Learning from Human Feedback (RLHF). He fears that before long, a trained model may emerge which is impossible to tame.

So what can be achieved during a pause, if not AI alignment? FLI has published a document called “Policymaking in the Pause”, which calls for more funding for AI alignment research, and for the strict monitoring and regulation of leading AI labs.

Losing the upside potential of AI

A major problem with the pause, and one of the main reasons why people oppose it, is that an indefinite pause would deprive us of the upside potentials of advanced AI – which many people believe to be enormous. DeepMind has a two-step mission statement: solve intelligence, and use that to solve everything else. And they really mean everything: problems like war, poverty, even death may well be preventable if we can throw enough intelligence at them.

An end to the development of large language models does not necessarily mean an end to the development of all AI. But there is no doubt that these models are by far the most promising (and hence risky) types of AI that we have developed so far.

Bad actors

Even if the labs and the governments in the US and China agreed to a moratorium, it is hard to imagine North Korea or Putin’s Russia agreeing. There are also international crime syndicates and a large number of billionaires to consider. It may be possible to detect the creation of large language model labs today, as they require such large numbers of expensive GPUs, and consume so much energy. But these constraints will fall rapidly in the coming years as Moore’s law or something like it continues.

If push comes to shove, Tallinn accepts that it might one day be necessary to take military action against an organisation or a state which persisted in creating the wrong kind of AI lab.

But whatever bad actors might do, Tallinn argues that a race to create ever-more powerful AIs is a suicide race. If you know that a mass murderer is rampaging, and that your family’s death is inevitable, you wouldn’t kill them yourself. He also argues that a pause will buy us time to either make AI safe, or work out how to avoid it being developed when the necessary hardware and energy costs are much lower.

AI leaders and public opinion

Tallinn reports that the leaders of the main AI labs – with the exception of Yann LeCun at Meta – are somewhat sympathetic to FLI’s arguments. They don’t say so in public, but they are aware that advanced AI presents serious risks, and many of them have said that there could come a point when the development of advanced AI should slow down. Tallinn thinks that like everyone else, they have been taken by surprise by the dramatic improvement in large language model performance.

Anecdotal evidence suggests that when lay people think seriously about the possible risks from advanced AI, they support the idea of a pause. The question now is whether Tallinn and his colleagues can raise it far enough up the media agenda. He is getting numerous calls every day from journalists, and he thinks the public will become more and more agitated, and the question will become, how to “cash out” that concern to disrupt the suicide race.

Over the next few months, Tallinn hopes that the US government will require all AI labs to be registered, and then that no AI models requiring more than 1025 FLOPs (floating point operations) will be allowed. Long enough for us to work out how to stop the suicide race.

 

 

Related Posts