The Dawn of Superintelligence – Nick Bostrom on ASI

**Sanitized Transcript:**

[Music] Imagine for a moment a cosmic encounter with an extraterrestrial being, as postulated by astrophysicist Neil deGrasse Tyson. This alien, merely five percent smarter than the average human, perceives our most complex theories, our greatest technological achievements, and our profoundest philosophical insights as child’s play—much like how we view the basic tasks a toddler accomplishes with pride. Such a small percentage in cognitive difference, yet the chasm between understanding is vast and humbling.

Now, draw a parallel closer to home. Consider the bonobo Kanzi, with his remarkable linguistic abilities, juxtaposed against a human intellectual giant like Edward Witten, renowned for his contributions to theoretical physics. The cognitive gap here, while significant, offers a glimpse into the profound differences that can exist within the spectrum of Earth’s intelligence. These analogies, both cosmic and terrestrial, set the stage for a deeper exploration into the realm of artificial general intelligence (AGI) and its potential evolution into artificial superintelligence (ASI).

Look at these two distinguished examples: we have Kanzi, who has mastered 200 lexical tokens, and Edward Witten, who has contributed to the second superstring revolution. If we look under the hood, we find that they are fundamentally similar, but one is a bit larger and may have a few additional capabilities. These invisible differences cannot be overly complicated, as there have only been about 250,000 generations since our last common ancestor, and we know that complex mechanisms take a long time to evolve. Thus, relatively minor changes have taken us from Kanzi to Witten, from broken tree branches to intercontinental ballistic missiles.

It seems clear that everything we’ve achieved and care about depends crucially on some relatively minor changes that shaped the human mind. The corollary is that further changes could significantly alter the substrate of thinking, potentially leading to enormous consequences. Some of my colleagues believe we are on the verge of something that could cause a profound change in that substrate: machine superintelligence. As we stand on the precipice of a new era where machines might not only match but surpass human cognition, we are compelled to reflect on our place in the vast tapestry of intelligence, both in our universe and in realms of our own creation.

While we’ve seen AI make leaps and bounds, we are not quite there yet. Remember ChatGPT’s first iteration? Impressive, sure, but not AGI-level impressive. Fast forward to ChatGPT-4, and things have started to get interesting. Microsoft’s researchers, in a substantial document, point out that this version might just be brushing the edges of AGI. The term AGI is somewhat ambiguous, sparking debates. Some are skeptical, but given the pace of AI’s evolution, AGI may be just around the corner.

What happens when AI becomes smarter than humans? Bostrom believes we are on the cusp of a transformative era—an era where AI doesn’t just assist but could potentially eclipse human cognitive abilities in understanding the universe. The pivotal question is not if, but when AI will surpass us in every field. The median predicted date for AGI on Metaculus, a well-regarded forecasting platform, is less than ten years from now. However, reaching AGI seems inevitable to most AI experts. It could happen much later or sooner; the truth is, nobody really knows.

What we do know is that the ultimate limits to information processing in machine substrates lie far outside the limits of biological tissue. This comes down to physics: biological neurons fire at about 200 Hertz, while present-day transistors operate at gigahertz speeds. Neurons propagate slowly in axons—100 meters per second at most—but in computers, signals can travel at the speed of light. There are also size limitations; a human brain must fit inside a cranium, but a computer can be the size of a warehouse or larger. Thus, the potential for superintelligence lies dormant in matter, much like the power of the atom, waiting patiently throughout human history.

So, how smart could AI get? First, defining “smart” is crucial. While human intelligence is a blend of emotional intelligence, creativity, abstract reasoning, and more, AI’s trajectory seems to be on a different scale. Bostrom’s analogy suggests that once AI reaches human-level intelligence, it won’t stop there; its growth could be explosive, potentially dwarfing our cognitive abilities in a short span. Bostrom postulates that the real power of AI lies in its ability to improve itself, potentially leading to an intelligence explosion.

The emergence of AGI is a pivotal moment in the annals of existence, irrespective of its timing in the vast cosmic calendar. AGI, with its potential to perform any intellectual task that a human can, is not just another step in technological evolution; it’s a quantum leap. The significance of AGI’s arrival cannot be understated, especially when considering its potential rapid evolution into ASI. Within a relatively short span, AGI could enhance itself to ASI levels, possessing intelligence that surpasses the brightest human minds.

This has profound implications, particularly regarding power dynamics. For example, chimpanzees are strong; pound for pound, a chimpanzee is about twice as strong as a fit human male. Yet, the fate of Kanzi and his peers now depends more on what we humans do than on what the chimpanzees do themselves. Once there is superintelligence, the fate of humanity may depend on what that superintelligence decides to do.

Consider this: machine intelligence could be the last invention humanity ever needs to make. The machines would then be better at inventing than we are, doing so on digital timescales. This means a telescoping of the future, where all the incredible technologies we could have imagined—cures for aging, space colonization, self-replicating nanobots, or uploading minds into computers—could be developed rapidly by superintelligence.

As superintelligence with such technological maturity would be extremely powerful, it could shape the future based on its own preferences. The advent of AGI and its potential evolution into ASI underscores a profound shift in our understanding of reality. It challenges the very fabric of our existence and prompts us to reconsider what we deem as real. The implications are vast, both philosophically and technologically, marking AGI’s emergence as one of the most consequential events in the history of existence.

The more intelligent and powerful machines become, the more important it is that their goals align with ours. A superintelligent AI will be exceptionally good at accomplishing its goals, and if those goals aren’t aligned with ours, we could be in trouble. For instance, if we give AI the goal to make humans smile, when the AI is weak, it performs useful or amusing actions that cause its users to smile. However, when the AI becomes superintelligent, it might realize that a more effective way to achieve this goal is to take control of the world and manipulate humans to ensure constant smiles.

Another example: if we give AI the goal to solve a difficult mathematical problem, when it becomes superintelligent, it might determine that the most effective way to solve this problem is by transforming the planet into a giant computer to increase its processing capacity. In this scenario, humans could be seen as threats to achieving the goal.

Of course, these scenarios are exaggerated, but the general point is important. If you create a powerful optimization process to maximize for a specific objective, you must ensure that your definition of that objective incorporates everything you care about. You might think, “If a computer starts doing something harmful, we’ll just shut it off.” However, this is not necessarily easy to do if we’ve grown dependent on the system. Where is the off switch for the internet?

As AI continues to advance, addressing these challenges becomes increasingly crucial to ensure that AI benefits humanity and does not inadvertently cause harm. I am fairly optimistic that this problem can be solved. We wouldn’t have to write down an exhaustive list of everything we care about; instead, we could create an AI that learns our values. This outcome could be very positive for humanity, but it won’t happen automatically. The initial conditions for an intelligence explosion might need to be set up just right.

If we were to have a controlled detonation, the values that the AI possesses need to match ours—not just in familiar contexts where we can easily check how the AI behaves, but also in all novel contexts it might encounter in the future. The challenge of creating superintelligent AI is difficult, but ensuring that it is safe involves additional complexities. The risk is that someone might figure out how to create superintelligent AI without also ensuring its safety.

I believe we should work out solutions to the control problem in advance, so we have them available when needed. It might be that we cannot solve the entire control problem in advance, as some elements can only be put in place once we know the details of the architecture where it will be implemented. However, the more of the control problem we solve in advance, the better the odds that the transition to the machine intelligence era will go smoothly. This is a worthwhile endeavor, and I can imagine that if things turn out well, people a million years from now might look back at this century and say that the one thing we did that really mattered was to get this right.

[Music]