The pseudo-religious origins of the AI bubble

Transhumanism, not technology, is fuelling much of the current AI optimism

Photo by Stephen Leonardi. Taken at Burning Man.

For a detailed substantive explanation of why I think AI is almost certainly in a bubble, see my previous article, “Is the AI industry in a bubble?”.

Mass irrationality is a common feature of investment bubbles. The current AI bubble is unusual in that the mass irrationality largely stems from a pseudo-religious belief system which pervades a community of technical experts. The AI bubble traces back to ideas about transhumanism, the Singularity, and artificial general intelligence that can be found in decades-old sources like Ray Kurzweil’s 1999 book The Age of Spiritual Machines. In that book, Kurzweil predicted that the first AI would achieve human-level intelligence in 2029. I think it’s not a coincidence that many contemporary predictions have placed the date for the first artificial general intelligence around 2030. The allure of the old prophecy is hard to resist!

My first exposure to the ideas of Kurzweil and other transhumanists was around 2006. I have been following this ideological community for around twenty years. Like conventional religion, transhumanism is a complicated mix of good and bad that’s hard to untangle. In his “Letter from a Birmingham Jail”, Martin Luther King Jr. wrote about how his Christian faith inspired him to be an “extremist for love”, even as many white Christians cited Scripture as justification for the subjugation of their Black neighbours. That’s the contradictions of religion in a nutshell. Transhumanism, as a modern techno-religion, is no less contradictory.

There is something beautiful, intuitive, and probably true in the idea that human beings are an evolving thing, and that if technology can help us evolve further, that is good. Yet, in practice, much of what goes under the name “transhumanism” ranges from amusing but harmless eccentricity to complete madness. One example: political libertarianism, which most people rightly see as a forlorn concept, has a hugely outsized influence in transhumanist communities. There is no straightforward logical connection between libertarianism and transhumanism. This is probably just a result of the demographics and personality of the sort of people who are drawn to transhumanist communities.

Since the release of ChatGPT in November 2022, what — for the previous sixteen years that I’d been in the loop — was a tiny ideology, approximately on the scale of My Little Pony fan fiction, has exploded into the mainstream. This is such a strange feeling for me. All of a sudden, obscure transhumanist bloggers I read as a teenager are getting interviewed by the likes of Ezra Klein. A Pulitzer Prize-winning journalist wrote an opinion column in the New York Times endorsing Kamala Harris for president because a Harris administration would be… better at preparing the United States for artificial general intelligence? What? This is so bizarre.

I have a theory that the irrational exuberance, and irrational fear, that we’re currently seeing around AI is in large part a symptom of the mainstream world digesting transhumanist ideas for the first time. That intuitive, beautiful core part of transhumanism — the abstract thing, not the thing that exists in practice — is so mesmerizing, so alluring, it is difficult not to be radicalized by it. As with conventional religion, or as with much of political ideology, there is truth and falsity, sanity and madness, wrapped up into one thing, and it takes time, temperance, and quiet contemplation to work out what’s good and what’s bad.

ChatGPT is incredibly impressive and fascinating, not to mention perplexing from the perspective of cognitive science and philosophy of mind. When something violates your expectations in that way, and the old rules you thought the world operated by no longer seem to hold true, for a time it seems as if anything might be possible. If you showed a modern smartphone to a typical pre-industrial European, you could probably convince them it can turn lead to gold and make its user immortal. When your mind is sufficiently blown, you don’t know where to draw the line between what’s possible and what’s impossible. So, ChatGPT opened the door to rethinking everything we thought we knew about AI and the mind.

When that door opened, decades of accumulated transhumanist belief was waiting in the wings to rush in and fill the void. The void was more in the mainstream than in the AI industry itself. By the time ChatGPT arrived, the minds of people who work in the AI industry had already been percolating with transhumanist ideas for decades. That’s the other unusual thing about this: in other areas of technical expertise, like biology or manufacturing, it’s not typical for the field to have its own bespoke pseudo-religion. AI engineers in Silicon Valley dream of the Machine God and fear the Machine Devil. It would be strange if engineers at Boeing and Airbus had their own microreligion about, I don’t know, humanity’s destiny of building colossal flying cities. The credibility of the transhumanist ideas that have recently entered the mainstream is riding a lot on the fact that many people with technical competence in AI happen to believe them.

Silicon Valley believes it’s building God and fears it may accidentally build the Devil, but Wall Street doesn’t seem bought into this. Investors have watered down the transhumanism of the technical people in the AI industry. But when you’re watering down God and the Devil, you still end up with something gargantuan in its ramifications. Investors seem to anticipate economic transformation — automation of jobs on a massive scale, or at least parts of jobs. The starting point that everyone is walking back from — the Singularity — is so extreme that to be conservative in your predictions for AI is to say that it won’t be transformative for another 5 or 10 years. Investors are in a tough spot where the technical experts they’re relying on to form their views are in the thrall of a millennialist worldview. I assume biotech and SaaS investors don’t have that problem.

The AI industry, and the investment world that’s funding it, is in need of some sober second thought. Maybe some of these transhumanist ideas will come true eventually. Maybe eventually some form of AI will bring about mass automation and economic transformation — I hope so. But for all we know, that will happen in 2130 rather than 2030. A tantalizing “what if?” should not become a “this will happen soon” without some rigorous, empirical justification between the two. In lieu of rigorous justification, we see things like the famous METR graph of AI getting exponentially better at long-term planning, which turns out to be complete nonsense. What’s worse, despite understanding and in fact disclosing some of the key limitations of their work, METR has still promoted the famous graph as supporting a conclusion — that AI is rapidly getting better at long-term planning in a general sense — that they themselves have previously said it does not support.

Blogs and other self-published writing have an important place in the world. But this is the sort of area where the rigour of academic journals is sorely needed. Peer review is a form of institutionalized second thought. Peer review has its flaws, but do you know what really has some huge flaws? No peer view. With things like the METR graph, the sort of errors that reviewers would catch before publication are now written up months after the graph has shaped the thinking of millions of people. Who knows if even a tenth of the people who have seen the graph will ever see a competent refutation of it.

Even worse than the METR graph is AI 2027, which barely tries to hide the fact that it’s nonsense on stilts. What is the AI 2027 forecast ultimately based on? The gut intuitions of its authors, who are all deep believers in transhumanist or transhumanist-adjacent ideology. Do the authors admit the forecast is ultimately just based on their gut intuitions? Yes, if you click through to the right page and scroll down the right section. Why is this worthy of anyone’s attention? The virality of AI 2027 seems to stem from the fact that it misleads readers about what it is. It masquerades as some kind of scientific model. (A model which, by the way, uses completely botched math.) What it really is, at bottom, is a few random guys who said, “I don’t know, I just have a feeling.” I wish that a flamethrower of truth would eviscerate this kind of garbage. This is embarrassingly, unacceptably, egregiously ridiculous. This is pure science fiction playing dress up as science. Enough! Stop!

I am not cherry-picking the worst two examples I can find. These are, in fact, the two most commonly cited sources of “evidence” for a near-term explosion of AI capabilities. I have tried to find something better. Everything I can find is about on this level. This is religious belief in the pejorative sense. It is belief where evidence — apologetics — follows belief, rather than the other way around. A few years ago, when I started looking into the reasoning and evidence behind predictions that artificial general intelligence will be created in the near future, I assumed there had to be some kind of solid, respectable thinking behind it. Now I have seen that’s not close to true. It’s so much worse than I thought.

I am not anti-transhumanist or anti-AI. In some important sense, I am transhumanist. If science and technology can continue for the next 1,000 years in the way they have for the last 200, at some point, probably something radical, beautiful, fascinating, and even divine will happen. But I think it’s wrong to believe this will happen within the next 5 years just on the basis of gut intuition or some prophecy from 1999.

I am very pro-AI in the sense that I regret how much capital has been poured into large language models, which seem to be a dead end at this point. I wonder what would have happened if that capital had instead gone into fundamental AI research. Perversely, the race to build incrementally better LLMs seems to have sucked resources away from fundamental research. I would love to live in a world populated by competent AI robots, like self-driving cars and the humanoid robot butlers that are faddish but forlorn. This won’t be possible with the current paradigm of AI tech. We need continual learning, better long-term planning, better generalization, much better computer vision, and the ability to learn from much less data. For that, we need fundamentally new ideas that either no one has thought of yet or that are currently obscure. I would love to see more investment go into finding these ideas. I want my self-driving car! I want my robot butler!

The AI bubble is perhaps not so different from other bubbles in some important ways. Crypto, as a financial fad, is almost as strange as AI. All these years later and actual practical applications of crypto are very few and far between. Yet the coin valuations remain eye-watering: $1.4 trillion for Bitcoin, $250 billion for Ethereum. Like AI, crypto does intersect with ideology: an amorphous political and economic philosophy that is skeptical of fiat currency and central banks, and sees crypto as some kind of populist, revolutionary, utopian rebellion against the system. But AI still has to take the cake as the strangest bubble of all time. $300 billion in funding fuelled largely by a pseudo-religion preoccupied with machine gods and devils is just so bizarre. It feels too strange to be true.