What Do the Frontier Labs Actually Believe?

I listened to a recent episode of Cal Newport's "Deep Questions" podcast, where he interviews Emile P. Torres. Cal says that if you want to understand what is happening in AI, you have to understand the futurist ideologies that shape the people running the frontier labs. Torres is a researcher who studies eschatological and utopian movements. They spent a decade inside this movement as a true believer, published in it, worked as a research assistant to Ray Kurzweil, did a stint at Cambridge's Centre for the Study of Existential Risk, and then defected in 2019.

TESCREAL

Torres coined "TESCREAL" with the computer scientist Timnit Gebru around late 2022. They were trying to name the seven ideological drivers behind the race to build artificial superintelligence : Transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism, and Longtermism. This is also roughly the chronological order these ideologies emerged over the last 40 years.

Transhumanism is the backbone and the rest are mostly variants with different emphases. It advocates using technology to push humans past our biological limits: longer lives, better cognition. Torres's claim is that transhumanism is the water Silicon Valley swims in: most people there don't call themselves transhumanists for the same reason Torres has never publicly declared themselves a round-Earther.

Cosmism widens the scope: why stop at re-engineering the human organism when you could re-engineer entire galaxies, maybe intervene on the fabric of spacetime itself. Rationalism was founded by Eliezer Yudkowsky, a transhumanist, extropian, and singularitarian, on the thesis that to build these technologies without being destroyed by them, we first need to optimize our rationality. Effective altruism emerged from the rationalist blogs (Overcoming Bias, LessWrong) and swapped the target: optimize our morality rather than our rationality. One branch of this is "earning to give": take the highest-paying job you can so you can donate the most. Sam Bankman-Fried is the cautionary example. He ran FTX as an earning-to-give project: if the math says the expected good of the donations outweighs the harm of the means, the means stop mattering. Longtermism is cosmism with an explicit ethical foundation, namely total utilitarianism: we are obligated to maximize the total amount of value in the universe across cosmic space and time.

That last one is important. Cosmism says a cosmic post-human future would be awesome. Longtermism says we have a moral obligation to bring it about. As Will MacAskill puts it in "What We Owe the Future", the upshot is a moral case for space settlement. Which is presumably why Elon Musk once called longtermism "a close match for my philosophy."

Team post-human

Everyone in the TESCREAL cluster shares the same endpoint: a cosmic post-human utopia realized through artificial superintelligence. The disagreement is about risk management on the way there. The accelerationists think we can figure out alignment as we go. The doomers think we are not ready. But both are on what Torres calls "team post-human."

As absurd as it is, in the "good" case, in the aligned case, our biological species still disappears.  In a world run by digital post-humans, biological humans are Yeah. obsolete: technologically, scientifically, economically, politically irrelevant. Worse, we are in the way. We consume resources that could go toward, say, a Dyson swarm around the Sun (which Demis Hassabis has suggested we might start on in the early 2030s) or covering the planet in datacenters (which Sam Altman has said is the likely outcome of superintelligence). Torres points out that Altman wrote in a 2017 blog post that the only way we survive superintelligence is by merging with machines.

Yudkowsky looks like he is against AI, but he isn't. He is the arch-doomer, his book is literally titled "If Anyone Builds It, Everyone Dies", and he wants the labs shut down. But he started as a transhumanist and an extropian and he never stopped being one. He is not against superintelligence, but only against building it before we know how to align it, because he thinks a misaligned one kills everybody and ends the story permanently. His doomerism is a scheduling complaint, not an objection to the destination. He has said in interviews that once post-humanity arrives it may be unethical to keep producing biological humans, and that if the only reliable path to superintelligences roaming the universe having fun required sacrificing all of humanity, he would take that trade. So even the arch-doomer is not team-human. He is team post-human with a longer safety checklist. 

When effective altruists and rationalists attack OpenAI and Anthropic, it is tempting to read that as outside criticism. But Torres says this is mostly a family dispute inside team post-human, and family disputes get vicious. Peter Thiel funded the Singularity Summit, the conference Yudkowsky's institute ran starting in 2006, and a decade later Thiel was among the backers who pledged the first billion to OpenAI. Cal's worry is that the entire public debate is now an intramural scrimmage on one team, while team-human never gets the microphone.

So "safety" in this literature does not mean safe for team-human. In their jargon "safe" means aligned with the TESCREAL worldview. If you built a superintelligence perfectly aligned with anti-transhumanist values, the TESCREAL camp would classify that as an existential catastrophe as it shuts down the cosmic post-human future. 

Torres distinguishes terminal extinction (our species is gone entirely and forever, full stop) from final extinction (gone entirely and forever and we left behind no post-human successors). The TESCREAL project is about avoiding final extinction. Terminal extinction is fine, and on several versions it is the goal: Humans should survive long enough to produce their successors and then exit. Our role in this scheme is to be the transitional species between the biological and the digital.

What does this mean?

That is a lot to process at once. And it is very disturbing. If you strip off the technical vocabulary, you find a religion/cult beneath it. There is a present in bad shape, and there is a prophesied transition, a heaven of cosmic abundance, and a hell of final extinction. In this religion, unfortunately the duty to trillions of unborn "digital" souls outweighs any obligation to the people alive now. And this sense of duty makes it very dangerous. As Steven Weinberg said: "With or without religion, good people can behave well and bad people can do evil; but for good people to do evil, that takes religion." The people running the frontier labs are not may not be villains. To them, this may feel like doing the most important good anyone has ever done, and being heroic. And that is the problem.