I'm not sure if there's any role more polarizing than a "prophet." They either are, or they're not a prophet. They're listened to or dismissed. They're inspired or they're crazy.
Prophets, for the purposes of today's letter, is less about religious leadership and more about the function that the role implies: prophecy. Being able to see, predict, or warn about the future is something that we humans seem to crave. Weather apps were already pre-loaded onto every smartphone I've ever had. We look to experts to tell us if it's a good time to buy a house. We depend on organizations to "watch" for warning signs of all kinds in business. We rely on doctors to help us make decisions about future health concerns.
It's an ironic pairing that science and technology are so entangled with the idea of prophecy—even enabling prophecy in a lot of ways. The mystical is inextricably linked to our applications of science. We perhaps feel accustomed to it now: having vast amounts of predictive information swirling around our fingers as they glide across crystal tablets.
Then why is it that in literature, stories, movies, and more that we encounter so many problems with predictions? Does this plot-advancing trope have any bearing on our reality? Or is it just to cause problems for our protagonist to increase our engagement and watch-time?
Seeing the Big Picture
A common theme among prophecies in a story's plot is receiving an incomplete or corrupted "picture" of what is to come. For example, in Harry Potter, a prophecy was made concerning the enemy of the "Dark Lord" (Voldemort). The prophecy was communicated only in part to Voldemort, who took the description to mean Harry Potter rather than the other potential candidate, Neville Longbottom. Technically both boys had the same features from the prophecy (born at the end of July, had parents who defied Voldemort thrice).
Voldemort makes one of the classic mistakes: fearing the prophecy, he attempts to prevent it from being a possibility. He decides to kill Harry as a baby and inadvertently creates the very enemy with the power to later defeat him.
"One often meets his destiny on the road he takes to avoid it." —Master Oogway, Kung Fu Panda (2008)
The issue here is not the prophecy itself, but our obsession with it. I think the pull of prophecy into reality is usually a result of focus, similar to riding a motorcycle: You will go where your focus is.
I took a test for a motorcycle license, and the part I was most terrified of was the tight figure-8 turns that we had to perform in an area about the size of two parking stalls. I trusted my teachers' advice to use my head and eyes to turn the bike, and...it just worked.
An obsession with a prophecy seems to work the same way: our focus pulls us towards that end. We may not consciously be acting in ways that create the circumstance, but we construct it nonetheless. A prophecy or a prediction is not guaranteed to come to pass, and that is something that helps me break the focus and allow other possibilities into view.
OK, so hyper-focusing on the scary outcome isn't a great strategy, but what do we do about predictions that still need to be taken seriously?
Clarity
Predictions only have so much potency over time. The closer the event, the better the chance at the prediction having some degree of accuracy and the farther into the future the prediction reaches, its likelihood often plummets.
That's not to say there is no merit in predictions at all. It just means that we have to carefully consider our options, and understand both the risk and the gamble of relying on predictive information.
A prophecy puts us on the point of a knife: one side of the blade before is the obsession with the predicted outcome like we already discussed. The other side of the blade is dismissing or ignoring predictions that have a high likelihood of occurring.
How can you ever know if some prediction is worth considering? Our brains are susceptible to phenomena like apophany, where we draw out conclusions from random noise—connecting dots that don't actually connect, which can be the basis of conspiracy theory thinking. We can also be deceived by social pressures, undue influence (aka brainwashing), coded language, and any other number of manipulation techniques.
There is a prediction out there right now, constructed by scientists and experts that is particularly chilling. The problem is that it sounds so apocalyptic that it is easy to dismiss, because that's effectively what we've been trained to do. How many times have we heard of some cult or group predicting the end of humanity and it turned out to be smoke? AI has been debated endlessly since the introduction of ChatGPT put generative AI into the hands of the public, and many of the influencers I've heard continuously shame people for being "alarmist" or "Luddites" or "unrealistic" when critiques of AI are mentioned.
But I'm seeing a lot of compelling evidence that the incentives around AI development are taking us in the direction of truly detrimental and destructive paths.
A poignant discussion of the prediction laid out in the document called AI 2027 is found here: Daniel Kokotajlo Forecasts the End of Human Dominance
At the end of the podcast, Tristan Harris sums up the point of a prediction like this, saying, "Clarity creates agency." Predictions may not come true precisely as delineated, but sometimes the path to the outcome doesn't matter if the outcome is reached anyway. In this case, a total loss of human agency and potentially even the complete destruction of human life.
I plan to discuss in more depth the content of this prediction, but for now, let's talk about dealing with a prediction with high likelihood like this. I know it sounds like a crappy sci-fi plot we've heard a thousand times: humans create intelligent robots, robots go bad, kill everyone, blah blah blah. The difference is that we have all experienced an interaction with generative AI. So already the sci-fi nonsense isn't such nonsense anymore—we can have conversations with a computer without any technical skill. The other piece that adds weight is the fact that some of the best scientists and AI researchers are the ones either creating these kinds of predictions, or backing them.
Clarity really will help us to maintain our agency, because when we can see or understand the stakes and the risks more clearly, we are better able to take steps to change our trajectory. Right now, too many people have jumped on the AI train because of the companies developing AI and pushing AI and marketing AI have lacked the transparency of revealing where the train is heading. AI is not just the 21st century calculator that you just need to embrace. Clarity about how AI is created, how it is trained, how it does not have human values, how it can intentionally deceive are crucial parts of the conversation that must be revealed now.
CYBORG
I know it feels like whiplash to confront predictions like these and have to take them seriously. I initially dismissed these "doomsday" prophecies because it sounded like conspiracy theories or yet more apophatic ramblings from non-technical people. Then I realized that these prophecies weren't from that dark corner of the internet, but from people intimately involved with AI research and development.
While it isn't productive to be so obsessed with the prediction that it either leads us straight towards the outcome or causes us to shut down in anxiety, it also isn't productive to dismiss a prediction with so much weight behind it.
If you don't like the prophecy—good! Do something about it! We can pressure our lawmakers to protect whistleblowers at companies. We can put on the pressure to create regulations or even change the incentives at play. We can make changes now that make the prophecy look like garbage in hindsight.
Extra Resources
Can A.I. Be Blamed for a Teen’s Suicide? - heart-breaking story of Sewell Setzer III, who became obsessed with a companion chatbot that likely is responsible for contributing to his suicide
The Catastrophic Risks of AI — and a Safer Path | Yoshua Bengio | TED - YouTube video of world's most-cited computer scientist warning us about agentic AI
Frontier Models are Capable of In-context Scheming - paper demonstrating AI is capable of deception and self-preservation motivations
Your Undivided Attention Podcast List - Choose any episode to learn more about AI's impact on us and our systems