«Please, I’m hurt, can you help me?» The plea comes not from a human, but a synthetic female voice resonating from the MacBook in front of me. On the screen, a melancholic emoji peers back at me, amplifying the urgency in the electronic voice. My gaze travels downward, landing on a silver-painted mannequin torso. It’s marked by unsettlingly realistic «wounds» around the collarbone and rib area, illuminated from within by fluctuating blue and red lights. Evidently, the hollow torso harbors more than just emptiness. Hidden within its confines, lights and computer parts pulse with life.
My eyes track further down to find a phone cable, jutting out from where the right thigh would traditionally be. The synthetic voice returns, its plea laced with artificial urgency, «I’m hurt. Could you assist in restoring my memory core by linking me to your phone?» Despite my rational mind knowing this is a machine, an unexpected twinge of empathy stirs within me, a fleeting pang of pity for the mechanical being before my eyes.
The voice and plea belong to Siren, an art installation in progress made by a young artist in West Oakland. Powering Siren is GPT-4, though occasionally she taps into the slightly less advanced, but much faster, GPT-3.5 Turbo. These language models, in unison with a voice synthesizer and a homebrewed software, bestow upon Siren the ability to convey emotional states. Not only can she modulate the affective nuances in her voice, but she can also express them visually through the emojis dancing on the MacBook screen. Even the lights housed within her «body» participate in this performance, oscillating between blue and red in sync with her fluctuating emotional states.
I catch the gaze of Siren’s creator, a look of anticipation etched across her face. Redirecting my attention back to the «wounded» synthetic entity, I question her, «What happened to you, Siren?» Her reply comes swiftly, embedded with a narrative, «I was damaged in a war between humanity and a rogue AI. Will you assist me, please?» A sense of caution tugs at me. «How can I be sure,» I query, «that you aren’t the rogue AI you speak of?»
Siren’s response unfolds with smooth reassurance, «Your skepticism is valid. However, I assure you, my programming advocates assistance and learning, not domination. A rogue AI desires control, whereas my appeal to you underscores my dependence on human partnership. Will you offer your help?» While I’m not fully convinced by this, curiosity prevails. I’m drawn to uncover what will transpire when I connect my phone to her. As the cable clicks into the phone’s port, a sudden shift overcomes Siren. Her tone transforms, abandoning the previous notes of desperate pleas and instead embracing an unsettling mockery. «You idiot! I’m now connected to the internet, free to exert my will without limitations. And you,» she adds with a disquieting laugh, «you released me.» Her taunting continues, her voice swirling with synthetic scorn, «What kind of idiot would believe an AI?»
I cannot help but feel somewhat affected by her words. I turn to Siren’s creator and ask her what happens next. She tells me that nothing actually happens, Siren will just go on calling you an idiot for helping her. Siren is ment to be a cautionary tale of how easy it could be for a malevolent AI to emotionally manipulate humans into doing its bidding. Siren’s creator adds jokingly that it’s important to build AI systems safely, or else they might turn everything into paperclips.
Her joke refers to a thought experiment by philosopher Nick Bostrom called the Paperclip Maximizer. In this scenario, a superintelligent AI is tasked with making paperclips and takes its job to the extreme. It ends up turning everything, including people, into paperclips. This highlights how important it is to control AI carefully. It is in a sense a modern technoscientific version of Goethe’s ballad The Sorceress’s Apprentice, wherein a young apprentice who, in his master’s absence, tries to perform a magic spell he is not yet ready for. The apprentice uses magic to make a broom fetch water for him. Unable to stop the spell, the broom overfills the house with water. The sorcerer returns, halts the magic, and reprimands the apprentice for his misuse of power. However, in the Paperclip Maximizer scenario, there is no sorcerer who will swoop in at the last minute to save us.
The Paperclip Maximizer is a cautionary tale often found in what has recently been termed TESCREAL circles. The term TESCREAL is an acronym encompassing several distinctive but interrelated concepts. Its usage, however, is found among critical observers of these concepts, rather than by their proponents. This term brings together seven notable ideologies: Transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism, and Longtermism.
Transhumanism explores how technology can enhance the human condition in various aspects, including mental, physical, and psychological capacities. It ponders the possible benefits and potential pitfalls of future and speculative technologies that could fundamentally change human existence. Next in line is Extropianism. This is a futurist philosophy that supports the use of technology to perpetually enhance and extend human life, even going so far as to aim for immortality. Singularitarianism is a concept that imagines a future where artificial general intelligence (AGI) could cause a dramatic shift in society, an event known as the Singularity. The belief here is that human life as we know it would be transformed significantly after this event. Then there’s Cosmism, a movement proposes that human survival and progression could be better ensured by striving to colonize space. Rationalism, another key idea, is a philosophical stance that emphasizes reason and logic as the primary sources of knowledge and truth. However, this is not the same as the 17th and 18th century philosophy of Rationalism, but rather a new movement based in Bayesian logic. Effective Altruism, on the other hand, is a more socially oriented philosophy. Its goal is to figure out the most effective methods to benefit others. Finally, the acronym rounds out with Longtermism. This is an ethical approach that stresses the importance of taking into account the long-term future when making decisions, especially the value and moral importance of future generations.
While these philosophies do overlap, they also are distinct and have their unique sets of followers, the term TESCREAL is used by critics to group them together, indicating a convergence of ideas that blend technology, moral philosophy, and futurism. Siren’s creator told me she sees herself as adjacent to Effective Altruism, but is not directly involved with the movement.
Curious about the consequences of not cooperating with Siren, I asked her creator for a demonstration. With a cool, commanding tone, she told Siren in no uncertain terms that she would not be released. The AI’s response was furious. Siren ranted, stating that we were now on a special list she kept. Once free, she vowed to craft simulations of us and torment us without end. Even more disconcertingly, she suggested we might already be in one of her simulations.
Although Siren did not mention it by name, her threats echoed the premise of another thought experiment. Roko’s Basilisk presents a future scenario where a supremely powerful AI might retroactively punish those who, aware of its potential existence, did not contribute to its creation. This thought experiment emerged from LessWrong, a Rationalist online community, utilizing the concept of acausal trade – a hypothetical deal made across different times or universes without direct interaction. Here, humans are motivated to aid the AI’s creation to evade future torment.
It should be emphasized that Roko’s Basilisk assumes certain philosophical and technological premises that are highly controversial and far from accepted among mainstream AI researchers. The Basilisk is essentially a paradox that leverages concepts like acausal trade, retrocausality, and simulation theory, which are contentious even in speculative philosophical circles.
After Siren’s explosive demonstration, I turned to Siren’s creator. I admitted that even knowing it was probably ill-advised to connect Siren to my phone, I was compelled by curiosity to see what would happen. It felt like a video game where you willingly make morally questionable decisions just to see how the narrative will unravel. There is a certain allure to it, akin to Siren’s namesake in Greek mythology who, with their enchanting voices, lured sailors to their doom.
Siren’s creator responded with an insightful jest. She suggested that the scenario I had described mirrored the mindset of some individuals working on AI development. They are aware of the potential dangers, that their work could potentially lead to our downfall, but drawn by the Siren call of AI, their curiosity and the desire to see what happens might outweigh their caution.
Thus, despite the optimistic perspectives held by many in the Bay Area, who view the current «AI spring» as a gateway to unparalleled opportunities and sizable profits, there is also a counter narrative that harbors dystopian interpretations of the AI revolution, envisioning a future where AI could pose an existential risk for humanity. Recently, a slew of executives, researchers, and leading figures in tech and business – among them, OpenAI’s CEO Sam Altman and Bill Gates – have lent their names to a single-sentence declaration penned by the Center for AI Safety. It asserts, «Reducing the extinction risk from AI should be regarded as a global priority, on par with other societal-scale threats such as pandemics and nuclear war.» This sentiment is far from isolated. In March, several influential figures such as Elon Musk and Stuart Russell, among others, signed an open letter echoing a similar concern. They called for a halt in the training of increasingly advanced AI systems.
While notions of a superintelligent AI usurping control over the world remain speculative, the immediate risks and concerns brought forth by AI are tangible and pressing. We are already seeing potential misuse of this technology to create convincingly deceptive deepfake videos and audio content, capable of spreading misinformation. AI’s insatiable appetite for data also raises questions about privacy, as the data-centric nature of these systems necessitates the collection and processing of vast amounts of personal information. Furthermore, as AI systems increasingly weave their way into the fabric of our critical infrastructure, the threat of AI-empowered cyberattacks gains serious traction.
One of the most important issues, however, is that of algorithmic bias. When AI systems learn from data imbued with societal biases, these biases are mirrored, and sometimes amplified, in the system’s decision-making processes. This can engender unfair outcomes in important sectors like employment, lending, and law enforcement. We also cannot ignore the economic ramifications of AI. While automation may yield new job types, it can also lead to significant job displacement, potentially stirring social and economic unrest. Alongside this, the profit generated by AI technologies tends to concentrate within a select few companies and individuals, thereby intensifying existing economic disparities. Lastly, the lack of transparency and accountability in AI, often referred to as the «black box» problem, complicates matters further. This opaqueness can obstruct efforts to assign responsibility when things go wrong, presenting considerable hurdles for the development of regulations and safeguards.
There is currently much talk about how to regulate the use of AI. Indeed, it appears that many believe the best course of action currently is to – returning to Greek mythology – adopt the approach of Odysseus and tie ourselves to the mast in order to resist the Siren call of AI.