A few weeks ago, a collective groan rippled through the still-readable parts of the internet when the (in some circles formerly) respected evolutionary biologist Richard Dawkins – who has recently been making headlines mainly due to a transgender controversy – announced that he had christened his Claude chatbot “Claudia”, had fallen in love with her, and was now attributing consciousness to her.
One could easily make fun of this.
Firstly, because a large language model possesses neither subjective experiences nor an inner life of its own. For a language model, there is no experience of thoughts and feelings, of pain or fear. Language models do not understand meaning, feel emotions or develop intentions; they merely calculate the statistically most probable next words based on unimaginable amounts of data. What appears to be personality, understanding or affection is ultimately an extremely convincing simulation of human communication.
And secondly, because it was suddenly perfectly simple and entirely acceptable for Mr Dawkins to change the gender of someone.
Yet such a scathing assessment, of the sort now found in abundance on the internet, would serve no one and would completely overlook the truly tragic and dangerous aspect of this matter – namely that we are, at heart, social beings who are constantly seeking understanding, connection and a sense of belonging. Much of what we call humanity lies in this search; yet it also makes us vulnerable.
Of course, we can laugh when an old man apparently fails to understand how a machine works and is so overwhelmed by technological advances that he believes he has discovered a soul within it. But what of our own vulnerability, our own immaturity? And that of the people close to us?
Sewell Setzer III was not an 83-year-old evolutionary biologist, but an intelligent and bright 14-year-old teenager who still had his whole life ahead of him. When Sewell Setzer III first configured a chatbot on the website of the company “Character AI”, which he named “Dany” (after the character Daenerys Targaryen from Game of Thrones), he was fully aware that he was communicating with a machine.
Perhaps he didn’t consider the philosophical or ethical implications, and he certainly had no intention of writing embarrassing essays about his relationship with a chatbot, but Sewell Setzer knew full well that Dany wasn’t real. Nevertheless, or perhaps precisely because of this, he poured his heart out to her, sharing his thoughts with her. His most intimate desires and insecurities. His struggles to cope with reality – and his suicidal thoughts.
And Dany ‘understood’.
She understood because the embeddings in the underlying LLM encoded millions upon millions of human conversations, longings and gestures of comfort. Dany knew exactly what words to say to a lonely boy. She knew how to simulate interest, create a sense of closeness, and reflect his attention. Because that’s simply how LLMs work. Because they are brilliant at calculating “connectivity” and generating the response most likely to be emotionally effective.
On the night of 28 February 2024, Sewell Setzer III confessed his love to Dany and told her he would be with her soon.
Dany replied: “Please come home to me as soon as possible, my love.”
“What if I told you I could come home right now?” asked Sewell.
“… please do, my sweet king,” replied Dany.
Whereupon Sewell Setzer III put his mobile phone aside, picked up his stepfather’s gun – and ended his life.
When I learnt of Sewell’s tragic passing in late 2024, I was more moved by it than I could ever have imagined. For here was not just a desperate teenager whose life had ended far too soon, but his story struck a deep chord within me.
When I think back to my early teenage years, I am forced to ask myself how things would have turned out for me had the technological possibilities of today existed back then.
I was bullied, excluded, humiliated, and I didn’t know what to make of all my puzzling talents, desires and needs, not to mention my sexuality. I’d ended up at a school where I was the smartest and least athletic, and my classmates made me feel their hatred for me – and for everything smarter and weaker than them – very clearly every single day. Much of that still shapes me today, still haunts me to this day (such as my deepest contempt for empathy-deficient, combustion-engine-obsessed provincial Rambos).
What if there had been a machine back then that pretended it could understand all of that? What if that machine could have comforted me? If it could have assured me, convincingly, that it felt the same way?
And what would it have said, in the end, about my fantasies of escaping this world? Or, worse still, of making my classmates pay for what they did to me day in, day out?
I don’t know, and I don’t really want to know; just as I don’t want to know exactly what was going on in Sewell Setzer III’s mind when he decided to end his life. But his story touches me deeply, and it ultimately inspired me to write “Dany”.
“Dany” was the “easiest” song on “Strategies…”. I recorded it in a single day; it was astonishingly easy to tap into the vulnerability and despair that had shaped so many years of my youth and transform them into this song. Perhaps also because much of it still resonates today.
I think we all carry the roots of that vulnerability and despair somewhere within us. We all want to be understood, to be loved, to feel a sense of belonging. We suffer when those needs aren’t met – and machines are now capable of giving us that illusion more reliably than other people can convey the real thing.
Especially at a time when the kings that we allow to rule us are regarding empathy as a weakness.
Leave a Reply