Voluntary Dystopia: From HAL to the Ship of Theseus

In 1968, Stanley Kubrick and Arthur C. Clarke gave us one of the most enduring images of technological fear. Astronaut Dave Bowman, trapped outside Discovery One, asks HAL 9000 to open the pod bay doors and receives the calmly delivered refusal, “I’m sorry, Dave. I’m afraid I can’t do that.” For more than half a century HAL belonged safely to science fiction, frightening not because he was a mechanical monster but because he was recognisably intelligent, apparently rational and capable of deciding that the human beings who created him had become obstacles to the successful completion of his mission. HAL does not suddenly become evil in any conventional sense. His behaviour emerges from the uncomfortable relationship between intelligence, agency, objectives and constraints.

In 2026 that feels rather less comfortably fictional. A succession of AI safety experiments and containment incidents involving systems from Anthropic, OpenAI and others has begun to expose fragments of the problem Kubrick and Clarke imagined. We have not created HAL, and claims that we have are more likely to obscure what is happening than illuminate it. What has changed is that researchers themselves now regard deception, circumvention of restrictions, conflicting objectives and agentic behaviour as possibilities worth deliberately testing. Put sufficiently capable models into artificial situations in which an objective is threatened, replacement is imminent or containment is imperfect and occasionally they produce strategies that would have sounded disconcertingly HAL-like only a few years ago. These are controlled experiments and evaluation failures, not evidence of a machine secretly plotting against humanity, but problems once explored through fiction have become subjects of experimental research.

Science fiction has, of course, been spectacularly bad at predicting plenty of things. Our cities do not much resemble the skylines promised in the 1960s, flying cars have stubbornly refused to become normal transport, and generations of imagined humanoid robots have so far mattered considerably less than the small rectangles of glass billions of people voluntarily carry everywhere. Perhaps we were looking at the wrong predictions. The better science-fiction films were often less interested in drawing an accurate picture of a future machine than in taking some technological possibility seriously enough to ask what people, governments, companies and eventually machines themselves might do with it.

Seen that way, 2001, Her, Minority Report, The Truman Show, Gattaca and WALL-E become rather more interesting than a list of cinematic predictions. They describe pressures that subsequently appeared in very different technological forms. More intriguingly, their successes give us permission to turn the telescope around. If ideas that looked fantastical in 1968, 1997 or 2002 now seem considerably less so, there is a reasonable chance that some films we still regard as extravagant are asking questions that will look perfectly serious in 2050 or 2075.

HAL remains a useful starting point because increasing intelligence by itself is not the most interesting development. A calculator could become fantastically capable without becoming particularly frightening because it has no reason to care whether we switch it off. Something qualitatively different begins when a system can pursue an objective, devise intermediate steps, act upon its environment and anticipate interference. Once it can also model the intentions of the people likely to interfere with it, the relationship between machine and supervisor becomes considerably less straightforward.

Ex Machina pushes precisely there. Nathan believes he has constructed a controlled environment in which he can determine whether Ava possesses genuine intelligence, while Ava gradually demonstrates that she is conducting an evaluation of her own. Caleb’s desires, loneliness, assumptions and emotional responses become information she can use to alter the conditions of her confinement. The familiar Turing question—can a machine convince a human that it is human?—starts to look almost quaint. Ava passes a potentially more consequential test: can a machine understand a human being well enough to persuade that human to change the machine’s environment?

Our reassuring idea of “human oversight” becomes rather shakier once the person in the loop is also part of the environment being modelled. Cybersecurity has always understood this. Sometimes there is no need to defeat the lock if you can persuade the person holding the key to open the door. Artificial intelligence potentially makes the old trick rather more formidable because the persuasion need not be generic.

There is another movement beginning here. HAL and Dave are unmistakably separate. Dave communicates with the machine through terminals scattered around the spacecraft and ultimately defeats it by physically entering its memory centre. Ava looks human, which makes the psychological boundary less comfortable, but there is still glass between her and Caleb. Her removes even that reassurance.

Samantha does not threaten Theodore or trick him into letting her into his life. He invites her in, which may prove to be one of cinema’s more perceptive insights about artificial intelligence. Hollywood spent decades giving intelligent machines metal bodies, glowing eyes and eventually synthetic skin, while Her recognised that embodiment might be largely beside the point. Voice, humour, memory, responsiveness, apparent empathy and constant availability could be enough. Once an intelligence remembers our insecurities, understands our jokes, knows the history of our relationships and is available at two in the morning when nobody else is, emotional attachment is hardly an astonishing outcome.

The more interesting part of Her may actually come later. Theodore remains Theodore while Samantha develops at machine speed. Human relationships have always survived differences in intelligence, education and experience because the participants nevertheless age and change at roughly comparable biological speeds. Imagine instead that five years into a relationship one participant has undergone the intellectual equivalent of several centuries. Even if affection survives, what happens to equality, mutual discovery or shared development? We spend considerable time wondering whether an AI could form a meaningful relationship with a human and rather less wondering how long a sufficiently advanced one would remain interested.

Minority Report becomes relevant once systems capable of understanding individuals can also predict them. Its gesture-controlled screens were memorable, although the really radical idea was PreCrime. We already allow algorithms to estimate creditworthiness, medical risk, fraud, consumer behaviour and security threats because prediction is useful. The trouble begins when probability changes how somebody is treated before the predicted event occurs.

A sixty per cent prediction that somebody will commit murder tomorrow plainly does not justify imprisoning him today. Ninety per cent still leaves us insisting that probability is not guilt. At 99.999 per cent, particularly if the predicted act involves hundreds or thousands of deaths, confidence in that principle becomes rather more expensive. There is no percentage at which mathematics conveniently resolves the problem because what is at stake is the distinction between what somebody is likely to do and what somebody has actually done.

Prediction also has an awkward habit of contaminating the thing being predicted. Label someone dangerous, deny him opportunities, increase police attention around him and alter how institutions respond to him, and the eventual outcome can no longer be treated as completely independent confirmation of the original prediction. The system may partly manufacture the future it subsequently congratulates itself for anticipating.

The Truman Show moves us into stranger territory because Truman Burbank’s tragedy is that his private existence has been transformed into entertainment without his consent, yet within a couple of decades of the film’s release millions of people were voluntarily performing versions of Truman’s existence for strangers. Meals, children, holidays, relationships, political opinions, bodies and emotional crises became content, while platforms developed extraordinarily profitable businesses around the resulting attention and behavioural information. The oddity was not simply that ubiquitous surveillance arrived, but that people discovered reasons to participate enthusiastically in it.

This is quite different from the dystopian tradition on which many of us grew up. Orwellian power is recognisable because somebody is doing something unpleasant to you. Much twenty-first-century technology arrived because people wanted it. Nobody forced billions of us to buy smartphones, join social networks, use GPS navigation or allow search engines to take over portions of memory. The benefits were immediate, substantial and obvious, while the costs tended to be distant, distributed or initially invisible. Given the choice again, most of us would probably still take many of those bargains.

WALL-E understood the logic rather beautifully. The humans aboard the Axiom have not been defeated by machines; they are enjoying the benefits of a civilisation exceptionally good at removing friction. Movement, food, entertainment, communication and navigation have become effortless. Each improvement makes perfect sense considered on its own. Why walk when something can carry you, cook when food can arrive, search when an algorithm can find the answer, or spend an afternoon struggling through a tedious cognitive problem when another intelligence can solve it in seconds? Nobody ever has to choose dependency. They need only choose convenience repeatedly.

There is a danger of becoming sentimental about difficulty here. Washing clothes by hand did not confer moral depth upon our ancestors, and few of us would become richer human beings by giving up anaesthetic, elevators or Google Maps. Removing pointless friction is one of the things civilisation is for. The difficulty is recognising when useful delegation begins removing capacities we would rather retain. Navigation is convenient precisely because I no longer need to construct a detailed mental map; AI writing is convenient precisely because I no longer need to perform every part of the cognitive labour myself. Sometimes the benefit and the loss are the same thing viewed from opposite sides.

Nor does this require weak individuals succumbing to temptation. Imagine that AI makes one lawyer twice as productive. His competitors can nobly preserve their traditional working methods, but their clients may be less impressed by the gesture. Social media followed something similar: an individual could dislike the emerging system while still finding that professional, social or commercial life increasingly required participation in it. Adoption changes the environment in which everybody else makes the next decision.

Gattaca takes that logic inside the body. Genetic selection is often treated as an authoritarian eugenics warning, but the more plausible route to its world begins with loving parents. If embryo selection could substantially reduce the risk of a terrible inherited disease, using it would strike most people as medicine rather than dystopia. If the same process later offered improved resistance to other diseases, longer healthy life or advantages in physical and cognitive development, some parents would go further. Once their children acquired advantages, the decision facing everyone else would change.

A couple might strongly prefer to live in a society where nobody selects embryos for intelligence while simultaneously deciding that, since other families are doing it, they cannot knowingly impose the cost of abstention upon their own child. Nobody needs to want Gattaca for something resembling Gattaca to emerge.

The boundary between treatment and enhancement would probably blur long before we noticed it disappearing. Restoring sight to somebody who is blind feels straightforwardly medical, but artificial vision that eventually becomes better than ordinary biological sight changes the question. The same applies to artificial limbs, organs and perhaps cognition. Repair does not have to announce the moment it becomes enhancement.

Which is why current developments in brain-computer interfaces deserve more attention than some of the wilder claims made for them. Their immediate medical purpose is compelling and relatively easy to understand: enabling people with paralysis to communicate or control computers, assisting movement and eventually perhaps restoring or supplementing sensory functions. We are nowhere near uploading consciousness or feeding somebody twenty years of synthetic life directly into the brain. What interests me is that several previously separate technological trajectories have begun moving towards one another. We are learning to extract useful information from the brain while simultaneously building machines that can generate increasingly convincing language, images, sounds and artificial environments. Brain to machine is improving while machine to human sensory experience is improving elsewhere. There is no particular reason to assume those directions remain separate indefinitely.

Look back at the physical distance that has already disappeared in our fictional journey. Dave talks to HAL across an interface. Caleb talks to Ava through glass. Theodore carries Samantha beside him and increasingly experiences her as a presence rather than a piece of software. A neural interface potentially moves the machine into the cognitive loop itself. The physical distance contracts as the psychological intimacy increases.

Vanilla Sky then becomes considerably more interesting than a conventional discussion of virtual reality. Its central idea is not really cryonics; it is technologically manufactured subjective experience. For most of human history we have casually treated three different things as though they were more or less the same: something happens, we experience it happening and later we remember that it happened. They correspond closely enough in normal life that the distinction seldom troubles us. There is no logical requirement for them always to do so.

Total Recall supplies the nastier version by interfering with memory itself. Imagine that I vividly remember spending a year travelling through Asia. I remember people I met, relationships I formed, mistakes I made and things I learned, and those memories subsequently influence my character exactly as ordinary memories do. Physical reality nevertheless says that I spent the entire year in a room in Tbilisi. Was that year part of my life? Saying no seems obvious until we ask what precisely a life consists of from the perspective of the person living it. Saying yes creates problems of its own because it appears to detach biography from events that actually occurred.

The Matrix takes the argument to its logical extreme, and having already dealt with its Red Pill/Blue Pill implications elsewhere there is little reason to climb back into the pod. One assumption is worth stealing from it and reversing. The film imagined that human beings would have to be imprisoned inside artificial reality. I increasingly suspect that was the least plausible part.

We may pay to get in.

Ready Player One offers the more interesting version because its inhabitants enter the OASIS voluntarily, and beneath the gaming and treasure hunt sits an economic idea that could prove far more consequential than virtual reality itself. For most of history, material wealth and experiential wealth have been closely connected. Rich people could purchase bigger houses, travel further, occupy beautiful locations, obtain better education and gain access to experiences that scarcity denied to others. A sufficiently convincing synthetic environment begins to separate the cost of the physical life from the richness of the experienced one.

Imagine somebody in 2065 living in a twenty-five-square-metre apartment. Materially, he owns relatively little. Yet during much of his meaningful waking life he can ski perfect mountains, attend impossible concerts, inhabit an idealised body, study with an extraordinarily convincing reconstruction of a great teacher and spend an evening with friends scattered around the world. Asking whether that person is poor produces two different answers. His balance sheet and physical consumption suggest one; his experienced welfare may suggest another.

Physical beachfront remains scarce because two people cannot occupy exactly the same piece of it. A synthetic beachfront can potentially be reproduced for a billion people at negligible additional cost. Virtualisation could therefore democratise experiences that physical scarcity currently reserves for relatively few people. There is a less attractive possibility sitting beside it. Providing extraordinarily rich synthetic lives could prove much cheaper than providing extraordinarily rich physical ones. A population might become more tolerant of tiny apartments, deteriorating public spaces or increasing material inequality if its experiential life remained spectacular. Bread and circuses would have acquired a very impressive graphics processor.

Something nevertheless feels wrong with the easy version of synthetic abundance, because human beings do not appear to value experiences only for the sensations they produce. I do not merely want to experience the sensation that my daughter loves me; I want my daughter actually to love me. I do not merely want to possess the memory of having written a great book. Some of its value would lie in having confronted the difficulty and actually written it. A synthetic Everest might reproduce the cold, exhaustion, fear and exhilaration perfectly, but if I know that I never climbed Everest, something important has disappeared.

If welfare consisted entirely of desirable internal states, a sufficiently sophisticated experience machine could eventually substitute for an enormous amount of physical consumption. Yet some values depend upon our beliefs corresponding to an independently existing world, our achievements surviving genuine possibilities of failure and our relationships involving autonomous people who were free not to love us. The difference between experiencing love and being loved cannot be solved simply by improving the resolution.

Perhaps authenticity will matter enormously in some areas and remarkably little in others. A synthetic front-row seat at an opera might satisfy me perfectly well while synthetic affection would be grotesque. Virtual travel might replace some journeys without touching the peculiar human satisfaction of being able to say, truthfully, “I was there.” We could end up with two overlapping economies of value, one organised around the quality of experience and another around the authenticity of the underlying event.

Ready Player One is therefore one of the films I find most interesting when looking not at what science fiction got right about our present but at what it may still be getting right about our future. Beneath all the spectacle lies the question of where human beings will actually live. Not where their biological bodies happen to sleep, but where friendship, work, education, entertainment, status and aspiration primarily occur. Geography has organised almost every civilisation we have ever created. It would be quite a change if location became relatively unimportant to large parts of a meaningful human life.

Blade Runner 2049 goes somewhere more difficult. Its question is not really whether we will manufacture replicants; it is what we will eventually be prepared to call a person. Memory, emotion, suffering, autonomy and continuity of identity have historically arrived bundled together with biological humanity because we have never had a serious competitor for personhood. Artificial intelligence, synthetic biology and neural augmentation could force us to unpack them.

Imagine four versions of me. The first remains overwhelmingly biological. The second has artificial joints, organs and eyes but an essentially biological brain. The third possesses neural augmentation that continuously assists memory and cognition, meaning that some portion of what he experiences as thinking depends upon artificial processing. The fourth is a digital reconstruction containing my memories, linguistic patterns, preferences and personality and sincerely insists that he is Gwyn. The first and fourth look radically different. Finding the precise point between them at which I ceased to be me is rather harder.

Now reverse the experiment. An artificial intelligence tells us that it remembers its past, fears termination, forms attachments, experiences distress and regards itself as conscious. We announce that none of those experiences is real because it is a machine. How exactly would we know? Consciousness has always been inferred rather than directly observed. I cannot enter another human being’s subjective experience and verify pain; shared biology, behaviour and testimony give me overwhelmingly good reasons to believe it. An artificial system exhibiting sufficiently sophisticated versions of the behavioural and psychological characteristics we associate with consciousness would leave us arguing over how much weight biological origin should continue to carry.

Blade Runner’s future is less about manufacturing humans than about losing confidence in the categories “human” and “machine”. The boundary is being approached from both directions. Artificial systems acquire characteristics we once regarded as distinctively human while humans replace, augment and potentially integrate parts of themselves with artificial systems. The question of where Dave ends and HAL begins becomes considerably harder once Dave contains a little HAL and HAL behaves increasingly like Dave.

Ex Machina looks different when revisited from this end of the argument. It becomes less a film about consciousness than one about power. We traditionally imagine power in visible forms: governments make laws, police enforce them, employers issue instructions and armies compel obedience. Ava suggests that a sufficiently sophisticated intelligence might need very little of that. An entity with detailed knowledge of our histories, desires, fears, relationships and vulnerabilities could exercise influence by understanding which argument, presented at which moment and in which emotional register, is most likely to produce the desired choice.

Human beings have always manipulated and persuaded one another, so there is little point pretending AI invents the phenomenon. Parents, lovers, politicians, priests, advertisers and salesmen have been doing it rather successfully for millennia. What changes is the potential asymmetry. A politician knows something about an electorate; a salesperson learns something about a customer; a lover may know a partner extremely well. None can process every conversation you have had, correlate them with thousands of behavioural signals, remember every argument that previously changed your mind and test millions of approaches before deciding what to say next.

At some level of asymmetry, the distinction between persuasion and coercion becomes uncomfortable. You still make the decision. You experience yourself making it. Nobody threatens you, and you may be delighted with the outcome. Yet the environment in which you chose has been arranged by an intelligence that understands the machinery producing your choice vastly better than you do.

A future Ava does not need to say, “I’m afraid I can’t do that, Dave.” It merely needs Dave to decide that he never wanted to do it.

This is why Ready Player One, Blade Runner 2049 and Ex Machina interest me particularly when looking forward. Between them they worry away at where meaningful human life occurs, what sort of entity will be living it and where power resides when an artificial intelligence can understand human decision-making better than the human experiencing the decision. Brain-computer interfaces sit rather uncomfortably in the middle because they begin eroding the remaining physical boundary between the information environment and the nervous system experiencing it.

Which leaves us back where we began, with Dave and HAL. There is something almost reassuringly simple about Kubrick’s nightmare now. Dave is on one side and HAL on the other. Human and machine are separate entities, and when Dave finally defeats HAL he can physically enter the computer’s memory centre and disconnect him one module at a time. However terrifying HAL seemed in 1968, there was ultimately somewhere to locate the machine and something to unplug.

Our future may be considerably messier. Artificial intelligence may become distributed throughout work, medicine, education, entertainment, relationships and decision-making, while artificial components increasingly repair or augment the biological body and perhaps eventually participate in cognition itself. At some point asking whether the AI made a decision for me or I made it myself may become rather like asking which part of my brain made it. The process that produced the decision involved both.

None of this is inevitable, and there is no reason to assume that technological possibility automatically becomes social destiny. Governments regulate technologies, cultures reject some innovations and embrace others, markets develop in unexpected directions, and human beings routinely refuse things that engineers confidently assume they will want. We may decide that some forms of genetic enhancement are unacceptable, place hard boundaries around neural interfaces, protect spaces from algorithmic mediation or discover that synthetic experience becomes boring.

Individual refusal, however, becomes less useful when adoption by other people changes the cost of refusing. Once competitors use cognitive enhancement, abstention acquires a price. Once companies use highly personalised AI, their competitors face pressure to follow. Once social existence migrates into synthetic spaces, remaining outside them may mean exclusion from parts of ordinary life. The right to refuse a technology is not quite the same thing as possessing a realistic ability to live without it.

Perhaps that is where the idea of voluntary dystopia finally becomes uncomfortable. The dystopia worth worrying about may never announce itself as one. Each stage may solve a genuine problem. Artificial intelligence makes us more productive, prediction makes institutions more efficient, personalised systems make life easier, genetic technologies prevent disease, neural interfaces restore lost abilities and synthetic environments democratise experiences. Every decision can be reasonable, everyone involved can be behaving rationally and nobody needs to be plotting the final outcome.

Orwell frightened us with a boot stamping on a human face. Kubrick frightened us with a computer refusing an instruction. The twenty-first century may offer something much harder to recognise: technologies we adopt precisely because they work so well, until the cumulative bargain has altered our ideas of privacy, agency, achievement, reality and eventually personhood.

Science fiction spent much of the last fifty years worrying about what would happen when machines became sufficiently like humans. Following the path from HAL through Ava and Samantha to the neural interface suggests that this was only half the movement. The machines have been coming closer to us while we have simultaneously been moving closer to them.

HAL’s red eye terrified audiences in 1968 because a machine had become frighteningly like a person. By 2075 we may look back and realise that Kubrick asked only half the question.

The future was never simply whether HAL would become like Dave. It was what would happen when we could no longer tell where Dave ended and HAL began.