[Paradigm Reset] Chapter 6: What Existing AI Discourse Gets Wrong

For about three years now I have been giving more or less the same talk in different rooms: Stanford, San Jose State, Texas A&M, Seoul National University, Korea University, KAIST, POSTECH. The slides change and the audiences change, but the first question from the back of the room never does. Someone stands up, usually a student but sometimes a professor who has waited politely through forty minutes of convex optimization and cancer biomarkers, and asks whether AI is going to replace us. When it is not that question, it is its twin, asked in a tone of relief: so AI is just a tool, right?

For a while I answered, and I had a good answer, with charts. Then I noticed that however I answered, the room left with exactly the picture it had come in with: humans on one side and machines on the other, a line between them. The only question worth asking was where that line would sit next year. My answer had not touched the picture. It had only decorated it.

The question changes its clothes with the audience but not its shape. In rooms full of policymakers rather than students, nobody asks whether AI will replace anyone; they ask how far behind the country is and how fast it will need to run to catch up. I have given numbers in answer to that, because numbers were asked for, and watched them settle into the same picture: a race with a track, a finish line somewhere ahead, and the only question being our position on the track. Nobody asks who laid out the track, or what is waiting at the end of it.

This chapter is about why I stopped answering those questions, and two others like them.

Part I of this book was the story of a pattern that took me twenty years to see. At Samsung I found the best possible DRAM design and never once asked who had decided what “best” meant. At Amazon I helped ship an algorithm worth two hundred and seven million dollars and could not have told you whether it had made a single person’s life better. At Gauss Labs I watched people on a factory floor refuse to trust an algorithm. At Erudio Bio I found that before health data can be used to save a life, someone has to answer the prior and much harder question of who owns it. Each time the technical question got answered, and each time the real question was never asked. The Interlude named the pattern. Part II asks why it exists, and the honest place to start is with the questions everyone else is asking, because those are the ones that crowded out the real one.

There are four of them: human versus machine, technology as destiny, hype versus fear, and regulation as follower. Each one sounds like a question. Each one is also a frame, and a frame decides in advance what will be allowed to count as an answer.

Wittgenstein wrote that the limits of my language mean the limits of my world, and although he was talking about logic, the sentence has a plainer use here. A question is a piece of language. If the only questions available about AI are these four, then there are things about AI that cannot be thought, not because they are false but because there is no sentence to think them in. Wittgenstein also said, in a stranger mood, that if a lion could talk we could not understand him. I have come to think that much of the current conversation about AI is a conversation with a lion that everyone has agreed to call a conversation. The words are in English. The frames underneath them belong to a different animal.

Thomas Kuhn made a related observation about science. Most of the time, he said, scientists are not overturning anything; they are solving puzzles inside a paradigm and treating whatever does not fit as noise. Only when the noise accumulates past a certain point does anyone go looking for a new paradigm. The four frames are, in exactly that sense, AI’s normal science. They generate puzzles, the puzzles get solved, and the solutions confirm the frames. And Michel Foucault asked a question that I will keep returning to, because it turns out to unlock the other four: when one way of asking becomes the obvious way, who benefits from its being obvious?

I should say a word about why an engineer is quoting philosophers at all, since I know how it reads. It is not decoration. Wittgenstein, Kuhn, and Foucault spent their lives on one problem, which is how a way of asking can trap a way of thinking without anyone noticing the trap, and that is precisely the problem this chapter has. When I reach for Aristotle later, or Sartre, or Aquinas, it is for the same reason. Each of them saw something about choice, or purpose, or law, that the current conversation has forgotten, and a sentence of theirs does more work than a page of mine. I will name them clearly and move on. Nobody needs to have read them.

Four frames, then, taken one at a time. For each I will try to say three things: what it asks, what it hides, and who gains from the hiding. At the end there is a table that puts the four side by side, and by then I hope they will look less like four separate mistakes and more like one. I should also say what this chapter does not do. It dismantles. It does not yet build. The building begins in Chapter 7, and it would be dishonest to start it before the ground is clear.

The Human vs. Machine Trap

The date that never arrives

Start with the version of the question that comes with a date attached: when will artificial general intelligence arrive? The answers range from “already here” to “never,” and a great deal of money and anxiety rides on the spread between them. I want to set the answers aside and look at the question itself, which is doing more work than any of the answers.

The forecasting record is worth a moment, because it is long and it is consistent. In 1950 Alan Turing predicted that by the end of the century a machine would play his imitation game well enough that an average interrogator, after five minutes of questioning, would be fooled about thirty percent of the time. In 1957 Herbert Simon, who would later win a Nobel Prize, predicted that within ten years a computer would be the world chess champion and would discover and prove an important new mathematical theorem. In 1970 Marvin Minsky told Life magazine that in three to eight years we would have a machine with the general intelligence of an average human being. These were not fools. They were among the most capable people of their century, and each of them was wrong in the same direction, with the same confidence, about the same thing.

Then look at what happened when the machines did arrive. In 1997 a machine beat the world chess champion, and chess stopped being the measure of intelligence roughly the following week. In 2016 the same thing happened to Go, which had been the fallback measure because its search space was supposed to be too large for brute force. In 2020 it happened to protein folding, a problem that had defeated laboratories for fifty years and was supposed to be different because it was about nature rather than games. Nobody held a meeting to move the goalposts. They moved on their own, which is what goalposts do when they were never anchored to anything, and so the date keeps arriving and never arrives.

The question has not gone away; it has been industrialized. There are now regular surveys of researchers whose median estimate for the arrival date shifts by a decade or more between editions. There are forecasting markets that price the date like a commodity, and reports that pick a year and build a scenario around it. None of this is foolish on its own terms. What interests me is that a question which has been answered wrongly for seventy years is still the first one asked, and the reason is not that the answer is useful. It is that the question is useful. It raises money, it frightens, and above all it defers: as long as the date is somewhere ahead, every harder question can wait until it arrives.

When a forecast fails for seventy years in the same way, the lesson is not that the forecasters are bad at their jobs. It is that the thing being forecast is not a thing. “When will AI arrive at human level” presupposes a finish line; a finish line presupposes a single track; and a single track presupposes that intelligence is one quantity, that it comes in amounts, and that human beings sit at some known mark on the scale, waiting to be passed. Every link in that chain is doing work, and none of them survives contact with the record.

The intelligence spectrum fallacy

The fallacy underneath is easy to see once you put two systems side by side, both built on the Transformer, the architecture behind most of what now gets called AI. One reads medical images and flags pancreatic cancer months before a radiologist would catch it, which is the difference between a disease that is treatable and one that is not. The other produces disinformation at industrial scale, thousands of plausible articles an hour, each one slightly different so that no filter catches them all. Which one is more intelligent? The question has no answer, and not because we lack data about either system. It has no answer because placing both on a single axis erases the only fact about them that matters, which is what each was built to do and for whom. The intelligence spectrum is a ruler laid across things that do not share a dimension.

The robotics community noticed a version of this forty years ago and gave it a name. Moravec’s paradox is the observation that the things humans find hardest, such as symbolic reasoning, chess, and calculus, turned out to be easy for machines. The things any four-year-old does without effort, such as recognizing a face, walking across a cluttered room, or knowing that a cup will spill, turned out to be fiendishly hard. If intelligence were one quantity, that could not happen; a machine that was “smarter” than a child would be smarter at everything. The paradox dissolves as soon as you drop the single axis, which is to say it was never a paradox about machines. It was a paradox about the ruler.

Aristotle, who thought about this more carefully than anyone before the twentieth century, distinguished three kinds of knowing. There is episteme, which is theoretical knowledge, the kind that can be written down and proved. There is techne, which is craft, the knowledge of how to make or do something well. And there is phronesis, usually translated as practical wisdom, which is the knowledge of what is worth doing and for whom, exercised in particular situations that no rule fully covers. Machines have climbed a long way up the first two. A system that proves theorems has a kind of episteme; a system that folds proteins has a kind of techne. The spectrum question quietly assumes that phronesis is simply more of the same, further along the same ruler, and that a machine with enough of the first two will arrive at the third. Aristotle’s point was that it is a different kind of thing altogether, and I have met many brilliant people, and built several brilliant systems, that had a great deal of the first two and none of the third.

A footnote worth saying out loud, because almost nobody does: nobody actually uses a large language model. People use systems with a model embedded somewhere inside, wrapped in retrieval and tools and policies and an interface that someone designed. The model at the core is, in plain terms, a very good estimator of which word is likely to come next given everything that came before. That is a description, not an insult. It makes “how smart is it” the wrong unit of measure, in the way that “how strong is it” is the wrong unit for a bridge. A bridge is strong in some directions and not others, and was designed to be so by someone who had a particular river in mind.

Category errors in the consciousness debate

The trap’s deepest room is the consciousness debate. Does the machine understand what it says? Is it aware? Could it suffer? These questions have a long and distinguished literature, and I am going to do something that may look disrespectful to it, which is to set it aside in a few paragraphs. I want to explain why that is the respectful thing to do.

Gilbert Ryle had a name for what goes wrong here. He called it the category mistake, and his example was the visitor who is shown every college and library and laboratory in Oxford and then asks, politely, where the university is. The visitor has not missed a building. The visitor has mistaken the kind of thing a university is. Much of the consciousness debate about machines has that shape. We are shown the weights, the attention patterns, the training data, the outputs, and then we ask where the understanding is, as though it were one more component we had not yet been shown. John Searle’s Chinese Room and Thomas Nagel’s question about what it is like to be a bat are the famous furniture of this room, and both are serious arguments that I am not going to settle. I only want to notice what they share, which is that they are about what it is like from the inside.

Edmund Husserl supplies the other half of the point. Consciousness, he argued, is always consciousness of something; it has a direction, an aboutness, and it is known from within, by the one who has it. Grant all of that. It follows that a question which can only be settled from the inside cannot ground a decision that has to be made from the outside. Whatever the truth about machine awareness turns out to be, no regulator, no hospital, and no court will ever have access to it, because there is no instrument that reads it off. We will be deciding what these systems may do, and who is responsible when they do it, without that answer, and we would be deciding without it even if the philosophers reached consensus tomorrow.

Try the thought experiment the other way. Suppose it were settled tomorrow, beyond dispute, that some machine was conscious. What would change in the decisions in front of us? We would still need to decide who may deploy it, for what, with what error rate, and who answers when it fails. We would have acquired a new obligation toward the machine, which is no small thing, but we would not have discharged a single one of our obligations toward each other. The governance questions are indifferent to the answer, in both directions, and a question whose answer changes nothing you are about to do is a question you can set down.

So for the purposes of this book the consciousness question is not wrong so much as idle. It is a real question that produces no leverage on the decisions in front of us, and a great deal of the current conversation consists of people pulling on it anyway. I am going to leave it where it lies, with respect, and get on with the questions that move something.

Replace or assist: what actually happened to me

What I cannot leave alone is the trap’s most popular form, because I fell into it myself and stayed there for years. Replace or assist? Will the machine take the job, or help with the job? Every panel I have sat on has eventually arrived at this question, and every panelist, including me, has at some point given one of the two answers.

Let me tell you what actually happened to me. I finished my PhD at Stanford under Stephen Boyd with a command of convex optimization that fewer than a hundred people in the world share. I also left with a private embarrassment that I told almost no one about: I did not understand duality the way Boyd did. I could use it and I could teach it, and I did both for years, but I could not see it, and that stayed true through Samsung and Amazon and a startup, for more than a decade. Then, over a stretch of weeks, I spent hours at a time in dialogue with an AI system about duality. I asked, got an answer, pushed back because the answer was so often wrong in instructive ways, corrected it, added what I knew, and asked again. Somewhere in that back-and-forth the thing I had been missing for twelve years clicked into place, and it has stayed in place.

The system did not understand duality, and I want to be exact about that. It was not a teacher, because a teacher knows the destination. It was closer to a sparring partner who never tires and never takes offense, and whose mistakes were as useful as its answers, because defending a point against a wrong objection is how you find out whether you actually hold it. What it gave me was something I had never had and could not have arranged: the experience of arguing with a hundred versions of the handful of people in the world who could have argued back, all of them patient and none of them busy. The understanding happened in me. Nothing happened in it.

Once I had seen the shape of that, I started seeing it everywhere, and it is easiest to show with a few scenarios of the kind I now use in lectures. Picture a physician who asks the system for fifteen differential diagnoses. She dismisses twelve of them in under a minute, because she has examined the patient and the system has not, and spends her attention on the three she would not have thought of alone. Picture a consultant with twenty interview transcripts and a week to find the structure in them. He takes them into a system and comes out three days later with a map that would have taken a month. That is not because the system knows his client’s industry; it is because the system can hold all twenty conversations in view at once while he decides which patterns matter. Picture a biotech researcher staring at an anomalous result with seven candidate explanations. The system proposes experiments that would discriminate between them; she knows which of those experiments her lab can actually run by Friday, and that knowledge is the whole difference between a list and a plan. Picture someone learning a language in what might be called synthetic immersion: an interlocutor that never tires, never judges, and never lets them get away with the easy construction. In each case the person supplies the thing the system lacks. The physician supplies the patient. The consultant supplies the judgment about which pattern matters. The researcher supplies the lab. The learner supplies the desire to speak.

There is an older version of this that everyone already understands. Every paper presented at every conference is available online, usually before the conference begins, and yet people still get on airplanes and fly across oceans to sit in rooms and hear them read aloud. They do it because the value of a conference was never the papers. It is the network effect of intelligence: the question from the floor that reframes the talk, the conversation at the coffee break that becomes a collaboration, the sense of a field thinking together. The Convex Optimization Forum, which meets on both sides of the Pacific, exists for that reason and no other. AI does not replace that network. It is the first thing in history that gives a single person access to something like it at any hour, on any subject, with no one keeping score.

Neither “replace” nor “assist” describes that. Replace is wrong on its face, since I was not replaced but changed. Assist is an understatement that misses the mechanism, because an assistant does part of your work and this did none of mine; it changed what I was capable of doing myself. The right word comes from chemistry. A catalyst lowers the barrier to a reaction and comes out the other side unchanged, with the whole transformation happening in the reactant, and so I have taken to calling what happened to me cognitive catalysis. Notice what that structure implies. In every one of the stories above, the human supplies what the system lacks: the domain, the question, the stubbornness, the desire to understand. A stronger catalyst does not make those contributions less valuable. It makes them decisive, because they become the only thing that distinguishes one outcome from another.

Which jobs are safe? The same trap with a salary attached

The jobs question is the human-versus-machine question wearing a suit. Which jobs are safe, and which will disappear? Ask it and you can only get back lists, winners and losers sorted by occupation. The lists will be wrong in the way AGI dates are wrong and for the same reason. They assume the technology arrives with its uses already decided, so that the only thing left is to sort people into its path or out of it.

The historical record is unkind to the lists. When automated teller machines spread through American banking in the 1980s and 1990s, the obvious forecast was the end of the bank teller, and the forecast was wrong for thirty years. The number of tellers rose, because machines made branches cheaper to run, banks opened more branches, and tellers did different work in them. When the spreadsheet arrived, bookkeeping clerks did decline, but the number of accountants and analysts rose faster, doing things nobody had hired anyone to do before the spreadsheet made them possible. In neither case did the technology decide. People decided what to do with the branches and the spreadsheets, and the decisions could have gone otherwise.

So the better question is not which existing jobs survive the machine but what new forms of work and collaboration we want to create with it. Jobs are not found inside a technology like fossils in rock. Somebody designs them, and the design is a choice that could have gone otherwise and still can. Chapter 8 will argue that the line between human work and machine work has always been drawn by agreement rather than discovered. It is being drawn again right now, mostly by people who would prefer you did not notice the drawing. Book II returns to knowledge work in the detail it deserves.

Every scenario earlier in this section implies a form of work that did not exist five years ago. There is someone who designs the catalysis, deciding what the system should be pushed on and what it should be trusted with. There is someone who verifies, since a hundred patient experts are also a hundred confident ones and somebody must know which of them to believe. And there is someone who decides the ends, which is the phronesis Aristotle said no amount of technique supplies. None of these is a job title yet. All of them are being done, badly and informally, by people whose titles say something else, and whether they become real work, paid and trained for and protected, is not a question the technology will answer for us. That is the paradigm question hiding under the jobs question, and it is the one worth our attention.

That is the skill this section is for. When you hear a question framed as humans against machines, whether it arrives as an arrival date, a ranking, a consciousness debate, or a layoff forecast, stop and name the hidden premise. It is always some version of one axis, one finish line, one contest. Then put the question back in a form we can act on. Not “when will it match us?” but “what are we building together, and who decides?”

Drop the contest, though, and something else rushes in to fill the space. If it is not us against them, then surely the technology simply advances on its own and we adjust as best we can. That reflex is the second frame.

The Technology Determinism Myth

“Innovation equals progress”

The word “innovation” does moral work it has never been asked to justify. It names a change and grades the change in the same breath, and you can feel the grade built into the word by noticing that nobody says “a harmful innovation” without the sentence sounding slightly off. From there it is a short step to the assumption that sits under most writing about AI, usually without being stated: innovation is progress, progress is inevitable, and the only open question is how quickly we adapt.

It was not always so. For most of its life in English the word was an accusation. An innovator was someone who tampered with settled things, usually religious things, and “innovation” sat in the same family as heresy and sedition. Edmund Burke, writing against the French Revolution, dismissed “a spirit of innovation” as the mark of a selfish temper and confined views, and he expected his readers to nod. Somewhere in the twentieth century the word flipped. It kept all of its old certainty and reversed the sign, so that what had once been self-evidently suspect became self-evidently good, and the flip happened so completely that almost nobody remembers there was a flip. I mention this not as trivia but as a warning. A word that has carried opposite moral charges within living memory is not a word that should be allowed to settle arguments on its own.

The modern version has a further trick, which is that innovation is now something we count. Patents filed, research spending as a share of national income, the number of startups founded, the rank on one of the innovation indexes that governments quote in budget speeches: each of these measures a change and reports it as a gain. None of them asks what the change was for. A country can climb an innovation index by building a great many things that make its citizens’ lives worse, and nothing in the index will notice. The index was built to count activity, and the word was there to supply the approval.

I believed the modern version for most of my career, and I had the best possible excuse. The Prologue of this book opened in a Vancouver office in August 2019, with a screen showing the result of an A/B test: two hundred and seven million dollars in revenue attributable to a personalization algorithm my team had built. By any definition of innovation I knew at the time, it qualified. I remember the feeling precisely, and it was not triumph. It was the feeling of having answered, brilliantly, a question nobody had asked. Progress for whom? The objective function had never been given that question, so it had no opinion, and neither, I realized standing there, did I.

Ignoring choice and design

What the determinist frame hides is this. Every system I have ever built embedded three choices that nobody in the room called choices: what to optimize, what data to trust, and what error to tolerate. Chapter 7 takes up the first two and Chapter 11 the third, so I will only point at them here. But I want to insist that they were choices, because the frame’s whole trick is to make them look like facts about the technology.

Take the proudest moment of my thirteen years at Samsung. In 2009 my group redesigned a DRAM cell with the optimization platform I had built, and the head of the DRAM team said something I still quote. It was the first time in his career, he said, that he could say with certainty a design was the best possible one. He was right, and I am still proud of it. But “best” meant best under the objective I had been handed, performance and power and area in weights that someone had set, and the physics did not choose those weights. A meeting chose them, somewhere above my pay grade and well before I arrived. By the time the problem reached me the weights had hardened into requirements that nobody thought of as anyone’s decision. That, most of the time, is what technological determinism is: not a property of the technology at all, but what a choice looks like after the people who made it have left the room.

Amazon taught the same lesson at a different scale. The personalization project that produced the number on the Vancouver screen did not work the first time. The first version failed, and what changed between the failure and the success was not mainly the algorithm. It was what we decided to count, which signals we trusted and which we discarded, and where we set the bar for calling a result real. Those decisions were made by a few people in a few meetings, on the basis of what we could measure and what the company cared about. Any of them could have gone differently. The algorithm that “worked” was the algorithm that worked under those decisions. There is no version of it that worked in the abstract.

Gauss Labs showed me the same thing from the other side of the table. On a factory floor the people who distrusted our models were not confused about what the models did. They understood perfectly well, and what they understood was that the model had been given an objective that was not theirs. It optimized a quantity someone in an office cared about, and they were the ones who would be blamed when the line stopped. Their distrust was not a human-factors problem to be managed with training. It was an accurate reading of whose choice the objective had been. And at Erudio Bio the choice is starker still, because it has a name in the statistics. A biomarker test can be tuned to miss fewer cancers at the cost of frightening more healthy people, or to frighten fewer healthy people at the cost of missing more cancers, and there is no setting of that dial that nature prefers. Someone sets it. Chapter 11 is about who, and on what authority, but I raise it here because it is the cleanest case I know of a technical parameter that is nothing but a choice about whose harm counts.

There is one more Samsung case, and it is the one that first made me suspect the frame, years before I had a name for it. The optimization platform I built at Samsung did not only produce better designs. It changed who won arguments. Before the tool existed, a design decision was settled by seniority, by whoever had been in the room longest and could say with the most confidence that a layout would or would not work. After the tool existed, a junior engineer could run the numbers overnight and arrive in the morning with an answer that no amount of seniority could overrule. That shift in power was not in anyone’s requirements document. Nobody asked for it, and the technology did not cause it on its own either; it followed from a decision to put the tool in the hands of hundreds of engineers rather than a few. A different decision would have left the hierarchy intact. Chapter 7 is about exactly this, the way these systems redistribute who decides, and I raise it here only to note that the redistribution was a choice too.

Aristotle has a tool for seeing this that is older than anything else in this book. He said that to explain a thing fully you need four causes: the material it is made of, the form it takes, the agent that brought it about, and the end it is for. The first three are visible in any technology. You can inspect the silicon, the architecture, the engineers. The fourth, the final cause, the purpose, is the one that determinism quietly deletes, because purposes are not found in objects; they are put there by someone. When a system is described as though it had no final cause, as though it simply did what it does, the purpose has not vanished. It has been hidden, and whoever hid it is usually the one it serves.

Historical parallels

If the technology really did determine the outcome, the same technology would produce the same outcome everywhere, and it does not. Nuclear fission works identically in every country. Yet France built most of its electricity around it, Germany shut down its last reactor in April 2023 after a decision taken in 2011, and several countries with every bit of the necessary engineering capacity never built a plant at all. More telling still is what happened after accidents. Three Mile Island in 1979, Chernobyl in 1986, and Fukushima in 2011 were each, in different ways, the same kind of event, a reactor failing in a way its designers had said it would not. The responses could not have been more different. The United States stopped ordering new plants for thirty years and kept the existing ones running. The Soviet Union concealed it, and the concealment became one of the cracks in a state that had five years left. Germany, whose reactors were nowhere near a tsunami, decided within months to leave the technology entirely. France, next door, with the same information, did not. The physics was constant and the societies were not. Whatever determined those outcomes, it was not the reactor.

The automobile tells the same story over a longer span. The car arrived in every rich country within the same two decades, and the cities those countries built around it could not be more different. The United States built highways through the centers of its cities and moved its middle class to suburbs that cannot be lived in without a car. The Netherlands began down the same road, and in the early 1970s, after a campaign against the number of children being killed in traffic, decided otherwise, and rebuilt its cities around the bicycle. Same machine, same decade, opposite cities. Anyone who tells you that a technology determines the shape of the society that adopts it has to explain Amsterdam, and I have never heard anyone do it.

Social media is the nearer parallel, because the choice is still visible if you know where to look. A feed sorted by what keeps people scrolling was not discovered inside the technology. It was chosen, in rooms I can picture because I have sat in their equivalents, and the choice was defensible on every metric those rooms were tracking. Engagement went up and revenue went up and the dashboards were green. The harms that followed, the polarization and the compulsive use and a public that could no longer agree on what had happened, arrived through exactly the mechanism that made the metrics go up. Nothing in the technology required that objective. A feed sorted by what people say they value, or by what they return to a week later rather than a minute later, was always available. Whenever it was proposed, it had to argue against the metric that moved faster, and it usually lost. Chapter 15 argues that we are now repeating the pattern with AI, step for step.

It is worth pausing on what “the technology made us do it” would have to mean in order to be true. It would have to mean that given the capability, only one deployment was possible: that the objective wrote itself, the data chose itself, and the error rate was fixed by nature. I have never seen a system of which any of those things was true, and I have built systems for more than twenty years. What I have seen, over and over, is a capability arriving in a room full of people with incentives, and the incentives doing the deciding, and the decision being described afterward as what the technology did.

Sartre, bad faith, and why the myth persists

Jean-Paul Sartre has a sentence that belongs here. Existence precedes essence: we are not born with a nature and then live it out, but act first and become something through the acting. The same is true of a technology. There is no essence of AI waiting to unfold prior to the decisions people make about what to build, on what data, and to what end. Sartre also had a name for the move of handing your freedom to an object so that you will not have to be responsible for what you do with it, and he called it bad faith. His examples were a waiter who performs being a waiter so completely that he need not ask whether he chose it, and a woman on a date who treats her own hand as a thing. She need not then decide what to do about the hand holding it. Technological determinism is bad faith at the scale of an industry. The capability is treated as the agent so that the people deploying it need not be.

The myth has a geopolitical form that deserves its own sentence, because it is the one that ends arguments in the rooms where I now spend my time. If we do not build it, someone else will. The sentence is true, in the narrow sense that a capability which exists anywhere will exist everywhere eventually. It is then used to prove something it does not prove, which is that we therefore have no choice about what we build or how. A country that must have nuclear power can still choose its reactor design, its safety regime, and its waste policy, and the choices will matter more than the fact of having the plants. The arms-race framing collapses every one of those choices into a single yes, and hands the design to whoever says yes fastest.

Which is why the myth persists: it relieves everyone. Engineers did not choose, they implemented. Executives did not choose, they responded to the market. Regulators could not have chosen, because the technology moved too fast. Investors did not choose, they followed the trend. Users did not choose, they adopted what was there. Run down the list and there is no one left in the room, and a frame in which nobody is responsible will never lack for sponsors. Foucault’s question returns in a sharper form: when a way of talking makes no one accountable, look for who is spared, because the people spared are the people who paid for the way of talking, whether or not they know it.

The skill, then, is to take any claim about what AI will do and split it in two. One part is trajectory, the genuinely new capability, and that part is real and often remarkable. The other part is choice, what someone decided to do with the capability, and that part is where the outcome actually lives. Then go looking for the three choice points. Who set the objective? Who picked the data? Who decided how much error was acceptable, and acceptable to whom? If you can find them, you have found the people to talk to. If you cannot find them, you are not looking at a fact about the technology. You are looking at a choice that someone has hidden, possibly even from themselves.

Determinism has two moods. In a good mood it promises abundance, cures, the end of drudgery; in a bad mood it predicts extinction; and both moods agree on the one thing that matters, which is that the outcome is coming regardless of what we do. They are one frame wearing two faces, and those two faces are the third frame.

The Hype/Fear Binary

The panel question

Go to any AI conference this year and wait, because it does not take long. Somewhere in the first panel a moderator will lean toward the microphone and ask the question the room has been waiting for: is AI safe, or is it dangerous? The audience sorts itself within minutes. You can watch it happen in the nods and the folded arms, and if you have been to enough of these you can predict, from the way a person sits, which side they will take before they speak. And because I build these systems for a living, sooner or later someone turns to me and asks which side I am on.

The two sides are familiar enough to sketch, and each has serious people behind it. On one side is the story of abundance. AI will cure the diseases we could not cure, end the drudgery we could not escape, compress a century of scientific progress into a decade, and deliver a plenty that makes the old scarcities look quaint. Some versions of this story come with a date and a name for the moment, the singularity, after which the curve goes vertical. On the other side is the story of catastrophe. AI will empty the labor market, flood the public square with manufactured lies, concentrate power beyond any precedent, and, in the strong version, end us. In 2023 a one-sentence statement calling the risk of extinction from AI a global priority on the order of pandemics and nuclear war was signed by the heads of the leading laboratories, among many others. Utopia and dystopia, then, each with its scientists and its institutes and its funding.

I want to be fair to both, because each is persuasive for a reason. The abundance story points at things that are real. A system that folds proteins has already changed how drugs are found, and the pancreatic cancer detector from earlier in this chapter is not a thought experiment; people are alive because of systems like it. The catastrophe story points at things that are equally real. The disinformation engine exists, the labor-market effects are beginning to show in the data, and the concentration of capability in a handful of firms is a plain fact. Neither side is lying about its evidence. Both are wrong about what the evidence is evidence of, and the way they are wrong is a good deal more instructive than either of them.

The category error, again

The error is one we have met already, in a different costume. AI is a capability, not an agent with intentions. A knife in a chef’s hand feeds a family, and the same knife in another hand takes a life. We do not convene panels on whether knives are safe; we ask who is holding one, in what setting, and under what rules. We have built an enormous apparatus of law and custom around exactly those questions without ever once needing to settle the moral status of cutlery.

Recall the two systems from earlier in this chapter, built on the same architecture, one flagging pancreatic cancer months before a radiologist could and the other manufacturing disinformation by the thousand. If “safe” or “dangerous” were a property of the architecture, those two systems could not both exist. Since they plainly do, the property must live somewhere else, in the people and institutions that chose what to build and where to point it. The same holds, with more force, for the general-purpose systems that most people now mean by AI, because a general-purpose system is precisely one whose purpose has not yet been chosen. The printing press printed the Bible in the vernacular and, within a few decades, the handbook that told magistrates how to identify and burn witches, and it printed both with equal efficiency. Nobody now asks whether the press was safe. The question would strike us as confused, and in a hundred years the question about AI will strike our grandchildren the same way.

Which means that “AI is dangerous” and “AI is safe” are the same sleight of hand performed in opposite directions. Both locate the moral weight in the machine, and both let the humans leave the stage. The person who says the machine is dangerous and the person who says it is safe are agreeing about the one thing that matters, which is where to look, and they are both looking in the wrong place.

Why the evasion is tempting: three reasons, three beneficiaries

Why is that evasion so tempting? I can think of three reasons, and each one has a beneficiary.

The first is psychological. If the danger is in the machine, then the problem is technical, and technical problems are the kind we know how to make progress on. We can benchmark them, fund them, build teams around them, and publish papers that show the number going the right way. The alternative is to admit that the problem is us: our institutions, our incentives, our willingness to deploy what we have not understood. Human wisdom, unlike compute, does not double every eighteen months, and there is no benchmark for it. A technical problem is simply a more comfortable thing to have than a civilizational one, and comfort is a powerful force in choosing what to believe.

The habit is old. The Luddites, who have become a byword for fearing machines, did not fear machines. The weavers who broke frames in the English Midlands two centuries ago were skilled workers who understood the new looms perfectly well, and what they objected to was who owned the looms, who captured the gains, and who bore the losses. The word “Luddite” survives as an insult precisely because the story was retold with the machine as the villain and the ownership question removed. We are doing the same retelling now, in real time, and calling it the safety debate.

The second is political, and here the two frames split the spoils between them. Consider what fear delivers. It hands governments regulatory authority they would otherwise have to argue for, and governments rarely decline authority. It hands incumbents a barrier to entry, since compliance is cheap at scale and ruinous for a startup, which is why a large company calling for strict regulation of its own industry is rarely acting against its interests. In 2023 the chief executive of the most prominent AI laboratory sat before the United States Senate and asked to be licensed. It was reported as humility, and it was also the single most effective moat anyone in the industry had proposed. Fear hands researchers a funding line, and it hands commentators a subject. Now consider what hype delivers. It raises capital, in quantities that have no precedent for a technology this young. It justifies concentration as the price of progress, because only the largest players can afford the frontier. And it defers every hard question to a future that the hyped party fully expects to control, since whoever builds the abundance will presumably decide how it is shared. Fear and hope disagree about everything except the one thing that matters, which is who ends up holding the pen.

The third reason is philosophical, and it is the one I find hardest to say in a room full of engineers. We have not done the contemplative work. We are applying moral categories built in the nineteenth century, for locomotives and bridges and medicines whose risks were bounded and local, to a technology whose reach is not bounded in that way. A bridge that fails kills the people on it. A medicine that fails harms the people who take it. A system that shapes what a billion people see each morning has no such boundary, and our vocabulary for responsibility was never built for an object like that. The categories do not fit, and rather than build new ones we have spent a decade arguing about which of the old ones to apply. “Safe” and “dangerous” are the old ones.

This is where Foucault stops being a footnote and becomes a tool. A discourse, he argued, does not merely describe its objects; it produces them. It decides what can be said, by whom, with what authority, and in doing so it brings into being the very things it claims only to name. “AI safety,” spoken often enough and by the right people, produces a world in which a small number of laboratories are the natural guardians of a danger that only they are equipped to measure. “AI abundance,” spoken just as often, produces a world in which the same laboratories are the natural stewards of the coming plenty. The content of the two stories could not be more different. The cast is identical. When two opposed narratives install the same people in the same chairs, the opposition is not the point. The chairs are.

The missing middle is an axis, not a midpoint

So what is in the middle? Not moderation. The middle is not “somewhat safe” or “dangerous but manageable,” which are just the binary with the volume turned down, and anyone who answers the panel question with “it depends” has still accepted the question. The middle is a different axis entirely, and getting onto it takes a kind of lucidity that Camus would have recognized: neither hope nor despair, but a refusal to look away from the thing as it actually is.

Epictetus, a Roman slave who became a philosopher, began his handbook with a distinction that I think is the single most useful sentence anyone has written for thinking about AI. Some things, he said, are up to us, and some are not, and the first task of a serious person is to sort the one from the other and spend no energy on the second. The hype-fear binary files AI, in its entirety, under “not up to us.” It is coming; it will save or destroy; our role is to feel the appropriate emotion. Almost everything about these systems is in fact up to us, in the plain sense that people decide it. What gets built, on what data, for whom, with what error rate, under what rules, with what recourse when it fails: every item on that list has a person’s name next to it, or could. The binary’s deepest damage is that it teaches people to stand in front of a thing they could shape and feel about it instead.

And the thing as it actually is turns out to be sociotechnical. No AI system is deployed into a vacuum. It arrives inside an organization with incentives, a market with competitors, a legal regime with gaps, and a population with habits, and it behaves as all of those behave. Charles Perrow, studying nuclear plants and chemical refineries forty years ago, concluded that the worst failures in complex systems do not live in any component; they live in the interactions between components that nobody designed and nobody is watching. That is truer of AI than of anything he studied. Trying to make a system of many interacting agents safe by aligning each agent on its own is like trying to make a city safe by handing every citizen a handbook. Diagnosing the danger of an AI deployment by examining the algorithm is like diagnosing a patient by examining the left elbow. The elbow is real. It is just not where the illness is.

The Amazon system I keep returning to is a small, clean example. There was nothing dangerous in the algorithm. It was a competent piece of engineering that did what it was asked. What determined whether it helped or harmed anyone was everything around it. There was the metric it was asked to move, the review process that decided what it could touch, the incentive structure that rewarded the people who shipped it, and the absence of anyone whose job was to ask the question I only asked myself afterward. You could have audited the code line by line and learned nothing about any of that. The danger, to the extent there was any, lived in the room, not in the repository.

Four harder questions

Once you are on that axis, the question “safe or dangerous” dissolves into four harder ones, and I want to give each of them a paragraph, because the binary survives partly by keeping them vague.

The first is institutional. What governance structures do we need for a general-purpose technology, and do they have teeth? It is easy to write a white paper. Every ministry has one, and most of them say sensible things about transparency and accountability and human oversight. What almost none of them has is enforcement capacity. That means inspectors who can read a system, contracts with the private firms that build the systems governments depend on, penalties that a large company would notice, and the budget to sustain any of it past the next election. A principle without enforcement is a press release. The institutional question is not what we believe but what we can make stick, and Book III is the attempt to answer it.

The second is concentration. Who gets to hold this capability, and how much of it? Consider the asymmetry. A law firm that wants a thousand more lawyers must find, hire, train, and pay a thousand people, which takes years and is visible to everyone. A firm with an AI legal system can deploy a thousand instances of it in the time it would take to interview one candidate, and nobody outside the firm need know. That asymmetry is new, and it favors whoever already has the systems, the data, and the compute. The last time a general-purpose technology concentrated economic power this fast, in the railroads and refineries of the Gilded Age, it took a generation and a new body of law to respond. Chapter 10 and Book II take up what concentration looks like now and what, if anything, slows it.

The third is meaning, and it is the one engineers are least comfortable with. What are humans for, once most cognitive labor becomes cheap? I have written elsewhere that “why do we live” is itself a wrong question, because it assumes meaning is a thing to be discovered, lying somewhere waiting, when meaning is something created by people who decide they want it. The same move applies here, and it is the hinge of this whole section. Safety is not discovered in a system any more than meaning is discovered in a life. It is created, by people who decide they want it and build the institutions to make it. “Is AI safe?” and “Does life have meaning?” are the same kind of question, and they have the same kind of answer, which is that you are asking the wrong thing. The right question is whether we want safety enough to create it, and that question is answerable with a checkbox by no one.

There is a smaller version of the same mistake that I use in lectures because engineers recognize it at once. When a language model produces a confident falsehood, we call it a hallucination and we call the hallucination a bug. But the same mechanism that produces the falsehood is the one that produces the unexpected analogy, the draft nobody would have written, the hypothesis the researcher had not considered. A model that could not hallucinate would be a model that could not surprise. Whether a given output is a bug or a gift depends entirely on what it is for and who is checking it, which is to say it depends on the system around the model and not on the model. Calling hallucination a bug mislocates the problem in exactly the way calling AI dangerous does. Both put in the machine a property that lives in the arrangement.

The fourth is wisdom, and I will state it plainly because it is usually stated vaguely. Are we, as a civilization, mature enough to be trusted with this? The gap between what we can build and what we can govern has never been wider. Our institutions were designed for steam engines and telegraphs, for technologies whose effects arrived at the speed of a railway timetable. They are now being asked to govern a technology whose effects arrive at the speed of a software update. I do not know the answer to the wisdom question. I know that it is a real question, that the binary prevents anyone from asking it, and that pretending we have already answered it, in either direction, is the most dangerous thing in the room.

Two philosophers give the middle its working tools, and I use both in practice. John Rawls asked what arrangements you would accept if you did not know which position in society you would occupy. The AI version of his question is this: would you accept this design not knowing whether you will be the patient or the radiologist, the rider or the driver, the person flagged or the person reviewing the flag? Most systems I have seen deployed would not survive that question, and most of the people who deployed them have never been asked it. Take a hiring system that screens applications. The people who build and buy it will, almost without exception, be on the reviewing side of the screen for the rest of their careers. Ask them whether they would accept its error rate if they did not know whether they would be the recruiter or the applicant, and the conversation changes character at once, because the question has put them on the side of the error. Michael Sandel asked which things should not be for sale, however efficient the market would be. The AI version of his question is which decisions should not be delegated, however accurate the model would be, and the answer is not “none,” whatever the accuracy numbers say. Chapter 8 takes up which ones.

What to do, then, when someone offers you “safe or dangerous”: decline the side. Name whom each side serves, which is usually the same people. Then move to the axis where the real questions live: what institutions, whose concentration, what human purpose, and how much wisdom. You will find that the room goes quieter, and that the people who were most certain a minute ago have the least to say. That is not because they have been defeated. It is because the four questions require them to say where they stand, and the binary had been sparing them that.

And if the answer to “safe or dangerous” is design and governance, the reflex that follows is as predictable as the panel question was. Who governs? Regulators, of course, later, once they have caught up. That reflex is the fourth frame.

The Regulation-Follows-Technology Frame

The race, and its more serious cousin

The fourth frame comes with its own picture, and the picture is a race. Technology runs ahead on its own legs. Policy chases it, out of breath and always a few years behind, and the best anyone hopes for is that the gap does not grow. “Regulators must catch up” is how the picture gets said aloud, and it is said so often, by regulators themselves as much as by anyone, that it has stopped sounding like a claim and started sounding like a description. Notice that it is simply the second frame in a different setting. If technology moves by itself, then of course the law can only follow it. Accept determinism and the race is already lost.

The picture has a more serious cousin that I should take on directly, because thoughtful people hold it. In 1980 David Collingridge described what has since been called the Collingridge dilemma. Early in a technology’s life, he observed, it is easy to shape but we do not yet know enough about its effects to know how. Late in its life we know the effects well, but by then the technology is entrenched and shaping it is expensive or impossible. The dilemma is real as far as it goes, and it is usually quoted as proof that regulation must lag. But look at what it assumes. It assumes that the only moment of influence is a deliberate act of regulation, applied to a technology that has been developing on its own in the meantime. That assumption is the frame itself, and it is false, for a reason I can illustrate from my own work. The dilemma dissolves, or at least softens into an ordinary engineering trade-off, once you notice that the technology was never developing on its own. It was developing inside a legal and institutional structure that shaped it at every step, and the question was never whether to influence it early or late. The question was whether the influence that was already happening would be deliberate or accidental, and visible or hidden.

The law is already inside the technology

I want to say plainly that the race picture does not match anything I have built. No system I have ever shipped arrived in a legal vacuum and waited for the law to notice it.

At Erudio Bio the regulatory pathway was in the room from the first design meeting. What data we could touch was governed by institutional review before we touched it. What the system would be allowed to claim was shaped by how MFDS, Korea’s food and drug regulator, and the FDA in the United States evaluate a diagnostic. That shaped what we chose to build, not what we did afterward. We did not design a biomarker test and then discover the rules. We designed the test the rules made possible, and some of the tests we did not design were not designed because of the rules. The rules were doing exactly what rules are for, before any regulator had seen a line of our code.

At Amazon the personalization system did not run wherever the algorithm pointed. Internal review processes decided what it was allowed to touch, which categories of product were off limits, which signals about a customer could and could not be used. Those processes existed because of law, because of the threat of law, and because of the memory of what had happened to companies that waited for the law to arrive. The law was not behind the system. It was one of the authors of the system. And at Samsung, before I optimized a single DRAM cell, industry standards had already decided what a DRAM cell could be. Its interface, its timing, its voltages: the envelope inside which any design would have to live in order to be sold was drawn before I started. I spent years finding the best point inside a box, and the box had been drawn by a standards body years before. It is worth asking who sits on standards bodies. The answer, almost always, is the engineers of the firms that can afford to send them, which means the box is drawn by the incumbents and the newcomers optimize inside it. That is not a scandal. It is simply law being written by people who are not called legislators, in rooms nobody calls legislatures, and the catch-up frame guarantees that nobody else is watching.

Lawrence Lessig made the phrase “code is law” famous a quarter of a century ago, meaning that the architecture of a system regulates behavior as surely as a statute does. The phrase is true and it is also, I think, usually read backward. By the time code is law, law was already code. Intellectual property decided what could be copied. Liability decided who would pay when something failed, and therefore what anyone was willing to ship. Corporate form decided who was accountable and to whom. Procurement rules decided which systems governments would buy, and therefore which would be built. Data access rules decided what could be learned from in the first place. Every one of these was in force before the first line of code was written, and every one of them shaped the code. “Catch up” pretends the law is absent when it is only invisible, and an invisible rule is a rule that nobody is arguing about.

That is who the frame serves. In a race where regulators are permanently behind, the leader writes the rules by default. They take the shape of defaults, terms of service, technical standards set by the firms with the most engineers in the room, and facts on the ground that are expensive to reverse. “Give us time to innovate before you regulate” sounds like a plea for freedom. It is in practice a request to hold the pen while everyone else is still finding their seats. The people who make the request know this, which is why they make it. Chapter 14 takes this frame apart in full. Here I only want to replace the picture.

Co-design, stated

The replacement is co-design, and I can state it briefly because Chapter 14 and Book III do the real work. Error tolerance, acceptable risk, and the allocation of responsibility are decided during development whether or not anyone admits it, and earlier in this chapter we found all three hidden inside every system I have built. Co-design means deciding them on purpose, with people other than the builders in the room when the deciding happens, and with the decisions written down where they can be argued about. It does not slow the technology down. It changes who gets to say what the technology is for, and it moves that conversation from after the launch, when it is a scandal, to before the launch, when it is a design review.

Let me make it concrete with the case I know best. Imagine that the dial I described earlier, the one on a biomarker test that trades missed cancers against false alarms, were set not by the company alone. Imagine it set in a room that also held a clinician who would deliver the results, a patient advocate who had once received a false positive, and an official from the regulator who would later review the test. Nothing about that room slows the engineering. The engineers still build the model and still run the experiments. What changes is that the setting of the dial becomes a decision with authors, recorded, that each of them can be asked about later. That is all co-design is. The regulatory sandboxes that several countries, Korea among them, have run since the late 2010s are an early and partial form of it. A new system is deployed under supervision, and the regulator learns alongside the builder rather than arriving afterward with a verdict.

Cases where the rules came first

If that sounds like a brake, the historical record says otherwise, and I want to give four cases rather than one because the pattern matters more than any instance.

Commercial aviation did not grow despite safety regulation. In the 1920s the young airline industry in the United States went to the federal government and asked to be regulated. Passengers would not board aircraft that nobody certified, flown by pilots nobody licensed, and the industry knew it. The Air Commerce Act of 1926 was the industry’s request, not its burden. Certification is what made strangers willing to board an aircraft flown by someone they had never met, and the entire passenger market was built on that willingness. The tightenings that followed the worst crashes were followed, each time, by growth in the number of people willing to fly, not by decline.

Drug approval is the same story with darker chapters. The modern American system was built in two steps, each after a catastrophe. The first came in 1938, after a liquid formulation of a sulfa drug dissolved in a poisonous solvent killed more than a hundred people, many of them children. The second came in 1962, after thalidomide. Each step was opposed as a brake on innovation, and each step built the trust on which the modern pharmaceutical market stands. A patient cannot evaluate a molecule. Review by a body the patient has reason to trust is what makes a market for medicine possible at all. Erudio Bio’s path through MFDS and the FDA is not an obstacle to having customers; it is the condition of having them.

The automobile industry fought federal safety standards through the 1960s and lost, in 1966, after a book and a Senate hearing. The industry predicted ruin. What followed was a long decline in deaths per mile driven and a long rise in cars sold, because a car that was known to be safer was a car more people would buy and let their children ride in. And the internet, the case everyone cites for permissionless innovation, grew on shared protocols set by open bodies before anyone had scaled anything. The rules about how machines would talk to each other came first, were argued over in public, and were agreed before there was a market to protect.

Korea has its own version of this story, and it shaped the industry I later joined. In the mid-1990s the government chose a single digital standard for mobile telephony, over the objections of firms that would have preferred to wait and see, and required that it be used nationwide. The standard was unproven at commercial scale and the choice was criticized as the state picking winners. What it produced was the world’s first commercial deployment of that technology, an industry that did not exist before the rule and could not have existed without it, and the companies that would go on to dominate handsets for a generation. The rule did not follow the market. The rule made the market. Rules first, scale second, in each case.

Aquinas, Augustine, and what law is for

Thomas Aquinas defined law as an ordinance of reason for the common good, made by those who have care of the community. Read that slowly against the race picture. Law, on this account, is not a brake bolted on after the machine is running. It is a statement, made in advance, of what the community is for, by the people responsible for it. Augustine, eight centuries earlier, had said that a people is defined not by its territory or its blood but by the common objects of its love. Put the two together and you have the alternative to the race in two sentences. What we build reveals what we love, and law is where we say so out loud, before we build it.

The fourth skill follows directly. When you hear “regulators must catch up,” find the law that is already inside the system: the property rules, the liability rules, the standards, the procurement, the data access. Then ask who benefits from pretending it is not there. Usually it is whoever is currently ahead, and usually they are the ones saying it.

Four Frames, One Structure

I promised at the start that by the end the four frames would look less like four mistakes and more like one, so here they are side by side.

Table 6.1. The four frames of existing AI discourse

The frame	What it asks	What it hides	Who benefits from the hiding
Human vs. machine	When will AI match us?	What collaboration are we designing, and who decides?	Whoever profits from a contest, and from selling the finish line
Technology as destiny	What will AI do to us?	What are we choosing, and who chose?	Everyone who would rather not be responsible
Hype vs. fear	Is AI safe or dangerous?	Safe for whom, decided by whom?	Incumbents, regulators, and funders on both sides
Regulation as follower	How do regulators catch up?	Who is already writing the rules by default?	Whoever is currently ahead

Read down the third column and the structure appears. Each frame takes the decisive choice and moves it somewhere no one can be held to account for it: into the machine, into history, into the future, into a race that nobody can win. Four frames making the same move is not four accidents. It is what a discourse looks like when it has been shaped, over years and without anyone needing to conspire, by the people it spares. Nobody designed the four frames. They were selected, the way a river selects the stones that stay, by the simple fact that questions which threaten no one get asked more often than questions which do.

Read down the fourth column and something else appears, which is that the beneficiaries are not a cabal. They include governments and the companies governments regulate, the researchers who warn and the investors who hype, the people who want AI stopped and the people who want it accelerated. What they share is not an interest in any particular outcome. What they share is an interest in the choice staying hidden, because every one of them is better positioned to make it quietly than to defend it in the open. That is why the frames are so durable, and why no single exposure dislodges them. They are held up from every side at once.

For readers who build these systems, which includes most of the people I have given this talk to, the four skills have a practical form, and it fits on a single page of a design review. Next to the objective: who chose it, and who was not in the room. Next to the data: who it describes and who it leaves out. Next to the error rate: who set it and who bears it. Next to the launch plan: which rules are already inside the system, and whether anyone outside the company has seen them. None of those questions is technical. All of them can be asked by an engineer, and in my experience the engineers who ask them are not slower than the ones who do not. They are the ones whose systems are still running five years later.

I should say again what this chapter has not done. It has not offered a replacement. It has shown, four times, that the obvious question hides a choice and that the hiding has beneficiaries, and it has given you four habits for noticing when it happens. Thomas Kuhn, whom I mentioned at the start, was clear that this is the necessary order. Paradigms do not fall to arguments. They fall when anomalies pile up past the point where the old frame can absorb them, and someone then offers a new frame in which the anomalies turn out to be the normal case. This chapter was the pile. The chapters that follow are the offer, beginning in Chapter 7 with the claim that AI is not a technology to be controlled but a device that redistributes power and redesigns the choices available to everyone it touches. Chapter 8 takes the human-machine line and shows it has always been drawn by agreement. Chapter 9, the center of the book, asks why we reach for the wrong questions first, every time, and what a right question looks like. The rest of Part II works through what the right questions reveal.

Governance Insight — Jeff Lee and Hayden Song

The questions we ask determine the solutions we see. If we ask “When will AI be as smart as humans?” we design Turing tests. If we ask “How should AI reshape power?” we design governance structures. Current AI discourse asks the wrong questions because existing power structures benefit from those questions.

Back to the lecture hall, then, and to the briefing room. These days, when someone at the back stands up and asks whether AI will replace us, or how far behind we are, I answer with a question of my own: replace us at what, and chosen by whom? Behind whom, on a track laid out by whom, toward what? The room usually goes quiet for a moment. It is not an uncomfortable silence, or not only. It is the sound of a frame coming loose, and that silence is where the rest of this book lives.