[Paradigm Reset] Chapter 6: What Existing AI Discourse Gets Wrong
For about three years now I have been giving more or less the same talk in different rooms. Stanford, San Jose State, Texas A&M, Seoul National University, Korea University, KAIST, POSTECH. The slides change. The audiences change. The first question from the back of the room does not. Someone stands up, usually a student, sometimes a professor who has waited politely through forty minutes of convex optimization and cancer biomarkers, and asks: “Will AI replace us?” When it is not that question, it is its twin: “So AI is just a tool, right?”
For a while I answered. I had a good answer. I had charts. Then I noticed that however I answered, the room left with the picture it had come in with: humans on one side, machines on the other, a line between them, and the only question being where the line would sit next year. My answer had not touched the picture. It had decorated it.
This chapter is about why I stopped answering that question, and three others like it.
Part I of this book was the story of a pattern. At Samsung I found the best possible DRAM design and never asked who had decided what “best” meant. At Amazon I helped ship an algorithm worth two hundred and seven million dollars and could not say whether it had made anyone’s life better. At Gauss Labs I watched people on a factory floor refuse to trust an algorithm. At Erudio Bio I found that before health data can save a life, someone has to answer who owns it. Each time the technical question got answered and the real question was never asked. Part II asks why, and the honest place to start is with the questions everyone else is asking, because those are the ones that crowded out the real one.
There are four of them. Human versus machine. Technology as destiny. Hype versus fear. Regulation as follower. Each one sounds like a question. Each one is also a frame, and a frame decides in advance what will count as an answer.
Wittgenstein wrote that the limits of my language mean the limits of my world. He was talking about logic, but the sentence has a plainer use. A question is a piece of language. If the only questions available about AI are these four, then there are things about AI that cannot be thought, not because they are false but because there is no sentence to think them in. Thomas Kuhn observed that most of the time scientists solve puzzles inside a paradigm and treat whatever does not fit as noise; the four frames are AI’s normal science. And Michel Foucault asked a question I will keep returning to: when one way of asking becomes the obvious way, who benefits from its being obvious?
Four frames, then, taken one at a time. For each I will say what it asks, what it hides, and who gains from the hiding. By the end the four should look less like four mistakes and more like one.
The Human vs. Machine Trap
Start with the version of the question that has a date attached. When will artificial general intelligence arrive? The answers range from “already here” to “never,” and a great deal of money and anxiety rides on the spread. Set the answers aside and look at the question itself. “When will AI arrive at X” presupposes a finish line. A finish line presupposes a single track. A single track presupposes that intelligence is one quantity, that it comes in amounts, and that human beings sit at a known mark on the scale.
Every part of that is doing work. In 1997 a machine beat the world chess champion, and chess stopped being the measure of intelligence roughly the following week. In 2016 the same thing happened to Go. In 2020 it happened to protein folding, a problem that had defeated laboratories for fifty years. Nobody held a meeting to move the goalposts. They moved on their own, which is what goalposts do when they were never anchored to anything. The date keeps arriving and never arrives. When a forecast behaves that way for decades, the lesson is not that the forecasters are bad. It is that the thing being forecast is not a thing.
The fallacy underneath is easy to see once you put two systems side by side. Both are built on the Transformer, the architecture behind most of what now gets called AI. One reads medical images and flags pancreatic cancer months before a radiologist would catch it. The other produces disinformation at industrial scale, thousands of plausible articles an hour. Which one is more intelligent? The question has no answer, and not because we lack data. Placing both on a single axis erases the only fact about them that matters, which is what each was built to do and for whom. The intelligence spectrum is a ruler laid across things that do not share a dimension.
A footnote worth saying out loud: nobody actually uses a large language model. People use systems with a model embedded inside, wrapped in retrieval, tools, policies, and an interface. And the model at the core is, in plain terms, a very good estimator of which word is likely to come next. That is a description, not an insult, and it makes “how smart is it” the wrong unit of measure, the way “how strong is it” is the wrong unit for a bridge.
The trap’s deepest room is the consciousness debate. Does it understand? Is it aware? Gilbert Ryle had a name for what goes wrong here: the category mistake, the visitor who is shown every college in Oxford and then asks where the university is. Edmund Husserl supplies the other half. Consciousness is always consciousness of something, and it is known from within. A question that can only be settled from the inside cannot ground a decision that has to be made from the outside. Whatever the truth about machine awareness, no regulator, hospital, or court will ever have access to it. For our purposes the question is not wrong so much as idle, and I am going to leave it where it lies.
What I cannot leave alone is the trap’s most popular form, because I fell into it myself. Replace or assist? Will the machine take the job or help with the job?
Let me tell you what actually happened to me. I finished my PhD at Stanford under Stephen Boyd with a command of convex optimization that fewer than a hundred people in the world share, and with a private embarrassment: I did not understand duality the way Boyd did. I could use it and teach it. I could not see it. That stayed true for more than a decade. Then, over a stretch of weeks, I spent hours at a time in dialogue with an AI system about duality. I asked, it answered, I pushed back, because it was often wrong in instructive ways. I corrected it, added what I knew, asked again. Somewhere in that back-and-forth the thing I had been missing clicked into place. The system did not understand duality; I want to be exact about that. What it gave me was the experience of arguing with a hundred versions of the handful of people who could have argued back, all of them patient, none of them busy. The understanding happened in me. Nothing happened in it.
Neither “replace” nor “assist” describes that. Replace is wrong on its face; I was not replaced, I was changed. Assist is an understatement that misses the mechanism. The right word comes from chemistry. A catalyst lowers the barrier to a reaction and comes out the other side unchanged; the transformation happens entirely in the reactant. Call it cognitive catalysis. And notice what the structure implies. In every version of this story the human supplies what the system lacks: the domain, the question, the stubbornness, the desire to understand. A stronger catalyst does not make those less valuable. It makes them decisive.
The jobs question is the same trap with a salary attached. Which jobs are safe? Which will disappear? Ask it and you can only get back lists, winners and losers sorted by occupation, wrong in the way AGI dates are wrong and for the same reason. The better question is not which existing jobs survive the machine but what new forms of work and collaboration we want to create with it. Jobs are not found inside a technology like fossils in rock. Somebody designs them, and the design is a choice. Chapter 8 will argue that the line between human work and machine work has always been drawn by agreement rather than discovered.
That is the skill this section is for. When you hear a question framed as humans against machines, whether an arrival date, a ranking, a consciousness debate, or a layoff forecast, stop and name the hidden premise: one axis, one finish line, one contest. Then put the question back in a form we can act on. Not “when will it match us?” but “what are we building together, and who decides?”
Drop the contest and something else rushes in to fill the space. If it is not us against them, then surely the technology simply advances and we adjust. That reflex is the second frame.
The Technology Determinism Myth
The word “innovation” does moral work it has never been asked to justify. It names a change and grades the change in the same breath. Nobody says “a harmful innovation” without the sentence sounding slightly off, which tells you the grade was built into the word. From there it is a short step to the assumption under most writing about AI: innovation is progress, progress is inevitable, and the only question is how quickly we adapt.
I believed a version of this for most of my career, and I had the best possible excuse. The Prologue opened in a Vancouver office in August 2019, with a screen showing the result of an A/B test: two hundred and seven million dollars in revenue attributable to a personalization algorithm my team had built. By any definition of innovation I knew, it qualified. I remember the feeling precisely, and it was not triumph. It was the feeling of having answered a question nobody had asked. Progress for whom? The objective function had not been given that question, so it had no opinion, and neither, I realized, did I.
What the determinist frame hides is this. Every system I have ever built embedded three choices that nobody in the room called choices. What to optimize. What data to trust. What error to tolerate. Chapter 7 takes up the first two and Chapter 11 the third. They were choices, and the frame’s whole trick is to make them look like facts about the technology.
Take the proudest moment of my thirteen years at Samsung. In 2009 my group redesigned a DRAM cell with the optimization platform I had built, and the head of the DRAM team said something I still quote: it was the first time he could say with certainty that a design was the best possible one. He was right, and I am still proud of it. But “best” meant best under the objective I had been handed: performance, power, area, in weights someone had set. The physics did not choose those weights. A meeting chose them, and by the time I arrived the meeting was over and the weights had hardened into requirements. That is what technological determinism is, most of the time. It is what a choice looks like after the people who made it have left the room.
If the technology really did determine the outcome, the same technology would produce the same outcome everywhere. It does not. Nuclear fission works identically in every country. France built most of its electricity around it. Germany, after decades of argument and a vote, shut down its last reactor in 2023. Several countries with the engineering capacity never built a plant. The physics was constant; the societies were not. Whatever determined those outcomes, it was not the reactor.
Social media is the nearer parallel, because the choice is still visible if you look. A feed sorted by what keeps people scrolling was not discovered inside the technology. It was chosen, in rooms I can picture because I have sat in their equivalents, and the choice was defensible on every metric those rooms were tracking. Engagement went up. Revenue went up. The harms that followed, polarization, compulsive use, a public that could no longer agree on what had happened, arrived through exactly the mechanism that made the metrics go up. Nothing in the technology required that objective. Chapter 15 argues that we are repeating the pattern with AI, step for step.
Jean-Paul Sartre has a sentence that belongs here. Existence precedes essence: we are not born with a nature and then live it out; we act first and become something through the acting. The same is true of a technology. There is no essence of AI waiting to unfold prior to the decisions people make about what to build, on what data, to what end. Sartre also had a name for handing your freedom to an object so that you will not have to be responsible for it. He called it bad faith. Technological determinism is bad faith at the scale of an industry.
Which is why the myth persists. It relieves everyone. Engineers did not choose; they implemented. Executives did not choose; they responded to the market. Regulators could not have chosen; the technology moved too fast. A frame in which nobody is responsible will never lack for sponsors. Foucault’s question returns: when a way of talking makes no one accountable, look for who is spared.
The skill, then. Take any claim about what AI will do and split it in two: the part that is trajectory, the new capability, and the part that is choice, what someone decided to do with it. Then find the three choice points. Who set the objective? Who picked the data? Who decided how much error was acceptable? If you cannot find them, you are not looking at a fact about the technology. You are looking at a choice someone has hidden, perhaps even from themselves.
Determinism has two moods. In a good mood it promises abundance, cures, the end of drudgery. In a bad mood it predicts extinction. Both agree the outcome is coming regardless of what we do. They are one frame wearing two faces, and those two faces are the third frame.
The Hype/Fear Binary
Go to any AI conference this year and wait. It does not take long. Somewhere in the first panel a moderator will lean toward the microphone and ask the question the room has been waiting for: is AI safe, or is it dangerous? The audience sorts itself within minutes. You can watch it happen, the nods and the folded arms. And because I build these systems for a living, sooner or later someone turns to me and asks which side I am on.
The two sides are familiar enough to sketch in a sentence each. On one, AI cures the diseases we could not cure, ends the drudgery we could not escape, and delivers an abundance that makes the old scarcities quaint. On the other, it empties the labor market, floods the public square with manufactured lies, and, in the strong version, ends us. Utopia and dystopia. Both are wrong in the same way, and the way they are wrong is more instructive than either of them.
The error is one we have met already, in a different costume. AI is a capability. It is not an agent with intentions. A knife in a chef’s hand feeds a family; the same knife in another hand takes a life. We do not convene panels on whether knives are safe. We ask who is holding one, in what setting, under what rules. Recall the two systems from earlier in this chapter, built on the same architecture, one flagging pancreatic cancer and the other manufacturing disinformation. If “safe” or “dangerous” were a property of the architecture, those two could not both exist. They do. So the property lives somewhere else, in the people and institutions that chose what to build and where to point it.
Which means that “AI is dangerous” and “AI is safe” are the same sleight of hand performed in opposite directions. Both locate the moral weight in the machine. Both let the humans leave the stage.
Why is that evasion so tempting? I can think of three reasons, and each one has a beneficiary.
The first is psychological. If the danger is in the machine, the problem is technical, and technical problems are the kind we know how to make progress on: benchmark them, fund them, build teams around them. The alternative is to admit that the problem is us, and human wisdom, unlike compute, does not double every eighteen months. A technical problem is a more comfortable thing to have than a civilizational one.
The second is political, and here the two frames split the spoils. Fear hands governments regulatory authority they would otherwise have to argue for. It hands incumbents a barrier to entry, because compliance is cheap at scale and ruinous for a startup; a large company that calls for strict regulation is rarely acting against its interests. It hands researchers a funding line. Hype does the mirror work. It raises capital. It justifies concentration as the price of progress. It defers every hard question to a future that the hyped party expects to control. Fear and hope disagree about everything except the one thing that matters, which is who ends up holding the pen.
The third is philosophical. We have not done the contemplative work. We are applying moral categories built in the nineteenth century, for locomotives and bridges and medicines whose risks were bounded and local, to a technology whose reach is not bounded in that way. The categories do not fit, and rather than build new ones we have been arguing about which of the old ones to apply.
This is where Foucault stops being a footnote and becomes a tool. A discourse, he argued, does not merely describe its objects; it produces them. “AI safety,” spoken often enough, produces a world in which a small number of laboratories are the natural guardians of a danger that only they are equipped to measure. “AI abundance,” spoken often enough, produces a world in which the same laboratories are the natural stewards of the coming plenty. The content of the two stories could not be more different. The cast is identical.
So what is in the middle? Not moderation. The middle is not “somewhat safe” or “dangerous but manageable.” It is a different axis, and getting onto it takes a kind of lucidity Camus would have recognized: neither hope nor despair, just a refusal to look away from the thing as it actually is.
And the thing as it actually is turns out to be sociotechnical. No AI system is deployed into a vacuum. It arrives inside an organization with incentives, a market with competitors, a legal regime with gaps, a population with habits. Trying to make a system of many interacting agents safe by aligning each agent on its own is like trying to make a city safe by handing every citizen a handbook. Diagnosing the danger of an AI deployment by examining the algorithm is like diagnosing a patient by examining the left elbow. The elbow is real. It is not where the illness is.
Once you are on that axis, the question “safe or dangerous” dissolves into four harder ones. What institutions do we need for a general-purpose technology, with real enforcement capacity rather than white papers? Who gets to concentrate it, when a single firm can deploy a thousand instances of an AI legal system in the time it would take to interview one lawyer? What are humans for, when most cognitive labor becomes cheap? And are we, as a civilization, mature enough to be trusted with this? Chapters 7 and 10 take up the first two, Chapter 13 what autonomy means under these systems, and the last two run under the whole series. None has a checkbox answer, which is exactly why the binary is so attractive and so useless.
Two philosophers give the middle its working tools. John Rawls asked what arrangements you would accept if you did not know which position in society you would occupy. Ask that of an AI system: would you accept this design not knowing whether you will be the patient or the radiologist, the rider or the driver, the person flagged or the person reviewing the flag? Michael Sandel asked which things should not be for sale, however efficient the market. The AI version is which decisions should not be delegated, however accurate the model. Chapter 8 takes up which ones.
What to do, then, when someone offers you “safe or dangerous”: decline the side. Name whom each side serves, which is usually the same people. Then move to the axis where the real questions live: what institutions, whose concentration, what human purpose, how much wisdom.
And if the answer to “safe or dangerous” is design and governance, the reflex that follows is as predictable as the panel question was. Who governs? Regulators, of course. Later. Once they have caught up. That reflex is the fourth frame.
The Regulation-Follows-Technology Frame
The fourth frame comes with its own picture, and the picture is a race. Technology runs ahead on its own legs. Policy chases it, out of breath, always a few years behind, and the best anyone hopes for is that the gap does not grow. “Regulators must catch up” is how the picture gets said aloud. Notice that it is simply the second frame in a different setting. If technology moves by itself, then of course the law can only follow it. Accept determinism and the race is already lost.
I want to say plainly that the picture does not match anything I have built. No system I have ever shipped arrived in a legal vacuum and waited for the law to notice it. At Erudio Bio the regulatory pathway was in the room from the first design meeting. What data we could touch was governed by institutional review before we touched it. What the system would be allowed to claim was shaped by how MFDS, Korea’s food and drug regulator, and the FDA in the United States evaluate a diagnostic, and that shaped what we chose to build, not what we did afterward. At Amazon, personalization did not run wherever the algorithm pointed; internal review processes decided what it was allowed to touch. At Samsung, before I optimized a single DRAM cell, industry standards had already decided what a DRAM cell could be.
The law is not behind the technology. It is inside it. Intellectual property decides what can be copied. Liability decides who pays when something fails. Corporate form decides who is accountable and to whom. Procurement rules decide which systems governments will buy. Data access rules decide what can be learned from. Every one of these was in force before the first line of code, and every one shaped the code. “Catch up” pretends the law is absent when it is only invisible, and an invisible rule is a rule nobody is arguing about.
That is who the frame serves. In a race where regulators are permanently behind, the leader writes the rules by default, in the shape of defaults, terms of service, and facts on the ground that are expensive to reverse. “Give us time to innovate before you regulate” sounds like a plea for freedom. It is a request to hold the pen while everyone else is still finding their seats. Chapter 14 takes this frame apart in full. Here I only want to replace the picture.
The replacement is co-design, and I can state it in three sentences because Chapter 14 and Book III do the work. Error tolerance, acceptable risk, and the allocation of responsibility are decided during development whether or not anyone admits it; earlier in this chapter we found them hidden inside every system I have built. Co-design means deciding them on purpose, with people other than the builders in the room. It does not slow the technology down. It changes who gets to say what the technology is for.
If that sounds like a brake, the historical record says otherwise. Commercial aviation did not grow despite safety certification; certification is what made strangers willing to board an aircraft flown by someone they had never met, and the passenger market was built on that willingness. Drug approval is the same story. A patient cannot evaluate a molecule. Review by a body the patient has reason to trust is what makes a market for medicine possible, and Erudio Bio’s path through MFDS and the FDA is not an obstacle to having customers; it is the condition of having them. The internet grew on shared protocols set by open bodies before anyone had scaled anything. Rules first, scale second, in each case.
Thomas Aquinas defined law as an ordinance of reason for the common good, made by those who have care of the community. Not a brake bolted on after the machine is running. A statement, made in advance, of what the community is for. Augustine, eight centuries earlier, had said that a people is defined by the common objects of its love. Put the two together and you have the alternative to the race. What we build reveals what we love, and law is where we say so out loud, before we build it.
The fourth skill follows directly. When you hear “regulators must catch up,” find the law that is already inside the system: the property rules, the liability rules, the standards, the procurement, the data access. Then ask who benefits from pretending it is not there. Usually it is whoever is currently ahead.
Four Frames, One Structure
I promised at the start that by the end the four frames would look less like four mistakes and more like one. Here they are side by side.
Table 6.1. The four frames of existing AI discourse
| The frame | What it asks | What it hides | Who benefits from the hiding |
|---|---|---|---|
| Human vs. machine | When will AI match us? | What collaboration are we designing, and who decides? | Whoever profits from a contest, and from selling the finish line |
| Technology as destiny | What will AI do to us? | What are we choosing, and who chose? | Everyone who would rather not be responsible |
| Hype vs. fear | Is AI safe or dangerous? | Safe for whom, decided by whom? | Incumbents, regulators, and funders on both sides |
| Regulation as follower | How do regulators catch up? | Who is already writing the rules by default? | Whoever is currently ahead |
Read down the third column and the structure appears. Each frame takes the decisive choice and moves it somewhere no one can be held to account for it: into the machine, into history, into the future, into a race nobody can win. Four frames, one move. That is not four accidents. It is what a discourse looks like when it has been shaped, over years and without anyone needing to conspire, by the people it spares.
Thomas Kuhn, whom I mentioned at the start, was clear that paradigms do not fall to arguments. They fall when anomalies pile up past the point where the old frame can absorb them, and someone offers a new frame in which the anomalies are the normal case. This chapter was the pile. The chapters that follow are the offer, beginning in Chapter 7 with the claim that AI is not technology to be controlled but a device that redistributes power and redesigns the choices available to everyone it touches.
Governance Insight — Jeff Lee and Hayden Song
The questions we ask determine the solutions we see. If we ask “When will AI be as smart as humans?” we design Turing tests. If we ask “How should AI reshape power?” we design governance structures. Current AI discourse asks the wrong questions because existing power structures benefit from those questions.
Back to the lecture hall, then. These days when someone at the back stands up and asks whether AI will replace us, I answer with a question of my own. Replace us at what, and chosen by whom? The room usually goes quiet for a moment. That silence is where the rest of this book lives.