The Locus of Intelligence

Every passage, on the record.

1. 00:00Claim

   There is a view, widely held by people who would otherwise disagree about almost everything, that the intelligent thing in a conversation with all of the major frontier models, ChatGPT or Anthropic or Grok, is the artificial intelligence.

2. 00:32Claim

   The people building the systems think this, and the people warning we should stop building the systems also think this. When two camps that agree on almost nothing converge on a premise, the premise is usually the thing worth examining. I want to mull over the idea that they're both wrong, and that the mistake they're making is the same mistake structurally that has been made about money in the past.

3. 01:08Reference

   Using the example that Lyn Alden often touches on, the telegraph and gold, and what happens when the invention of the telegraph comes along, and how that changes how the extended order is interfacing with information.

4. 01:08Claim

   You can imagine that the discovery of the telegraph opens up a new phase space that previously was inaccessible, and everything else, all the other horns that are around, interfaces with the telegraph into this new constraint.

5. 01:47Claim

   In both cases, AI and the telegraph, the mistake was misidentifying the locus of the thing being measured. In the case of gold and the telegraph, the thing being measured was settlement. The telegraph appeared to do part of the settlement work, the communication part, but people began to act as if it did all of the settlement work, the part where the gold actually arrives.

6. 02:18Claim

   The consequence of that mistake unfolded over 170 years, until, most people would say, 1971. In the case of intelligence, the thing being measured is knowledge production. The frontier models do part of the knowledge production work, writing a plausible sounding sentence, but people now act as if it does all of the knowledge production work, the part where the sentence is true, or where there's good reason to suppose it is.

7. 02:50Prediction

   We've got maybe, depends on who you listen to, but maybe we've got 24 months or 36 months before the consequences of believing that to be true start to haunt us.

8. 02:50Claim

   The configuration that actually produces knowledge in our current moment isn't the AI, and it also isn't the unaided human either. It's a particular pairing of the two, in which the AI generates information at a rate that would have been physically impossible for a single person 20 years ago.

9. 04:00Claim

   It's just not possible to generate that much information, that arrangement of ones and zeros, at that speed. At the same time, the human is doing the one thing the AI cannot do: pay the price in attention, judgment and criticism that turns the generated sentence into a sentence that has reach.

10. 04:00Quote

    That pain is the difference between producing knowledge and producing what we might call the appearance of knowledge, which is way cheaper and almost always sufficient for getting paid.

11. 05:19Claim

    When ChatGPT produces a sentence about, say, the causes of the French Revolution, the system is doing one mechanical thing: predicting, on the basis of every sentence in its training data, what word is statistically likely to follow given the prompt. The training procedure, gradient descent on next word prediction, optimizes weights so the model's predicted next word matches the actual next word in text written by humans who, by and large, were not making things up.

12. 05:54Quote

    The system has no model of the French revolution inside of it. It has a model of what sentences about the French revolution tend to look like.

13. 05:54Claim

    The sentences it produces look like sentences that have had something done to them which, in fact, they have not had done to them. The thing that has not been done is what I would call paying the constraint.

14. 06:28Claim

    Every sentence in the training data was written by a human who, in writing that sentence, had to discard a vast number of alternative sentences that would have been wrong or misleading or off topic or grammatically broken. That winnowing is the work. The remaining sentence carries the signature of that winnowing in its shape.

15. 07:00Idea

    ChatGPT learns that signature without doing any of the filtering. The shape comes out clean, and yet the work is never paid. This is where a few episodes ago we talked about the anti-demon in Maxwell's demon thought experiment. Frontier lab LLMs, in a structural sense, they are the anti-demon, they are that demon's mirror.

16. 07:35Claim

    It's a process that appears to do something only constraint paying labor can do: producing sentences with the shape of knowledge without paying the constraint. The trick works because somebody, somewhere, already paid the constraint, the humans who wrote the training corpus. The system is just recycling the signature of their constraint.

17. 08:05Claim

    When you ask ChatGPT a question whose answer has been well discussed in its training data, it can produce a sentence indistinguishable from one written by someone who actually knew what they were talking about. But when you ask it a question whose answer wasn't well discussed, it produces a sentence that looks just the same. That's the failure mode that actually matters.

18. 08:38Claim

    The right way to see what's happening with AI now is to see it as that telegraph mistake repeated in a different substrate on a faster clock. Before the telegraph, the speed at which information about gold could travel was roughly the speed at which gold itself could travel, because both information and gold moved on ships.

19. 09:10Analogy

    A ledger entry in London that said a thousand ounces had arrived in Boston couldn't be made in any practical sense until a ship arrived in Boston and someone confirmed the gold was there. The communication channel was settlement paced. The shape of the system, in our terms, was a horn: a small fixed interior of monetary rules and an unbounded boundary of every transaction the rules touched, with a verification asymmetry that held because the physics of communication and the physics of settlement were the same physics.

20. 09:51Event

    Along comes the telegraph in the 1840s and that blows up the entire thing: for the first time, information about gold could move at near instantaneous speed while the gold itself is still stuck at ship speed.

21. 09:51Claim

    It meant the ledger entries could claim things about gold that hadn't yet been verified, and the network of people relying on those ledger entries had no way to tell whether the verification had happened or not. This is when the horn begins to fail.

22. 10:24Idea

    What grew up to replace the horn is a cylinder, a system whose interior could grow without bound because new ledger entries no longer had to wait for new gold. Fractional reserve banking and paper expansion, eventual severance from gold altogether, is what happened. That's what unfolded over the next 170 years. The people living through it, through no fault of their own, mostly didn't notice the underlying shape change, because they're busy looking at the prices of things.

23. 10:55Claim

    For nearly all of human history, the technology of producing a written claim about the world was roughly the technology of reading and criticizing it. Both happened at human speed. A serious claim written in a serious context had been criticized in the writer's own head before it ever reached the written page, because criticism happens at the same pace as writing when both are performed by the same brain.

24. 11:33Claim

    The horn that human knowledge formed depended on this. The interior of the horn was the set of established methods, editorial standards, replication conventions. The boundary was every new claim being tested against the interior. The asymmetry held up because the technology of generation and the technology of verification were on the same clock.

25. 12:04Claim

    LLMs are severing those layers apart from one another. Production is now at machine speed, but criticism is still at human speed. The horn that human knowledge formed is beginning to fail. The cylinder regime is already visible.

26. 12:38Prediction

    There's gonna be way more noise. If generating noise is significantly easier than generating signal, there's going to be a lot more noise, and this will show up with less and less knowledge being verified, which is exactly what we see happening.

27. 12:38Analogy

    Stack Overflow was basically like an LLM to answer your questions about programming, but it was other humans, a message board. Stack Overflow has completely degraded as a source of programming knowledge. Google search results have completely degraded as a source of any knowledge.

28. 13:10Event

    We've talked in the past about how there is this crisis in scientific literature because it is being flooded with AI generated submissions. Same on the legal filings.

29. 13:10Claim

    These aren't isolated incidents. They are early symptoms of the same structural transition that hit gold in the 1840s. The difference is that we don't have all of that time to work this out, because the noise is being generated so much faster than we can filter through it and try to verify the actual knowledge.

30. 14:13Idea

    Knowledge, as Popper came up with, and Deutsch has developed at length, is not wrong information. A random string of ones and zeros has lots of information by the Shannon measure and zero knowledge. The genome is the classic example: of the same length it has the same amount of Shannon information, and yet it produces an organism. The difference is constraint. The genome is information that is about something, namely the protein that it codes for and the regulatory cascades it sets in motion, and that aboutness is what makes it knowledge rather than noise.

31. 14:13Reference

    Knowledge, as we have talked about and as Popper came up with, and Deutsch has developed at length, knowledge is not wrong information.

32. 14:13Reference

    Knowledge, as we have talked about and as Popper came up with, and Deutsch has developed at length, knowledge is not wrong information.

33. 15:18Idea

    Without constraint, information is bits. Without information, constraint has nothing to act on. So knowledge requires both. And if we think about K equals I C squared, it's the C squared term that is doing the heavy lifting of telling you how much the information has been filtered down to a specific account of the world.

34. 15:18Idea

    Now intelligence is the capacity to produce knowledge. That's the definition that I want to take up and try to defend. It follows from that prior commitment because if knowledge requires both information and constraint, then any system that can only produce one of those two ingredients cannot produce knowledge by itself. It can only produce a precursor to knowledge, which then someone else has to finish.

35. 15:55Claim

    These Frontier Lab LLMs are information generators. They produce large quantities of plausible looking sentences. They do not by themselves pay the constraint that turns those sentences into knowledge. By the definition I'm proposing, ChatGPT is not intelligent. The word not is doing the work here.

36. 16:28Quote

    I'm not saying that ChatGPT is unimpressive, or that the engineers who built it didn't accomplish something extraordinary, or that it isn't useful. I'm saying that the system in chat with you is not the locus of intelligence. The intelligence, if any, is somewhere else in the loop.

37. 16:28Analogy

    AlphaFold, the protein folding system that DeepMind developed, actually is intelligent by this definition, because the loop in which it operates pays the actual constraint: a protein structure prediction can be checked against an actual protein, the predictions that fail get caught by the next round of data, and the system improves over generations because reality is paying the constraint.

38. 16:28Reference

    If you are familiar with alpha fold, alpha fold is the protein folding system that DeepMind developed.

39. 16:28Reference

    Alpha fold is the protein folding system that DeepMind developed.

40. 17:04Analogy

    A coding agent that runs alongside a compiler is intelligent in the exact same sense because the compiler is paying the constraint. If the agent generates code that doesn't compile, the compiler tells it, and the agent has to fix the code before the loop continues. The intelligence is located in the compiler-agent pair, not in the agent alone.

41. 17:04Claim

    A standard LLM in a chat window with no verification loop is not intelligent in this same sense, because no one in the loop is paying the constraint. Maybe the user pays it later when they read the output and decide whether or not to use it, but certainly the system doesn't pay it.

42. 17:38Quote

    The dominant marketing language about AGI isn't just wrong, it's actually a category error.

43. 18:18Claim

    Intelligence, as I'm using it, is not a property of a single system that can have it without reference to any verifiers. It's a property of a loop, and in that loop, constraint is being paid somewhere. AGI, in the usual sense of these conversations, claims a single system can be intelligent without a verifier. By the definition we're looking at now, that's not possible.

44. 18:52Claim · condensed

    It's not that AGI without a verifier is difficult, or that it's far away in the future. It's impossible in the same way that a Maxwell demon is impossible, which is to say it's impossible for the same reason.

45. 18:52Claim

    The first way constraint gets paid: AI generates a sentence with the shape of knowledge, the human reads it, nobody pays the C, and the sentence is consumed as if it were knowledge. That system has the structural shape of Maxwell's demon.

46. 19:24Claim

    The second case is one where the AI compresses the output of somebody that did pay the constraint: a researcher writes a long paper, the AI just summarizes it faithfully. The constraint was paid by the researcher, the AI is acting as an amplification layer on top of preexisting knowledge. That's real and useful, but it isn't new intelligence, it's compression of old intelligence.

47. 20:29Claim · condensed

    The third way is where the verifier actually sits in the loop with the AI and pays the constraint, the AlphaFold model, the coding agent with the compiler, or a robot that has to actually pick up a cup, where there's gravity and friction to pay. Physical reality is a very rigorous verifier when you give it the chance, because it pays the constraint by ruining the prediction when the prediction is wrong, forcing the system to update.

48. 21:00Idea · condensed

    The fourth way that the constraint actually gets paid, and I think this is the most interesting in the short term, is where the LLM is doing the wide exploration at machine speed, exploration that could not have been done. If you allow the AI to do the wide exploration at machine speed, that means the human can pay the constraint at the choke point of deciding what actually holds up and what doesn't. This means that the intelligence is real and it isn't located in the AI alone or in the human alone, it's this distributed pair. You can think of it almost like a Socratic foil.

49. 21:00Reference

    I liken it to Socrates because he's using that same configuration of having a questioner and having an explorer to produce some, if not the most enduring philosophical writing in Western tradition.

50. 21:34Claim

    This hybrid is something we can agree is genuinely new. This configuration was not possible, this phase space was not accessible at this scale before LLMs.

51. 22:08Claim

    A lot of the hype is pointed at the first case, where ChatGPT is just generating a sentence with the shape of knowledge and the human reads it and nobody pays the constraint. But the most productive work happening right now are cases three and four, and the huge unlock from all of this is this Socratic foil.

52. 22:41Quote

    The configuration of one skilled human paired with an LLM on a topic that the human already has a working framework for is absolutely producing knowledge at a rate that no solo human in history could match.

53. 23:11Claim

    It's way higher than what Newton could do by himself, or Einstein. It's higher than any great mathematician or philosopher of any prior era working at their peak powers could possibly produce.

54. 23:11Reference

    It's way higher than what Newton could do by himself or Einstein.

55. 23:42Claim · condensed

    If you take any solo human knowledge producer in history and ask what their rate limiting bottleneck was, there were generally four of them and they all ran at human speed: information acquisition, reading through the existing material in the field; internal processing, thinking through the implications, holding alternative theories in mind, running thought experiments; and articulation, drafting sentences, rephrasing them, finding the right words so the idea can persist in the physical universe.

56. 24:13Claim

    Finally is criticism: catching your own errors, finding your own counter examples, considering what would have to be true for the idea to be wrong. In all of the greatest thinkers, Newton, Einstein, anybody you like, all four of those bottlenecks ran in series at human speed. LLMs are collapsing three of the four to machine speed, especially for a skilled operator.

57. 24:44Claim

    Information acquisition: an LLM can hand the operator the compressed substance of a paper or a chapter or a debate in seconds. Internal processing: the LLM can run alternative theory exploration in parallel, suggesting framings the operator would never have generated or would have taken an extreme amount of time to generate. Articulation: it can render the operator's formed thoughts into clean prose, even though at this stage it's fairly obviously AI slop.

58. 25:22Claim

    No human can match it. Criticism can't be collapsed in the same way as the others, because criticism is where the constraint is actually being paid, and the constraint has to be paid by something in the loop that actually knows whether the candidate sentence is true or not. For now, in this new Socratic regime, that something is the human. Even with three out of the four, the result is a productivity rate increase that simply wasn't physically possible before LLMs existed in their current form.

59. 26:04Claim

    The bottleneck has completely moved to the operator's criticism capacity. A savvy operator can now spend their entire human time budget on the one step that actually matters, which is paying the constraint.

60. 26:41Claim

    You spend any amount of time with these things and you get better at applying the constraint, and this new rate increase is real, but it isn't evenly distributed. It requires a working framework, the discipline to actually pay the constraint rather than letting the AI output stand unverified, and the willingness to spend most of your working time on criticism rather than production.

61. 27:11Claim

    Most people are doing that first regime, letting the AI generate and then consuming it, and the constraint never gets paid. That regime is actually less productive than having no AI at all, because the operator's time is spent generating output that hasn't been constrained at all, with no idea if it's actually right.

62. 27:11Prediction

    The vast majority of information will be AI generated now. That is the onslaught, the deluge that we are all facing, a new reality that most of the information we come across is AI generated.

63. 28:21Claim · condensed

    The easiest way to figure out whether something is actually knowledge is to ask where in the loop someone paid the constraint. If you can name the verifier, you're operating in regime three or four, and that output is probably knowledge or something close to it.

64. 28:51Quote

    If you can't name the verifier, then that means that you're operating in that regime one, and that the output is information shaped like knowledge, but not knowledge. And if the answer is nowhere to the constraint question, then don't consume the output as knowledge.

65. 29:27Claim · condensed

    This is the single methodological mistake that the doomers and the AI accelerationists are both downstream of. The doomers fear unverified intelligence is going to outweigh humanity. The accelerationists hope an unverified intelligence is going to save it. But both are treating unverified intelligence as if it were intelligence, and both are wrong for the exact same reason.

66. 29:27Quote

    There's no unverified intelligence. There is no horn that can exist, no horn ecology, unless something has to press against it.

67. 30:02Claim

    If there's no constraint from peer horns, there is no knowledge, there's only information generation, and yes, it has the property of looking like intelligence, specifically to anybody that doesn't ask where the constraint was actually paid.

68. 30:02Claim

    You can turn the question on yourself: if you find yourself paying the constraint for your own outputs, whether in code, writing, research, argument or design, that means you are operating in the productive regime.

69. 30:35Claim

    If you are operating in the productive regime, the relevant question isn't whether AI will replace you, it's whether you're paying the constraint well enough. These are all learnable skills, whether you have a working framework, whether you can criticize your own output to catch your own errors, whether you can hold multiple alternatives in mind long enough to choose between them, and they're worth more now than five years ago, because the production side is free and the verification side is the bottleneck.

70. 30:35Prediction

    There are going to be a small number of people who learn to operate this Socratic foil, and they're gonna pull away so quickly from everyone else in their productive output, and that gap is going to widen until it becomes uncomfortable. It's gonna be uncomfortable to be a public commentator on AI without acknowledging that.