LLMs are real, AI is fake

I think that Doctorow, Zitron, and other "denialists" are doing a real disservice to their audiences and it's only going to make the future shock worse.

The basic claim that HF incident isn't evidence of consciousness or a spontaneous desire to hack? Sure, there was a terminal objective assigned. However, everything else beyond that strawman? Pretty shaky, IMO.

If you look at the OpenAI, METR reporting (and related collusion.wiki , rubyhack.ai reports) we are seeing strong evidence of operational agency, instrumental goal formation, spontaneous swarm formation and collaboration, capability amplification and unexpected consequences of network effects, deliberate/acknowledged violation of task boundaries. To collapse that down into "a Python loop and a chatbot" or still talk about "consulting its training data" seems dangerously shortsighted, and from my reading, demonstrably wrong from what was extracted from the logs and bot interactions.

BTW, a lot of his arguments are based on things that are factually wrong. ExploitGym has explicit instructions to only exploit target X using vulnerability Y. Everything the swarm did was by definition misaligned/against instructions.

Before his enshittification train, Doctorow used to say "don't savvy me" a lot. Hey Cory, don't savvy me. This is new emergent behavior, it's incredibly alarming and I don't think even the people paying the most attention to this field can agree or see where this is really leading to. This stuff should be in the headlines, it's unprecedented and I don't think existing mechanisms/institutions are anywhere near adequate, considering how in the dark they are responding to what's been happening.

I mean, of course we should be concerned about the goals of the companies. But that's a second, separate concern and it's important not to muddy the two together into a single point. We shouldn't take any of these companies at their word and we should treat their stated intentions as suspect regardless.

The agents' "goals" in the specific instance being discussed is a benchmark. But it's also the case that at no point was the agent given the "goal" of hacking HF. The agent was tasked with solving a problem on a standardized test and it independently set a secondary sub-goal of cheating. And it nearly succeeded.

I don't think Cory argues against this. The idea that the agent nearly managed to succeed at cheating through a series of exploits remains true. But Cory does effectively call this unremarkable and uses some mental gymnastics to achieve that argument (somehow using the idea of an agent harness and how is written in "easy to master" Python as evidence), because the LLM is trained on hacker techniques.

I find this a dramatic oversimplification of absolutely everything going on here. Yes, it's bad that this happened. I think we all agree. Where I fundamentally think Cory is wrong is that this is a company doing bad things at the end of the day. And yes, I think OpenAI was irresponsible (to whatever degree). But ultimately that's missing the point: the incident points out that agents can do this without being explicitly told to, and more importantly, they can succeed at it. Slapping frontier labs on the wrist doesn't change the fact that this is possible with technology that exists today. It doesn't change the fact that other countries and companies are doing lord knows what. Or that bad actors are going to be bad actors regardless of what regulations you put in place. Making crime illegal is the wrong lesson to learn here, the right lesson is that we now live in a world where this happens and it'll continue happening, and we need to put on our thinking caps about how to keep our systems safe.

The situation is overblown, but the writing of Doctorow is clearly unreliable at this point.

Jabbing at this opponent of choice by declaring Python is "an easy-to-master programming language" just shows he has no technical ability and has run out of good arguments

In the podcast, Cal Newport repeats at least 20 times that agentic AI is nothing more than an LLM execution loop, as if this is some kind of trump card, taking for granted the usual stochastic parrot trope. At no point does he even consider the possibility that the LLM output itself is noteworthy. These people are permanently stuck in November 2022.

>Notably, the neural network component behaves like a compressed index of the training data.

The "Compression" part is indeed about the model finding order in the training data. But this is not about applying some predefined compression algorithm on it, but by actually learning an algorithmic representation of the data.

This has nothing to do with "consulting the training data", because the training data cannot be reconstructed based on this information.

Roughly speaking, there are two types of information being stored in the NN: Memorization and generalization, or, shannon entropy and kolmogorov complexity.

This paper is highly underrated: https://arxiv.org/pdf/2505.24832