The threat is comfortable drift toward not understanding what you're doing • Hacker News Board

Re: The threat is comfortable drift toward not understanding what you're doing

#301

Post by ModernMech » Sun, Apr 05, 2026, 2:40 PM UTC

roenxi wrote: post #40

djaro wrote: post #14

Earlier quoted context omitted.

> So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade…

That doesn't sound like much of an issue. Bob was already going to encounter problems that are too large and complex for him to solve, agents or otherwise. Life throws us hard problems. I don't recall if we even assumed Bob was unusually capable, he might be one of life's flounderers. I'd give good odds that if he got through a program with the help of agents he'll get through life achieving at least a normal level o…

> Not going to win that battle in the long term.

I would take that bet on the side of the wet meat. In the future, every AI will be an ad executive. At least the meat programming won't be preloaded to sell ads every N tokens.

Re: The threat is comfortable drift toward not understanding what you're doing

#302

Post by piiritaja » Sun, Apr 05, 2026, 2:41 PM UTC

caxap wrote: post #292

If this article was written a year ago, I would have agreed. But knowing what I know today, I highly doubt that the outcomes of LLM/non-LLM users will be anywhere close to similar. LLMs are exceptionally good at building prototypes. If the professor needs a month, Bob will be done with the basic prototype of that paper by lunch on the same day, and try out dozens of hypotheses by the end of the day. He will not be ch…

"LLMs won't take anything away if you are still willing to take the time to understand what it's actually building"

But do you actually understand it? The article argues exactly against this point - that you cannot understand the problems in the same way when letting agents do the initial work as you would when doing it without agents.

from the article: "you cannot learn physics by watching someone else do it. You have to pick up the pencil. You have to attempt the problem. You have to get it wrong, sit with the wrongness, and figure out where your reasoning broke. Reading the solution manual and nodding along feels like understanding. It is not understanding. Every student who has tried to coast through a problem set by reading the solutions and then bombed the exam knows this in their bones. We have centuries of accumulated pedagogical wisdom telling us that the attempt, including the failed attempt, is where the learning lives. And yet, somehow, when it comes to AI agents, we've collectively decided that maybe this time it's different. That maybe nodding at Claude's output is a substitute for doing the calculation yourself. It isn't. We knew that before LLMs existed. We seem to have forgotten it the moment they became convenient."

Re: The threat is comfortable drift toward not understanding what you're doing

#303

Post by ThrowawayR2 » Sun, Apr 05, 2026, 2:41 PM UTC

NiloCK wrote: post #162

jacquesm wrote: post #21

Earlier quoted context omitted.

Precisely. The first 10 rungs of the ladder will be removed, but we still expect you to be able to get to the roof. The AI won't get you there and you won't have the knowledge you'd normally gain on those first 10 rungs to help you move past #10.

People would have said the same about graphing calculators or calculators before that. Socrates said the same thing about the written word. The determining factor is always "did I come up with this tool". Somehow, subsequent generations always manage to find their own competencies (which, to be fair, may be different). This isn't guaranteed to play out, but it should be the default expectation until we actually see g…

The calculator analogy is wrong for the same reason. Knowing and internalizing arithmetic, algebra, and the shape of curves, etc. are mathematical rungs to get to higher mathematics and becoming a mathematician or physicist. You can't plug-and-chug your way there with a calculator and no understanding.

The people who make the calculator analogy are already victims of the missing rung problem and they aren't even able to comprehend what they're lacking. That's where the future of LLM overuse will take us.

Re: The threat is comfortable drift toward not understanding what you're doing

#304

Post by b00ty4breakfast » Sun, Apr 05, 2026, 2:43 PM UTC

roenxi wrote: post #105

b00ty4breakfast wrote: post #104

Earlier quoted context omitted.

>But there is also a more subtle thing, which is we're trending towards superintelligence with these AIs do you have any evidence for that, though? Besides marketing claims, I mean.

I've always quite liked https://ourworldindata.org/grapher/test-scores-ai-capabiliti... to show that once AIs are knocking at the door of a human capability they tend to overshoot in around a decade.

we have to look at what LLMs are and are not doing for this to be applicable; they are not "thinking", there is no real cognition going on inside an LLM. They are making statistical connections between data points in their training sets. Obviously, that has born some pretty interesting (and sometimes even useful) results but they are not doing anything that any reasonably informed person would call "intelligent" and certainly not "super intelligent".

Re: The threat is comfortable drift toward not understanding what you're doing

#305

Post by tonfa » Sun, Apr 05, 2026, 2:43 PM UTC

bigfishrunning wrote: post #213

CuriouslyC wrote: post #142

Earlier quoted context omitted.

Anthropic has shared that API inference has a ~60% margin. OpenAI's margin might be slightly lower since they price aggressively but I would be surprised if it was much different.

Is that margin enough to cover the NRE of model development? Every pro-AI argument hinges on the models continuing to improve at a near-linear rate

Yeah but the argument people make is that when the music stops cost of inference goes through the roof.

I could imagine that when the music stops, advancement of new frontier models slows or stops, but that doesn't remove any curent capabilities.

(And to be fair the way we duplicate efforts on building new frontier models looks indeed wasteful. Tho maybe we reach a point later where progress is no longer started from scratch)

Re: The threat is comfortable drift toward not understanding what you're doing

#306

Post by HauntingPin » Sun, Apr 05, 2026, 2:45 PM UTC

sumeno wrote: post #189

stavros wrote: post #49

Earlier quoted context omitted.

Seems that teaching Bob trivial things would be a simple solution to this predicament.

That's what the program he just took was supposed to be for, learning not output. You've just reinvented the article from first principles, congrats

Sometimes I wonder how deeply some people actually read these articles. What's the point of the comments if all we're doing is re-explaining what's already explained in such a precise and succint manner? It's a fantastic article. It's so well-written and clear. And yet we're stuck going in a circle repeating what's in the article to people who either didn't read it, or didn't read it with the care it deserves.

Re: The threat is comfortable drift toward not understanding what you're doing

#307

Post by ozim » Sun, Apr 05, 2026, 2:45 PM UTC

sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

There is still a lot of engineering to be done with LLMs. Maybe not exactly writing code but I think a lot of optimization problems will be there no matter what.

Some people treat toilet as magic hole where they throw stuff in flush and think it is fine.

If you throw garbage in you will at some point have problems.

We are in stage where people think it is fine to drop everything into LLM but then they will see the bill for usage and might be surprised that they burned money and the result was not exactly what they expected.

Re: The threat is comfortable drift toward not understanding what you're doing

#308

Post by lapcat » Sun, Apr 05, 2026, 2:46 PM UTC

simianwords wrote: post #294

lapcat wrote: post #260

Earlier quoted context omitted.

> I'm simply claiming that not everyone needs to be Linux Kernel specialists. This is an uninteresting and indeed silly claim, because nobody has ever asserted the opposite. The point is that society needs some Linux kernel specialists, and some astrophysicists, but AI is undermining their production. > This is precisely what AI is allowing: it automates things I don't need to know deeply so that I can focus on thing…

I'm responding to this quote > Frank Herbert (yeah, I know I'm a nerd), in God Emperor of Dune, has a character observe: "What do such machines really do? They increase the number of things we can do without thinking. Things we do without thinking; there's the real danger." Herbert was writing science fiction. I'm writing about my office. The distance between those two things has gotten uncomfortably small. If we bot…

> I'm responding to this quote

> > Frank Herbert (yeah, I know I'm a nerd), in God Emperor of Dune, has a character observe

The article author and I share a love of Frank Herbert, God Emperor of Dune, and the quote in question. Nonetheless, it's a mistake to focus on this quote rather than on the rest of the article. The quote is nothing more than a nice literary reference; it's not central to the argument.

The character who spoke the quote is a magically prescient human-sandworm hybrid, thousands of years old, speaking to his distant relative who was specially bred by him to be invisible to the magical prescience, so let's take the quote with a grain of... sand. ;-)

> If we both agree that an astrophysicist may not need to understand things (even in their own domain) to make progress then we are in agreement.

Your parenthetical remark is actually the main problem!

Re: The threat is comfortable drift toward not understanding what you're doing

#309

Post by Jensson » Sun, Apr 05, 2026, 2:47 PM UTC

thepasch wrote: post #212

lelanthran wrote: post #158

Earlier quoted context omitted.

> The question is whether it’s more important to be able to do things, or more important to have a good sense and a keen eye for what to do at any given moment. Those aren't mutually exclusive. "People who do things" can do both, and doing the latter is a function of doing the former, so they tend to do the latter sufficiently well. "People who prompt things" can only do the latter, and they routinely do it poorly.

> “People who prompt things” can only do the latter, and they routinely do it poorly. Right, but what I don’t agree with here is the idea that this category of people will never be able to improve into the first category of people. The value of an experienced anything is that they realize there is a big chasm between something that works now and something that will continue to work long into the future. I don’t agree…

> I don’t agree that doing everything yourself manually is the only thing that can grant you that understanding, because I don’t think that understanding is domain-specific. It evolves naturally as soon as someone realizes that their list of unknown unknowns is FAR larger than their list of known anythings, and that the first step in attempting to solve a problem is to prune that list as far as you can get it while realizing you will never ever be able to reduce it to zero.

This isn't true, a car mechanic never evolves into an engineer, a nurse never evolve into a doctor. A car mechanic can learn to do some tasks you normally need an engineer for and same with nurses, but they never build the entire core set of skills that separates engineers from mechanics and doctors from nurses.

There are maybe some exceptions to this, but those exceptions are so rare that it doesn't matter for this discussion. A few people still learning it properly wont save anything.

Re: The threat is comfortable drift toward not understanding what you're doing

#310

Post by conception » Sun, Apr 05, 2026, 2:47 PM UTC

Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

Sadly I don’t see how our current social paradigm works for this. There is no history of any sort of long planning like this or long term loyalty (either direction) with employees and employers for this sort of journeyman guild style training. AI execs are basically racing, hoping we won’t need a Schwartz before they are all gone. But what incentives are in place to high a college grad, have them work without llms for a decade and then give them the tools to accelerate their work?