The threat is comfortable drift toward not understanding what you're doing • Hacker News Board

Re: The threat is comfortable drift toward not understanding what you're doing

#631

Post by iainctduncan » Mon, Apr 06, 2026, 6:18 PM UTC

It is astonishing to me how many people here are defending "not actually knowing what the hell you are doing", on the basis that LLMs will "keep getting better".

The bit they are missing, IMHO, is that if LLMs keep getting better, doing the steering-the-LLM version keeps getting easier, and being an LLM-using-expert rapidly drops to a value of zero. Like literally fucking nothing. If anyone can do it easily with an LLM, why would anyone pay you anything? Why would they care? You might as well be a teenager at a fast food job.

In this scenario, only actually understanding will be any kind of differentiator at all. And if that scenario doesn't come to pass, it will still be a far better differentiator to have a clue.

Re: The threat is comfortable drift toward not understanding what you're doing

#632

Post by TrnsltLife » Mon, Apr 06, 2026, 6:24 PM UTC

conception wrote: post #310

Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

Sadly I don’t see how our current social paradigm works for this. There is no history of any sort of long planning like this or long term loyalty (either direction) with employees and employers for this sort of journeyman guild style training. AI execs are basically racing, hoping we won’t need a Schwartz before they are all gone. But what incentives are in place to high a college grad, have them work without llms fo…

I guess that Anathem novel was prescient. We need academic monasteries where people go in and have no access to tech for a period of time. At least no LLMs/AI. They think with their heads and their chalkboard and their sand pits like the geometers of old.

Re: The threat is comfortable drift toward not understanding what you're doing

#633

Post by Nasrudith » Mon, Apr 06, 2026, 6:34 PM UTC

adriand wrote: post #436

conception wrote: post #397

Earlier quoted context omitted.

These movements are growing in every western nation. The trend has been growing over decades. It would be nice to see it reverse but seems unlikely before calamity.

It’s a deliberate process powered by rightwing and capitalist interests designed to create a dumber, less educated and more distracted population. A war as stupid as the one with Iran would not have been possible three decades ago. As ill-advised as the Iraq war was, Bush at least spent months explaining the rationale and building support for it, successfully. Now that’s not needed. I saw interviews with young Americ…

That canard about a vast conspiracy to explain why people don't agree with you? It sounds exactly like conservatives blaming the liberal schools for in essence the same thing, not agreeing with them.

Schools must be quite busy and schizophrenic what with the completely ideologically inconsistent indoctrination they are apparently doing.

Not to mention judging engagement in spring break goers is kind of a major selection bias. Those who are upset about it wouldn't be in the mood to vacation.

Re: The threat is comfortable drift toward not understanding what you're doing

#634

Post by throwaway27448 » Mon, Apr 06, 2026, 7:12 PM UTC

mrob wrote: post #614

throwaway27448 wrote: post #566

Earlier quoted context omitted.

Also entirely ineffective. Banning individual behavior won't prevent collective dysfunction and will only harm honest actors. The only answer that makes sense is reforming the acceptance of work to be resistant to the inevitable (ab)use of AI.

It's the exact same anti-cheating method used to great effect in top level e-sports. You can't trust competitor-supplied hardware, so the only option is for the institution to ban it and supply all hardware itself. Higher education is primarily about competition between job candidates. Eliminating cheating needs to be top priority or the whole system will collapse.

> Eliminating cheating needs to be top priority or the whole system will collapse.

It's going to collapse regardless because of the replication crisis. You might as well tackle the hard problem and figure out how to integrate replication into acceptance, or the consensus publication is intended to represent is meaningless. This is true regardless of whether a human or a robot is performing the work.

Re: The threat is comfortable drift toward not understanding what you're doing

#635

Post by rossant » Mon, Apr 06, 2026, 7:34 PM UTC

zozbot234 wrote: post #266

AlexWilkins12 wrote: post #72

Ironically, this article reeks of AI-generated phrases. Lot's of "It's not X, it's Y". eg: - "The failure mode isn't malice. It's convenience", - "You haven't saved time. You've forfeited the experience that the time was supposed to give you." - "But the real threat isn't either of those things. It's quieter, and more boring, and therefore more dangerous. The real threat is a slow, comfortable drift toward not unders…

Yes, the overwrought "It's not X it's Y" are signals of LLM involvement. No human uses them like that all the frickin' time . AI loves this construct way too much and cannot really tell whether the contrast is relevant/actually makes sense.

Seeing these constructs in a text is not just a vague hint that it was AI-generated. It's a smoking gun.

Re: The threat is comfortable drift toward not understanding what you're doing

#636

Post by augment_me » Mon, Apr 06, 2026, 8:22 PM UTC

MarkusQ wrote: post #491

doug_durham wrote: post #462

Earlier quoted context omitted.

The article is a thought experiment. The author hypothesizes that Bob isn't getting the same benefit that Alice is getting. That hypothesis could be wrong. I don't know and the author doesn't know. It could be that Bob is going to have a very successful career and will deeply know the field because he is able to traverse a wider set of problems more quickly. At this point, it's just hypothesis. I don't think that we…

We do know. There have always been ways that people could avoid the painful process of learning, and...they don't learn. Here's a competing thought experiment: Jorge's Gym has a top notch body building program, which includes an extensive series of exercises that would-be body builders need to do over multiple years to complete the program. You enroll, and cleverly use a block and tackle system to complete all the ex…

Playing devil's advocate here, but in theory, you could claim that setting up harnesses, targets, verification and incentives for different tasks might be the learning that you are doing. I think that there can be a fair argument made that we are just moving the abstraction a layer up. The learning is then not in the specifics of the field knowledge, but knowing the hacks, monkey patches, incentives and goals that the models should perform.

Re: The threat is comfortable drift toward not understanding what you're doing

#637

Post by bit-anarchist » Mon, Apr 06, 2026, 8:44 PM UTC

bluefirebrand wrote: post #445

Wowfunhappy wrote: post #386

Earlier quoted context omitted.

> Or even that [AI] can do 100% of coding right now. I don't actually think the article refutes this. But the AI needs to be in the hands of someone who can review the code (or astrophysics paper), notice and understand issues, and tell the AI what changes to make. Rinse, repeat. It's still probably faster than writing all the code yourself (but that doesn't mean you can fire all your engineers). The question is, how…

> The question is, how do you become the person who can effectively review AI code without actually writing code without an AI? I'd argue you basically can't. I agree, and I'd go a step further: You can be the absolute best coder in the world, the fastest and most accurate code reviewer ever to live, and AI still produces bad code so much faster than you can review that it will become a liability eventually no matter…

It isn't as binary as that. We already don't read all the code. When was the last time you checked the disassembly/IR of the programs you wrote? Very few will say recently. The others, despite not reading the entirety of the code, can still produce something with good quality.

Besides, it's not like QA ends at code analysis. In fact, that's pretty shallow for the field.

Re: The threat is comfortable drift toward not understanding what you're doing

#638

Post by deanCommie » Mon, Apr 06, 2026, 9:15 PM UTC

caxap wrote: post #292

If this article was written a year ago, I would have agreed. But knowing what I know today, I highly doubt that the outcomes of LLM/non-LLM users will be anywhere close to similar. LLMs are exceptionally good at building prototypes. If the professor needs a month, Bob will be done with the basic prototype of that paper by lunch on the same day, and try out dozens of hypotheses by the end of the day. He will not be ch…

This is the first fundamental flaw of the article

> Bob's weekly updates to his supervisor were indistinguishable from Alice's. The questions were similar. The progress was similar. The trajectory, from the outside, was identical.

No they won't be. They might be worse. They might be better. But they'll be very different.

And, like you said...

> Alice and Bob had the same year. One paper each.

No they won't. Alice would've taken a year. Bob would've taken a few days.

You've already covered why that might actually be OK, so I'll talk about the author's other error:

> This sounds idealistic until you think about what astrophysics actually is. Nobody's life depends on the precise value of the Hubble constant. No policy changes if the age of the Universe turns out to be 13.77 billion years instead of 13.79. Unlike medicine, where a cure for Alzheimer's would be invaluable regardless of whether a human or an AI discovered it, astrophysics has no clinical output. The results, in a strict practical sense, don't matter. What matters is the process of getting them: the development and application of methods, the training of minds, the creation of people who know how to think about hard problems. If you hand that process to a machine, you haven't accelerated science. You've removed the only part of it that anyone actually needed.

Keep asking why. why does the development of the application and methods, the training of minds matter?

the goal isn't abstract. The goal is still ultimately for the benefit of humanity, just like the cure for Alzheimer's.

Humanity learned physics, so we made rockets and now we have satellites, and the entire planet is connected with communication and information.

Humanity must continue to invest in astrophysics so that we do not get wiped out by a single rogue asteroid barreling through the cosmos, like the dinosaurs did.

Now i'm not saying that there isn't other benefit to making generically intelligent humans that know how to think. But at the end of the day, the purpose of astrophysics is no less existential than the purpose for developing medicine.

I want to know the age of the universe so that we can understand what created it, and if we can reverse entropy, and if there is anything beyond the universe. That is a quest for humanity that will take hundreds if not millions of years.

Re: The threat is comfortable drift toward not understanding what you're doing

#639

Post by impjohn » Tue, Apr 07, 2026, 12:43 PM UTC

sd9 wrote: post #91

jurgenaut23 wrote: post #81

Earlier quoted context omitted.

I understand your point, but this is a purely utilitarian view and it doesn’t account for the fact that, even if agents may do everything, it doesn’t mean they should, both in a normative and positive sense. There is a vast range of scenarios in which being more or less independent from agents to perform cognitive tasks will be both desirable and necessary, at the individual, societal and economic level. The question…

The people who pay my bills operate in a largely utilitarian fashion. They’re not going to pay me to manually program because I find it more enjoyable, when they can get Bob to do twice as much for less. This is why I say I don’t like it, but it is what it is.

I think this will become the prevailing sentiment when the dust settles.

Re: The threat is comfortable drift toward not understanding what you're doing

#640

Post by impjohn » Tue, Apr 07, 2026, 12:48 PM UTC

pigeons wrote: post #343

sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

> So if Bob can do things with agents, he can do things. But he does things wrong.

I believe the economic machine gives an edge to people who do more right things than wrong. Bob does things wrong, but given a 10x amount of output, the balance of right output vs wrong output may still be favored upon by the economy. A speculation, to be sure, we'll have to see how it pans out.