Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

321–330 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#321

Post by cmiles74 »

ChrisMarshallNY wrote: post #315

This is not wrong, but the "Bob and Alice" conundrum is not simple, either. In academia, understanding is vital. The same for research. But in production , results are what matters. Alice would be a better researcher, but Bob would be a better producer. He knows how to wrangle the tools. Each has its value. Many researchers develop marvelous ideas, but struggle to commercialize them, while production-oriented enginee…

I have to disagree that Bob will be a better producer, although I do agree that Bob will produce more. In this scenario, Bob isn't clear on which LLM output is valid and important and which is erroneous and misleading; I think that's a pretty critical distinction. It's the kind of thing that might go undetected for a long time, until a particular paper turns out to be important and it's discovered that it's also entirely wrong, wasting a lot of time and energy.

Re: The threat is comfortable drift toward not understanding what you're doing

#322

Post by mkovach »

jappgar wrote: post #261
mkovach wrote: post #240

This isn't new. It's been the same problem for decades, not what gets built, but what gets accepted. Weak ownership, unclear direction, and "sure, I guess" reviews were survivable when output was slow. When changes came in one at a time, you could get away with not really deciding. AI doesn't introduce a new failure mode. It puts pressure on the old one. The trickle becomes a firehose, and suddenly every gap is visib…

This is exactly the problem I see today. And it's not just a volume problem. Mediocre devs previously couldn't complete a project by themselves and were forced to solicit help and receive feedback along the way. When all managers care about is "shipping", development becomes a race to the bottom. Devs who used to collaborate are now competing. Whoever gets the slop into the codebase fastest, wins.

This is also very true, and while I consider it part of the authority to say no, this is a significant point.

Re: The threat is comfortable drift toward not understanding what you're doing

#323

Post by cmiles74 »

wongarsu wrote: post #169
jacquesm wrote: post #21

Earlier quoted context omitted.

Precisely. The first 10 rungs of the ladder will be removed, but we still expect you to be able to get to the roof. The AI won't get you there and you won't have the knowledge you'd normally gain on those first 10 rungs to help you move past #10.

There are a lot of people in academia who are great at thinking about complex algorithms but can't write maintainable code if their life depended on it. There are ways to acquire those skills that don't go the junior developer route. Same with debugging and profiling skills But we might see a lot more specialization as a result

Do they need to write maintainable code? I think probably not, it's the research and discovering the new method that is important.

Re: The threat is comfortable drift toward not understanding what you're doing

#324

Post by NiloCK »

nothrabannosir wrote: post #314
NiloCK wrote: post #162

Earlier quoted context omitted.

People would have said the same about graphing calculators or calculators before that. Socrates said the same thing about the written word. The determining factor is always "did I come up with this tool". Somehow, subsequent generations always manage to find their own competencies (which, to be fair, may be different). This isn't guaranteed to play out, but it should be the default expectation until we actually see g…

What do people mean exactly when they bring up “Socrates saying things about writing”? Phaedrus? > “Most ingenious Theuth, one man has the ability to beget arts, but the ability to judge of their usefulness or harmfulness to their users belongs to another; [275a] and now you, who are the father of letters, have been led by your affection to ascribe to them a power the opposite of that which they really possess. > "Fo…

But did this grind humanity to a halt?

Yes - specific faculties atrophied - I wouldn't dispute it. But the (most) relevant faculties for human flourishing change as a function of our tools and institutions.

Re: The threat is comfortable drift toward not understanding what you're doing

#325

Post by steveBK123 »

gedy wrote: post #298
ipaddr wrote: post #263

Earlier quoted context omitted.

Marketing is the moat llms haven't been able to overcome. Being able to create a Word clone is easier but the difficulty of selling it is as hard or harder than ever. Show me an llm that can sell my product and find market fit. In reality llms are taking away profitable tools and keeping the revenue themselves.

There's definitely folks working on automatically marketing via LLMs, but I have my doubts that it wont just numb people further to marketing as we are close to saturation.

My thought/worry on a lot of the LLM agentic workflow personal assistant stuff is it is just ripe for fraud. The money is more on the adversarial side.

People think they'll just have a personal bot out there buying airline tickets, hotel rooms, jeans, new phone, etc. Meanwhile as soon as you have agents like this out in the wild, the capital will flow to bad actors creating bots to game those bots.

The world is PvP unfortunately. There is more money to be made skimming agents trying to buy stuff than there is in getting people to pay for a personal assistant agent subscription.

It's like why a lot of ad-based stuff doesn't offer a premium option for people to pay to opt out (ex Youtube). The people who can afford to pay and avoid search/social media/etc advertising are exactly the people you can make a lot of money advertising to.

Re: The threat is comfortable drift toward not understanding what you're doing

#326

Post by coffeefirst »

ozim wrote: post #307
sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

There is still a lot of engineering to be done with LLMs. Maybe not exactly writing code but I think a lot of optimization problems will be there no matter what. Some people treat toilet as magic hole where they throw stuff in flush and think it is fine. If you throw garbage in you will at some point have problems. We are in stage where people think it is fine to drop everything into LLM but then they will see the bi…

Yep. I hate to predict the future but I’m betting on small, open models, used as tools here and there. Which is great, you can get 90% of the speed up with 5-10% of the cost once you account for the full costs in both time and money.

The economics and security model on full agents running in loops all day may come home to roost faster than expertise rot.

Re: The threat is comfortable drift toward not understanding what you're doing

#327

Post by leereeves »

Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

> And so the paradox is, the LLMs are only useful† if you're Schwartz

Was the LLM even useful for Schwartz, if it produced false output?

Re: The threat is comfortable drift toward not understanding what you're doing

#328

Post by Yokohiii »

caxap wrote: post #292

If this article was written a year ago, I would have agreed. But knowing what I know today, I highly doubt that the outcomes of LLM/non-LLM users will be anywhere close to similar. LLMs are exceptionally good at building prototypes. If the professor needs a month, Bob will be done with the basic prototype of that paper by lunch on the same day, and try out dozens of hypotheses by the end of the day. He will not be ch…

Bob will never figure out there is an error in his paper. If someone tells him, the LLM will have trouble to figure it out as well, remember the LLM inserted the error to make it "look right".

Your perspective is cut off. In the real world Bob is supposed to produce outcomes that work. If he moves on into the industry and keeps producing hallucinated, skewed, manipulated nonsense, then he will fall flat instantly. If he manages to survive unnoticed, he will become CEO. The latter rather unlikely.

Re: The threat is comfortable drift toward not understanding what you're doing

#329

Post by Wowfunhappy »

conception wrote: post #310
Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

Sadly I don’t see how our current social paradigm works for this. There is no history of any sort of long planning like this or long term loyalty (either direction) with employees and employers for this sort of journeyman guild style training. AI execs are basically racing, hoping we won’t need a Schwartz before they are all gone. But what incentives are in place to high a college grad, have them work without llms fo…

Then the social paradigm needs to change. Is everyone just going to roll over and die while AI destroys academia (and possibly a lot more)?

Last September, Tyler Austin Harper published a piece for The Atlantic on how he thinks colleges should respond to AI. What he proposes is radical—but, if you've concluded that AI really is going to destroy everything these institutions stand for, I think you have to at least consider these sorts of measures. https://www.theatlantic.com/culture/archive/2025/09/ai-colle...

Post reply on HN