Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

491–500 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#491

Post by MarkusQ »

doug_durham wrote: post #462
Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

The article is a thought experiment. The author hypothesizes that Bob isn't getting the same benefit that Alice is getting. That hypothesis could be wrong. I don't know and the author doesn't know. It could be that Bob is going to have a very successful career and will deeply know the field because he is able to traverse a wider set of problems more quickly. At this point, it's just hypothesis. I don't think that we…

We do know. There have always been ways that people could avoid the painful process of learning, and...they don't learn.

Here's a competing thought experiment:

Jorge's Gym has a top notch body building program, which includes an extensive series of exercises that would-be body builders need to do over multiple years to complete the program. You enroll, and cleverly use a block and tackle system to complete all the exercises in weeks instead of years.

Did you get the intended results?

Re: The threat is comfortable drift toward not understanding what you're doing

#492

Post by mattmanser »

carlosjobim wrote: post #199
mattmanser wrote: post #99

Earlier quoted context omitted.

There's a long, detailed, often repeated answer to your open question in the article. Namely, if you can't do it without the AI, you can't tell when it's given you plausible sounding bullshit. So Bob just wasted everyone's time and money.

You can verify by running the code and see if it works.

Seriously? The article is about scientists learning to do science, not programming.

I know we're not supposed to say RTFA, but your comment really takes the cake.

Re: The threat is comfortable drift toward not understanding what you're doing

#493

Post by Denvercoder9 »

throw310822 wrote: post #465
Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

> the paradox is, the LLMs are only useful† if you're Schwartz, and you can't become Schwartz by using LLMs. That you can't "become Schwartz" by using LLMs is an unproven assumption. Actually, it's a contradiction in the logic of the essay: if Bob managed to produce a valid output by using an LLM at all , then it means that he must have acquired precisely that supervision ability that the essay claims to be necessary…

There's no contradiction, the point is that Bob is able to produce valid output using LLMs, but only while he himself is being supervised; and that he doesn't develop the skills to supervise independently himself in the future.

Re: The threat is comfortable drift toward not understanding what you're doing

#494

Post by MarkusQ »

didgetmaster wrote: post #253
NiloCK wrote: post #162

Earlier quoted context omitted.

People would have said the same about graphing calculators or calculators before that. Socrates said the same thing about the written word. The determining factor is always "did I come up with this tool". Somehow, subsequent generations always manage to find their own competencies (which, to be fair, may be different). This isn't guaranteed to play out, but it should be the default expectation until we actually see g…

If you hand a broken calculator to someone who knows how to do math, and they entered 123 + 765 which produced an answer of 6789; they should instantly know something is wrong. Hand that calculator to someone who never understood what the tool actually did but just accepted whatever answer appeared; and they would likely think the answer was totally reasonable. Catching an LLM hallucinating often takes a basic unders…

> Catching an LLM hallucinating often takes a basic understanding of what the answer should look like before asking the question.

We had the same problem in the early days of calculators. Using a slide rule, you had to track the order of magnitude in your head; this habit let you spot a large class of errors (things that weren't even close to correct).

When calculators came on the scene, people who never used a slide rule would confidently accept answers that were wildly incorrect (example: a mole of ideal gas at STP is 22.4 liters. If you typo it as 2204, you get an answer that's off by roughly two orders of magnitude, say 0.0454 when it should be 4.46. Easy to spot if you know roughly what the answer should look like, but easy to miss if you don't).

Re: The threat is comfortable drift toward not understanding what you're doing

#495

Post by turtletontine »

> Bob's weekly updates to his supervisor were indistinguishable from Alice's. The questions were similar. The progress was similar. The trajectory, from the outside, was identical.

I don’t believe this. Totally plausible that someone would be able to produce passable work with LLMs at a similar pace to a curious and talented scientist. But if you, their advisor, are sitting down and talking with them every week? It’s obvious how much they care or understand, I can’t believe you wouldn’t be able to tell the difference between these students.

Re: The threat is comfortable drift toward not understanding what you're doing

#496

Post by julenx »

Yokohiii wrote: post #319
DavidPiper wrote: post #89

I've just started a new role as a senior SWE after 5 months off. I've been using Claude a bit in my time off; it works really well. But now that I've started using it professionally, I keep running into a specific problem: I have nothing to hold onto in my own mind. How this plays out: I use Claude to write some moderately complex code and raise a PR. Someone asks me to change something. I look at the review and thin…

This is my primary personal concern. I think it could be an silent psychological landmine going off way too late (sic). In a living codebase you spent long stretches to learn how it works. It's like reading a book that doesn't match your taste, but you eventually need to understand and edit it, so you push through. That process is extremely valuable, you will get familiar with the codebase, you map it out in your hea…

I believe the detachment gets exacerbated by the fact that others are simultaneously modifying the codebase at speeds that doesn't allow you to keep up. Depending on how the codebase boundaries and ownership are defined, this directly impacts your ability to reason about the whole system and therefore influence direction.

Re: The threat is comfortable drift toward not understanding what you're doing

#497

Post by throw310822 »

Denvercoder9 wrote: post #493
throw310822 wrote: post #465

Earlier quoted context omitted.

> the paradox is, the LLMs are only useful† if you're Schwartz, and you can't become Schwartz by using LLMs. That you can't "become Schwartz" by using LLMs is an unproven assumption. Actually, it's a contradiction in the logic of the essay: if Bob managed to produce a valid output by using an LLM at all , then it means that he must have acquired precisely that supervision ability that the essay claims to be necessary…

There's no contradiction, the point is that Bob is able to produce valid output using LLMs, but only while he himself is being supervised; and that he doesn't develop the skills to supervise independently himself in the future.

> only while he himself is being supervised

No, this is impossible unless Bob is presenting at each weekly meeting simply the output of the LLM and feeding the tutor's feedback straight into it. For a total of 10 minutes work per week, and the tutor would notice straight away at least for the lack of progress.

No, the article specifies that Bob actually works with the LLM, doesn't just delegate. He asks the agent to summarise, to explain, and to help with bug fixing. You could easily argue that Bob, having such an AI tutor available 24/7, can develop understanding much faster. He certainly won't waste his time with small details of python syntax (though working with a "coding expert" will make his code much cleaner and more advanced).

Re: The threat is comfortable drift toward not understanding what you're doing

#498

Post by krackers »

b00ty4breakfast wrote: post #85
gbro3n wrote: post #71

Earlier quoted context omitted.

I think a good analogy is people not being able to work on modern cars because they are too complex or require specialised tools. True I can still go places with my car, but when it goes wrong I'm less likely to be able to resolve the problem without (paid for) specialised help.

And just like modern vehicles rob the user of autonomy, so too for coding agents. Modern tech moves further and further away from empowering normal people and increasingly serves to grow the influence of corporations and governments over our day to day lives. It's not inherent, but it is reality unless folks stop giving up agency for convenience. I'm not holding my breath.

Cars are actually a good metaphor, it works on so many levels. Modern cars have "democratized" access to long-distance travel in a sense, and most people don't need to do any heavy maintenance themselves. But the flipside is that places that have adopted it have become "car dependent" and build cities assuming access to cars.

Are we net better off than if we didn't have cars and simply built public transport with walkable cities?

Re: The threat is comfortable drift toward not understanding what you're doing

#499

Post by senordevnyc »

pxc wrote: post #453
Wowfunhappy wrote: post #329

Earlier quoted context omitted.

Then the social paradigm needs to change. Is everyone just going to roll over and die while AI destroys academia (and possibly a lot more)? Last September, Tyler Austin Harper published a piece for The Atlantic on how he thinks colleges should respond to AI. What he proposes is radical —but, if you've concluded that AI really is going to destroy everything these institutions stand for, I think you have to at least co…

I was pretty interested until I got to this part: > Another reason that a no-exceptions policy is important: If students with disabilities are permitted to use laptops and AI, a significant percentage of other students will most likely find a way to get the same allowances, rendering the ban useless. I witnessed this time and again when I was a professor—students without disabilities finding ways to use disability ac…

Yeah, this proposal is likely straight up illegal.

Re: The threat is comfortable drift toward not understanding what you're doing

#500

Post by senordevnyc »

mrob wrote: post #399
Wowfunhappy wrote: post #329

Earlier quoted context omitted.

Then the social paradigm needs to change. Is everyone just going to roll over and die while AI destroys academia (and possibly a lot more)? Last September, Tyler Austin Harper published a piece for The Atlantic on how he thinks colleges should respond to AI. What he proposes is radical —but, if you've concluded that AI really is going to destroy everything these institutions stand for, I think you have to at least co…

>What he proposes is radical It sounds entirely reasonable and moderate to me.

It's neither reasonable nor moderate, which is why it'll never happen.
Post reply on HN