Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

541–550 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#541

Post by greedo »

SoftTalker wrote: post #396
mattmanser wrote: post #62

Earlier quoted context omitted.

The authors point went a little over your head. It doesn't matter if Bob can be normal. There was no point to him being paid to be on the program. From the article: If you hand that process to a machine, you haven't accelerated science. You've removed the only part of it that anyone actually needed.

And then you realize that most of science is unnecessary. As TFA points out, it doesn't matter if the age of the universe is 13.77 or 13.79 billion years. So you ban AI in science, you produce more scientists who can solve problems that don't matter. So what?

Funny thing about that. Knowing what science is important is the hard part, and you often don't know for years or even decades.

Re: The threat is comfortable drift toward not understanding what you're doing

#542

Post by cmiles74 »

ChrisMarshallNY wrote: post #389
cmiles74 wrote: post #321

Earlier quoted context omitted.

I have to disagree that Bob will be a better producer, although I do agree that Bob will produce more . In this scenario, Bob isn't clear on which LLM output is valid and important and which is erroneous and misleading; I think that's a pretty critical distinction. It's the kind of thing that might go undetected for a long time, until a particular paper turns out to be important and it's discovered that it's also ent…

Sounds like you're still thinking of Bob as a researcher. In production, there would be no "paper"; just some software/hardware product. If there was a problem, that would be fairly obvious, with testing (we are going to be testing our products, right?). I have been wrestling all morning, with an LLM. It keeps suggesting stuff that doesn't work, and I need to keep resetting the context. I am often able to go in, and…

Indeed! In the article Bob is an astrophysicist.

I think the difference between the workflow you describe and the description of Bob's is that you have a pretty good idea of what a working solution would look like. In my reading if the article Bob does not.

In my opinion, the software developer analogue of Bob would be someone who would often reach for the LLM as it's nearby and easy. Maybe at first they would be careful about reading and vetting the model's output. Over time they might grow comfortable and overly confident with the model's output and pay less and less attention. As they take on more complex tasks they begin to understand less and less about the LLM tooling output but they don't really notice, it all looks good and tests are passing. Eventually we see a production problem, maybe even an outage. When we narrow down the issue to a PR with Bob's name on it and ask him how it led to the production issue, Bob tries to be helpful but struggles to understand his own PR.

Re: The threat is comfortable drift toward not understanding what you're doing

#543

Post by Izkata »

conception wrote: post #349
FrojoS wrote: post #336

Earlier quoted context omitted.

> There is no history of any sort of long planning Sure there is. Its the formal education system that produced the college grad.

… between employees and employers. The proposal that everyone pay for college until they are in their 40s doesn’t seem viable.

With loans, that's kind of how it works now...

Re: The threat is comfortable drift toward not understanding what you're doing

#544

Post by Kon5ole »

defrost wrote: post #523
Kon5ole wrote: post #143

Earlier quoted context omitted.

>This is false. There absolutely are people that fall back on older tools when fancy tools fail. >They other week I finished putting holes in fence posts with a bit and brace as there was no fuel for the generator to run corded electric drills and the rechargable batteries were dead. It depends on the task though. If you are in a similar scenario as with your fence posts and want to edit computer programs, you can't.…

> If you are in a similar scenario as with your fence posts and want to edit computer programs, you can't. Solar panels charging old thinkpads suddenly doesn't work, or are we reliant upon software that requires cloud services to function in your scenario? > How many people in the world can recreate a ASML lithography machine .. Some .. are you advocating the answer should be none, in your future?

>Solar panels charging old thinkpads suddenly doesn't work, or are we reliant upon software that requires cloud services to function in your scenario?

It's your scenario, you said you had no batteries and had to drill holes by hand, I'm saying that's fine for drilling holes but if you have no batteries (or solar panels, those could charge your power tools as well so they're not in the scenario!) you can't alter computer code by hand.

Leaving computers aside, we can take a piano player - can play a piano but he might not know how to build one. A doctor can interpret X-rays but he can't build an X-ray machine.

And so on. We are living in the reality where many, many people are using tools they don't understand how to make already, and have done so for 100+ years.

>Some .. are you advocating the answer should be none, in your future?

I'm not advocating anything, I'm just saying that this is the reality already, it's not new with AI.

Ideally we'd forever keep a chain of key people who know how to make everything from atoms, but realistically that's a chain that will break. It's happened before, medieval dark ages for example.

Re: The threat is comfortable drift toward not understanding what you're doing

#545

Post by somesortofthing »

I used to feel this way but... honestly, I've found that pressing on with only a vague understanding of what's happening and then diving deep with the agent's own help if it keeps making bad decisions leads to more output of comparable quality. Even without a deep understanding of the topic, you can usually tell when the LLM is BSing and you need to intervene. The model has much more knowledge "present-at-hand" than it'll actually apply to a given implementation, so you can substantially deepen your understanding with minimal reference to external resources by just taking a break from implementation to have a convo with it.

I'm sure this approach breaks down at the very frontiers of highly technical fields but... virtually all work, even work by educated professionals, happens outside that area anyway. On well-trodden ground, you can improve at supervising agents by doing things that test your ability to supervise agents.

Re: The threat is comfortable drift toward not understanding what you're doing

#546

Post by matheusmoreira »

krackers wrote: post #512
matheusmoreira wrote: post #371

I dunno. Claude helped me implement a new memory allocator, compacting garbage collector and object heap for my programming language. I certainly understood what I was doing when I did this. The experience was extremely engaging for me. Claude taught me a lot. I think the real danger is no longer caring about what you're doing. Yesterday I just pointed Claude at my static site generator and told it to clean it up. I…

This seems contradictory at first glance, if you didn't actually implement it then how well have you actually understood it? It's known from learning theory that engagement or even self-reported understanding doesn't imply that the student can actually solve problems presented to him. If someone claims to have "understood [middle school] algebra" but they aren't able to solve equations by themselves, you'd be skeptic…

I did implement it though. I wrote the code myself. Claude explained to me how all of those things worked, showed me code and examples to illustrate, walked me through the algorithms step by step. It turned out to not be as insurmountably complex as I thought it was. I'm also manually writing articles about all of those things to crystallize everything I learned.

Claude has been amazing for code review. Having someone to talk to about my own code is world changing for a solo developer like me.

Re: The threat is comfortable drift toward not understanding what you're doing

#547

Post by suzzer99 »

zozbot234 wrote: post #266
AlexWilkins12 wrote: post #72

Ironically, this article reeks of AI-generated phrases. Lot's of "It's not X, it's Y". eg: - "The failure mode isn't malice. It's convenience", - "You haven't saved time. You've forfeited the experience that the time was supposed to give you." - "But the real threat isn't either of those things. It's quieter, and more boring, and therefore more dangerous. The real threat is a slow, comfortable drift toward not unders…

Yes, the overwrought "It's not X it's Y" are signals of LLM involvement. No human uses them like that all the frickin' time . AI loves this construct way too much and cannot really tell whether the contrast is relevant/actually makes sense.

It almost never does.

Re: The threat is comfortable drift toward not understanding what you're doing

#548

Post by ChrisMarshallNY »

cmiles74 wrote: post #542
ChrisMarshallNY wrote: post #389

Earlier quoted context omitted.

Sounds like you're still thinking of Bob as a researcher. In production, there would be no "paper"; just some software/hardware product. If there was a problem, that would be fairly obvious, with testing (we are going to be testing our products, right?). I have been wrestling all morning, with an LLM. It keeps suggesting stuff that doesn't work, and I need to keep resetting the context. I am often able to go in, and…

Indeed! In the article Bob is an astrophysicist. I think the difference between the workflow you describe and the description of Bob's is that you have a pretty good idea of what a working solution would look like. In my reading if the article Bob does not. In my opinion, the software developer analogue of Bob would be someone who would often reach for the LLM as it's nearby and easy. Maybe at first they would be car…

Yeah, I'm very much a "production" person, so that's my lens.

For me, I started off with Machine Code, but these days, I have no idea what's going on, below the compiler. I know enough to be dangerous, looking at a stack dump, but that's about it. My days of hex keypads are long gone.

I still get stuff done, that I could only dream of, back then. LLMs are really just another step on the evolutionary ladder.

I'm old enough to remember when calculators were banned in the classroom, for just about exactly the same reasons that people are complaining about LLMs.

Anyway, the die is cast. We'll have to see how things turn out.

Re: The threat is comfortable drift toward not understanding what you're doing

#549

Post by YeGoblynQueenne »

>> The supervisor still needs to know what the answer should look like, still needs to know which checks to demand, still needs to have the instinct that something is off before they can articulate why. That instinct doesn't come from a subscription. It comes from years of failing at exactly the kind of work that people keep calling grunt work.

i.e. science.

Re: The threat is comfortable drift toward not understanding what you're doing

#550

Post by throawayonthe »

ipaddr wrote: post #225
sgarland wrote: post #108

Earlier quoted context omitted.

I would love to see someone attempt to do multiplication who never learned addition, or exponentiation without having learned multiplication. There is a vast difference between “never learned the skill,” and “forgot the skill from lack of use.” I learned how to do long division in school, decades ago. I sat down and tried it last year, and found myself struggling, because I hadn’t needed to do it in such a long time.

Most people learn multiplication by memorizing a series of cards 2x2,2x3.. 9x9. Later this gets broken down to addition in higher grades.

i have never heard of this, multiplication was definitely introduced to both me/my peers and my siblings as "doing addition n times"
Post reply on HN