Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

71–80 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#71

Post by gbro3n »

sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

I think a good analogy is people not being able to work on modern cars because they are too complex or require specialised tools. True I can still go places with my car, but when it goes wrong I'm less likely to be able to resolve the problem without (paid for) specialised help.

Re: The threat is comfortable drift toward not understanding what you're doing

#72

Post by AlexWilkins12 »

Ironically, this article reeks of AI-generated phrases. Lot's of "It's not X, it's Y". eg: - "The failure mode isn't malice. It's convenience", - "You haven't saved time. You've forfeited the experience that the time was supposed to give you." - "But the real threat isn't either of those things. It's quieter, and more boring, and therefore more dangerous. The real threat is a slow, comfortable drift toward not understanding what you're doing. Not a dramatic collapse. Not Skynet. Just a generation of researchers who can produce results but can't produce understanding."

And indeed running it through a few AI text detectors, like Pangram (not perfect, by any means, but a useful approximation), returns high probabilities.

It would have felt more honest if the author had included a disclaimer that it was at least part written with AI, especially given its length and subject matter.

Re: The threat is comfortable drift toward not understanding what you're doing

#73

Post by sd9 »

torben-friis wrote: post #69
sd9 wrote: post #65

Earlier quoted context omitted.

I’m very good at the handcrafted stuff, I’ve been doing this a while. I don’t feel like giving up my self sufficiency, I just feel like the writing is on the wall.

By "you" I actually meant this hypothetical person who's only good enough for AI assisted. Though even for us who are already experienced, we should keep the manual stuff even if it's just as going to the gym. I don't see myself retaining my skills for long by just reviewing LLM output.

Yes sorry, I didn’t think you were addressing me directly, just adding my own thoughts.

I agree totally with the sentiment, and I am concerned about my own skills atrophying.

Re: The threat is comfortable drift toward not understanding what you're doing

#74

Post by robot-wrangler »

Another threat is that you can find tons of papers pointing out how neural AI still struggles handling simple logical negation. Who cares right, we use tools for symbolics, yada yada. Except what's really the plan? Are we going to attempt parallel formalized representations of every piece of input context just to flag the difference between please DONT delete my files and please DO? This is all super boring though and nothing bad happened lately, so back to perusing latest AGI benchmarks..

Re: The threat is comfortable drift toward not understanding what you're doing

#75

Post by oncallthrow »

kgwxd wrote: post #52
oncallthrow wrote: post #16

I think this article is largely, or at least directionally, correct. I'd draw a comparison to high-level languages and language frameworks. Yes, 99% of the time, if I'm building a web frontend, I can live in React world and not think about anything that is going on under the hood. But, there is 1% of the time where something goes wrong, and I need to understand what is happening underneath the abstraction. Similarly,…

> LLMs will get _so_ good that I don't feel the need to do this, in the same way that I don't think about the transistors my code is ultimately running on. The problem is, they're nothing like transistors, and never will be. Those are simple. Work or don't, consistently, in an obvious, or easily testable, way. LLM are more akin to biological things. Complex. Not well understood. Unpredictable behavior. To be safely u…

I suppose transistors is a bad example.

Perhaps a better analogy would be the Linux kernel. It's built by biological humans, and fallible ones at that. And yet, I don't feel the need to learn the intricacies of kernel internals, because it's reliable enough that it's essentially never the kernel's fault when my code doesn't work.

Re: The threat is comfortable drift toward not understanding what you're doing

#76

Post by hirako2000 »

stavros wrote: post #27
hirako2000 wrote: post #15

Earlier quoted context omitted.

There is an argument to make that tools that speed up a process whilst keeping acuity intact are legitimate. LLMs, the way they typically get used, are solely to save time by handing over nearly the entire process. In that sense acuity can't remain intact, even less so improving over time.

So?

You previous comment reads as if LLMs get some unjustified different treatment.

Do you agree the different treatment is justified ? (Many do not). Or are you asking , so what if acuity is diminished so long as an LLM does the job equally well?

Re: The threat is comfortable drift toward not understanding what you're doing

#77

Post by throwanem »

sd9 wrote: post #43
nidnogg wrote: post #36

Earlier quoted context omitted.

I don't like it either. But what is really guaranteeing other markets from flunking similarly later on? What's to say other jobs are going to be any better? Back in college, most of my peers would say "I'm not cut out for anything else. This is it". They were, sure enough, computer and/or math people at heart from an early age. More importantly, what's gonna be the next stable category of remote-first jobs that a per…

I don’t know, it’s only since about December that I felt things really start to shift, and February when my job started to become very different. Personally I’m looking at more physical domains, but it’s early days in my exploration. I think if I wanted to stick to remote work (which I have enjoyed since 2020), then the AI story would just keep playing out. I’m also totally open to taking a big pay cut to do somethin…

So recent? I've been on sabbatical (the real kind, self-funded) for eighteen months, and while my sense has been things have not stopped heading downhill since I stepped off the ride back in 2024, to hear of such a sudden step change is somewhat novel. "Very different" just how, if you don't mind my asking?

(I'm also looking for local, personally satisfying work, in exchange for a pay cut. Early days, and I am finding the profession no longer commands quite the social cachet it once did, but I'm not foolish enough to fail to price for the buyer's market in which we now seek to sell our labor. Besides, everyone benefits from the occasional reminder to humility! "Memento mori" and all that.)

Re: The threat is comfortable drift toward not understanding what you're doing

#78

Post by raldi »

djaro wrote: post #14
sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

> So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade…

To me it feels more like learning to cook versus learning how to repair ovens and run a farm. Software engineering isn’t about writing code any more than it’s about writing machine code or designing CPUs. It’s about bringing great software into existence.

Re: The threat is comfortable drift toward not understanding what you're doing

#79

Post by mbbutler »

oncallthrow wrote: post #16

I think this article is largely, or at least directionally, correct. I'd draw a comparison to high-level languages and language frameworks. Yes, 99% of the time, if I'm building a web frontend, I can live in React world and not think about anything that is going on under the hood. But, there is 1% of the time where something goes wrong, and I need to understand what is happening underneath the abstraction. Similarly,…

In my personal experience, the rate at which Claude Code produces suboptimal Rust is way higher than 1%.

Re: The threat is comfortable drift toward not understanding what you're doing

#80

Post by layer8 »

efields wrote: post #26

I literally don't know how compilers work. I've written code for apps that are still in production 10 years later.

You don’t need to understand compilers because the code it compiles, when valid according to the language specification, is supposed to work as written, and virtually always does. There is no language specification and no “as written” with LLMs.
Post reply on HN