Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

311–320 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#311

Post by Jensson »

lxgr wrote: post #282
qsera wrote: post #133

Earlier quoted context omitted.

>we would still have access to capable agents. But they would be outdated, right? Would an agent that can only code in COBOL would be as useful today?

> But they would be outdated, right? Outdated compared to what? In your counterfactual, VC funded agents don't exist anymore, no? Your argument, if I understand it correctly, is that they might somehow go away entirely when VC funding dries up, when more realistically they'll probably at most become twice as expensive or regress half a year in performance.

> Outdated compared to what? In your counterfactual, VC funded agents don't exist anymore, no?

Outdated compared to reality / humans, their knowledge cutoff is a year further behind every year they don't get updates. Humans continuously expands their knowledge, the models needs to keep up with that.

Re: The threat is comfortable drift toward not understanding what you're doing

#312

Post by xantronix »

sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

I'm glad you've posted this comment because I strongly feel more people need to see sentiment, and push back against what many above want to become the new norm. I see capitulation and compliance in advance, and it makes me sad. I also see two very valid, antipodal responses to this phenomenon: Exit from the industry, and malicious compliance through accelerationism.

To the reader and the casual passerby, I ask: Do you have to work at this pace, in this manner? I understand completely that mandates and pressure from above may instill a primal fear to comply, but would you be willing to summon enough courage to talk to maybe one other person you think would be sympathetic to these feelings? If you have ever cared about quality outcomes, if for no other reason than the sake of personal fulfillment, would it not be worth it to firmly but politely refuse purely metrics-focused mandates?

Re: The threat is comfortable drift toward not understanding what you're doing

#313

Post by nathan_compton »

FrojoS wrote: post #251
nathan_compton wrote: post #19

Earlier quoted context omitted.

People say this in a very large number of other contexts. Mathematica has been able to do many integrals for decades and yet we still make students learn all the tricks to integrate by hand. This pattern is very common.

Yes. But to be fair to your specific point, symbolic solving of integrals used to be a huge skill in the engineering education. Nowadays, it is not a focus anymore, because numerical solutions are either sufficiently accurate or, more importantly, the only feasible approach anyway.

There is much more to life than engineering.

Re: The threat is comfortable drift toward not understanding what you're doing

#314

Post by nothrabannosir »

NiloCK wrote: post #162
jacquesm wrote: post #21

Earlier quoted context omitted.

Precisely. The first 10 rungs of the ladder will be removed, but we still expect you to be able to get to the roof. The AI won't get you there and you won't have the knowledge you'd normally gain on those first 10 rungs to help you move past #10.

People would have said the same about graphing calculators or calculators before that. Socrates said the same thing about the written word. The determining factor is always "did I come up with this tool". Somehow, subsequent generations always manage to find their own competencies (which, to be fair, may be different). This isn't guaranteed to play out, but it should be the default expectation until we actually see g…

What do people mean exactly when they bring up “Socrates saying things about writing”? Phaedrus?

> “Most ingenious Theuth, one man has the ability to beget arts, but the ability to judge of their usefulness or harmfulness to their users belongs to another; [275a] and now you, who are the father of letters, have been led by your affection to ascribe to them a power the opposite of that which they really possess.

> "For this invention will produce forgetfulness in the minds of those who learn to use it, because they will not practice their memory. Their trust in writing, produced by external characters which are no part of themselves, will discourage the use of their own memory within them. You have invented an elixir not of memory, but of reminding; and you offer your pupils the appearance of wisdom, not true wisdom, for they will read many things without instruction and will therefore seem [275b] to know many things, when they are for the most part ignorant and hard to get along with, since they are not wise, but only appear wise."

Sounds to me like he was spot on.

Re: The threat is comfortable drift toward not understanding what you're doing

#315

Post by ChrisMarshallNY »

This is not wrong, but the "Bob and Alice" conundrum is not simple, either.

In academia, understanding is vital. The same for research.

But in production, results are what matters.

Alice would be a better researcher, but Bob would be a better producer. He knows how to wrangle the tools.

Each has its value. Many researchers develop marvelous ideas, but struggle to commercialize them, while production-oriented engineers, struggle to come up with the ideas.

You need both.

Re: The threat is comfortable drift toward not understanding what you're doing

#316

Post by imtringued »

greenchair wrote: post #232
xandrius wrote: post #113

Earlier quoted context omitted.

Then you're using it more towards vibe coding than AI-assisted coding: I use AI to write the stuff the way I want it to be written. I give it information about how to structure files, coding style and the logic flow. Then I spend time to read each file change and give feedback on things I'd do differently. Vastly saves me time and it's very close or even better than what I would have written. If the result is somethi…

AI assisted coding makes you dumber full stop. It's obvious as soon as you try it for the first time. Need a regex? No need to engage your brain. AI will do that for you. Is what it produced correct? Well who knows? I didn't actually think about it. As current gen seniors brains atrophy over the next few years the scarier thing is that juniors won't even be learning the fundamentals because it is too easy to let AI h…

I agree. In the beginning when I was starting, I let the AI do all of the work and merely verified that it does what I want, but then I started running into token limits. In the first two weeks I honestly was just looking forward for the limit to refresh. The low effort made it feel like I would be wasting my time writing code without the agent.

Starting with week three the overall structure of the code base is done, but the actual implementation is lacking. Whenever I run out of tokens I just started programming by hand again. As you keep doing this, the code base becomes ever more familiar to you until you're at a point where you tear down the AI scaffolding in the places where it is lacking and keep it where it makes no difference.

Re: The threat is comfortable drift toward not understanding what you're doing

#317

Post by cmiles74 »

Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

I think we already know what we need to do: encourage people to do the work themselves, discourage beginners from immediately asking an LLM for help and re-introducing some kind of oral exam. As the article mentions, banning LLMs is impractical and what we really need are people who can tell when the LLM is confidently wrong; not people who don't know how to work with an LLM.

I hope it will encourage people to think more about what they get out of the work, what doing the work does for them; I think that's a good thing.

Re: The threat is comfortable drift toward not understanding what you're doing

#318

Post by visarga »

> Whether that student walks out the door five years later as an independent thinker or a competent prompt engineer is, institutionally speaking, irrelevant.

I think this is a simplification, of course Bob relied on AI but they also used their own brain to think about the problem. Bob is not reducible to "a competent prompt engineer", if you think that just take any person who prompts unrelated to physics and ask them to do Bob's work.

In fact Bob might have a change to cover more mileage on the higher level of work while Alice does the same on the lower level. Which is better? It depends on how AI will evolve.

The article assumes the alternative to AI-assisted work is careful human work. I am not sure careful human work is all that good, or that it will scale well in the future. Better to rely on AI on top of careful human work.

My objection comes from remembering how senior devs review PRs ... "LGTM" .. it's pure vibes. If you are to seriously review a PR you have to run it, test it, check its edge cases, eval its performance - more work than making the PR itself. The entire history of software is littered with bugs that sailed through review because review is performative most of the time.

Anyone remember the verification crisis in science?

Re: The threat is comfortable drift toward not understanding what you're doing

#319

Post by Yokohiii »

DavidPiper wrote: post #89

I've just started a new role as a senior SWE after 5 months off. I've been using Claude a bit in my time off; it works really well. But now that I've started using it professionally, I keep running into a specific problem: I have nothing to hold onto in my own mind. How this plays out: I use Claude to write some moderately complex code and raise a PR. Someone asks me to change something. I look at the review and thin…

This is my primary personal concern. I think it could be an silent psychological landmine going off way too late (sic).

In a living codebase you spent long stretches to learn how it works. It's like reading a book that doesn't match your taste, but you eventually need to understand and edit it, so you push through. That process is extremely valuable, you will get familiar with the codebase, you map it out in your head, you imagine big red alerts on the problematic stuff. Over time you become more and more efficient editing and refactoring the code.

The short term state of AI is pretty much outlined by you. You get a high level bug or task, you rephrase it into proper technical instructions and let a coding agent fill in the code. Yell a few times. Fix problems by hand.

But you are already "detached" from the codebase, you have to learn it the hard way. Each time your agent is too stupid. You are less efficient, at least in this phase. But your overall understanding of the codebase will degrade over time. Once the serious data corruption hits the company, it will take weeks to figure it out.

I think this psychological detachment can potentially play out really bad for the whole industry. If we get stuck for too long in this weird phase, the whole tech talent pool might implode. (Is anyone working on plumbing LLMs?)

Re: The threat is comfortable drift toward not understanding what you're doing

#320

Post by steveBK123 »

csomar wrote: post #291
steveBK123 wrote: post #242

Earlier quoted context omitted.

Because a lot of valuable software is the implicit / organizational / human domain knowledge .. not the trillions of lines of code LLms all scraped and trained on.

There is a lot of software that is just code, though; especially at the foundational level.

I guess the thing is - we've always had open source, frameworks, libraries, whatever for all that though, haven't we?

So we can glue that together a bit faster, great.

What if we also stop producing new open source, frameworks, libraries, etc.

What about stories like Tailwind?

Post reply on HN