Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

341–350 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#341

Post by wizzwizz4 »

jnovek wrote: post #249
wizzwizz4 wrote: post #88

Earlier quoted context omitted.

From the article: > There's a common rebuttal to this, and I hear it constantly. "Just wait," people say. "In a few months, in a year, the models will be better. They won't hallucinate. They won't fake plots. The problems you're describing are temporary." I've been hearing "just wait" since 2023. We're not trending towards superintelligence with these AIs. We're trending towards (and, in fact, have already reached) s…

The rate of hallucination has gone down drastically since 2023. As LLM coding tools continue to pare that rate down, eventually we’ll hit a point where it is comparable to the rate we naturally introduce bugs as humans programmers.

LLMs are still making fundamentally the same kinds of errors that they made in 2021. If you check my HN comment history, you'll see I predicted these errors, just from skimming the relevant academic papers (which is to say they're obvious: I'm far from the only person saying this). There is no theoretical reason we should expect them to go away, unless the model architectures fundamentally change (and no, GPT -> LLaMA is not a fundamental change), because they're not removable discontinuities: they're indicative of fundamental capability gaps.

I don't care how many terms you add to your Taylor series: your polynomial approximation of a sine wave is never going to be suitable for additive speech synthesis. Likewise, I don't care how good your predictive-text transformer model gets at instrumental NLP subtasks: it will never be a good programmer (except as far as it's a plagiarist). Just look at the Claude Code source code: if anyone's an expert in agentic AI development, it's the Claude people, and yet the codebase is utterly unmaintainable dogshit that shouldn't work and, on further inspection, doesn't work.

That's not to say that no computer program can write computer programs, but this computer program is well into the realm of diminishing returns.

Re: The threat is comfortable drift toward not understanding what you're doing

#342

Post by vrganj »

lxgr wrote: post #270
vrganj wrote: post #197

Earlier quoted context omitted.

At some point the herding of idiot savants becomes more work than just doing the damn thing yourself in the first place.

I'm happy to herd idiots all my life if they come out of it smarter than they went in . The real tragedy with current LLM agents is that they're effectively stateless, and so all the effort of "educating" them feels wasted. Once continuous learning is solved, I predict the problem addressed by TFA to become orders of magnitude bigger: What's the motivation for anyone to teach a person if an LLM can learn it much fast…

At that point, I think it'll be time to admit to ourselves that capitalism is over.

The only reason we somewhat made it work is due to the interdependence between labor and capital. Once that's broken, the wheels will start falling off.

Re: The threat is comfortable drift toward not understanding what you're doing

#343

Post by pigeons »

sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

> So if Bob can do things with agents, he can do things.

But he does things wrong.

Re: The threat is comfortable drift toward not understanding what you're doing

#344

Post by Wowfunhappy »

NiloCK wrote: post #162
jacquesm wrote: post #21

Earlier quoted context omitted.

Precisely. The first 10 rungs of the ladder will be removed, but we still expect you to be able to get to the roof. The AI won't get you there and you won't have the knowledge you'd normally gain on those first 10 rungs to help you move past #10.

People would have said the same about graphing calculators or calculators before that. Socrates said the same thing about the written word. The determining factor is always "did I come up with this tool". Somehow, subsequent generations always manage to find their own competencies (which, to be fair, may be different). This isn't guaranteed to play out, but it should be the default expectation until we actually see g…

> People would have said the same about graphing calculators or calculators before that.

As it happens, we generally don't let people use calculators while learning arithmetic. We make children spend years using pencil and paper to do what a calculator could in seconds.

Re: The threat is comfortable drift toward not understanding what you're doing

#345

Post by cmiles74 »

leereeves wrote: post #327
Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

> And so the paradox is, the LLMs are only useful† if you're Schwartz Was the LLM even useful for Schwartz, if it produced false output?

Maybe it saved them some time? So far the studies seem to lean toward probably the LLM didn't save them any time.

Re: The threat is comfortable drift toward not understanding what you're doing

#347

Post by steveBK123 »

mikeaskew4 wrote: post #221
steveBK123 wrote: post #211

For the people arguing that the output is the code and the faster we generate it the better.. I do wonder where all the novel products produced by 10x devs who are now 100x with LLMs, the “idea guys” who can now produce products from whole clothe without having to hire pesky engineers.. where is the one-man 10 billion dollar startups, etc? We are 3-4 years into this mania and all I see on the other end of it is the L…

Could be possible that the 10x devs working at 100x are just starting down the homestretch… The 10x dev doesn’t just set out to build a hello world app, ya know.

I think its telling that the two main places I've seen the biggest in-roads in FinTech in terms of LLMs has been:

1) Stuff that was astonishingly not automated yet. I am talking about somebody opening up excel on one screen, and a website/pdf/whatever on the other.. and type stuff in to your excel sheet. So stuff where there wasn't any code involved previously, possibly due to diminishing returns of how adhoc it was to automate, skills mismatch, organizational politics or other reasons.

2) Lot of former BigData / crypto / SaaS guys who were in product/sales roles suddenly starting AI startups to help your company AI better. The product is facilitating the doing of AI.

Re: The threat is comfortable drift toward not understanding what you're doing

#348

Post by saulpw »

zozbot234 wrote: post #207
DavidPiper wrote: post #89

I've just started a new role as a senior SWE after 5 months off. I've been using Claude a bit in my time off; it works really well. But now that I've started using it professionally, I keep running into a specific problem: I have nothing to hold onto in my own mind. How this plays out: I use Claude to write some moderately complex code and raise a PR. Someone asks me to change something. I look at the review and thin…

Ask Claude to explain the code in depth for you. It's a language model, it's great at taking in obscure code and writing up explanations of how it works in plain English. You can do this during the previous change phase of course. Just ask "How would one plan this change to the codebase? Could you explain in depth why?" If you're expected to be thoroughly familiar with that code, it makes no sense to skip that step.

This is like asking Claude to explain some aspect of physics to you. It'll 'feel' like you understand, but in order to really understand you have to work those annoying problems.

Same with anything. You can read about how to meditate, cook, sew, whatever. But if you only read about something, your mental model is hollow and purely conceptual, having never had to interact with actual reality. Your brain has to work through the problems.

Re: The threat is comfortable drift toward not understanding what you're doing

#349

Post by conception »

FrojoS wrote: post #336
conception wrote: post #310

Earlier quoted context omitted.

Sadly I don’t see how our current social paradigm works for this. There is no history of any sort of long planning like this or long term loyalty (either direction) with employees and employers for this sort of journeyman guild style training. AI execs are basically racing, hoping we won’t need a Schwartz before they are all gone. But what incentives are in place to high a college grad, have them work without llms fo…

> There is no history of any sort of long planning Sure there is. Its the formal education system that produced the college grad.

… between employees and employers.

The proposal that everyone pay for college until they are in their 40s doesn’t seem viable.

Re: The threat is comfortable drift toward not understanding what you're doing

#350

Post by conception »

Wowfunhappy wrote: post #329
conception wrote: post #310

Earlier quoted context omitted.

Sadly I don’t see how our current social paradigm works for this. There is no history of any sort of long planning like this or long term loyalty (either direction) with employees and employers for this sort of journeyman guild style training. AI execs are basically racing, hoping we won’t need a Schwartz before they are all gone. But what incentives are in place to high a college grad, have them work without llms fo…

Then the social paradigm needs to change. Is everyone just going to roll over and die while AI destroys academia (and possibly a lot more)? Last September, Tyler Austin Harper published a piece for The Atlantic on how he thinks colleges should respond to AI. What he proposes is radical —but, if you've concluded that AI really is going to destroy everything these institutions stand for, I think you have to at least co…

Well, we are already rolling over and dying (literally) on everything from vaccine denial to climate change. So, yes, we are. Obviously yes.
Post reply on HN