Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

171–180 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#171

Post by atoav »

sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

The thing is Bob can use HammerAsAService™ to put in a nail. It is so cheap! Way cheaper than buying an actual hammer.

The problem with unlearning generic tools and relying on ones you rent by big corporations is that it is unreliable in the long term. The prices will be rising. The conditions will worsen. Oh nice that Bob made a thing using HammerAsAService™, but the terms of conditions (changing once a week) he accepted last week clearly say it belongs to the company now. Bob should be happy they are not suing him yet, but Bob isn't sure whether the thing that came out a month after was independently developed by that company or not just a clone of his work. Bob wishes he knew how to use a hammer.

Re: The threat is comfortable drift toward not understanding what you're doing

#172

Post by sd9 »

lelanthran wrote: post #170
sd9 wrote: post #152

Earlier quoted context omitted.

I am aware of that - I was adding something along the lines of: I don’t think people care if we deliver Alices any more.

> I am aware of that - I was adding something along the lines of: I don’t think people care if we deliver Alices any more. That's irrelevant to the goal of the program - they care. Once they stop caring, they'd shut that program down. Maybe it would be replaced with a new program that has the goal of delivering Bobs+AI, but what would be the point? I mean, the article explained in depth that there is no market for th…

There’s no market for the results, but there was a market for Alices, because they were the only people who could produce similar results historically. Now maybe there’s less of a market for Alices. Yes, maybe that means the program disappears.

Re: The threat is comfortable drift toward not understanding what you're doing

#173

Post by zozbot234 »

kelnos wrote: post #90
roenxi wrote: post #40

Earlier quoted context omitted.

That doesn't sound like much of an issue. Bob was already going to encounter problems that are too large and complex for him to solve, agents or otherwise. Life throws us hard problems. I don't recall if we even assumed Bob was unusually capable, he might be one of life's flounderers. I'd give good odds that if he got through a program with the help of agents he'll get through life achieving at least a normal level o…

> we're trending towards superintelligence with these AIs The article addresses this, because, well... no we aren't. Maybe we are. But it's far from clear that we're not moving toward a plateau in what these agents can do. > Whether a human does actual work or not isn't particularly exciting to a market. You seem to be convinced these AI agents will continue to improve without bound, so I think this is where the disc…

There's no good definition of superintelligence. A calculator is already way more capable than any human at doing simple mathematical operations, and even small AIs for local use can instantly recall all sorts of impressive knowledge about virtually any field of study, which would be unfeasible for any human; but neither of those is what people mean when they wonder whether future AIs will have superintelligence.

Re: The threat is comfortable drift toward not understanding what you're doing

#174

Post by lapcat »

simianwords wrote: post #164
lapcat wrote: post #161

Earlier quoted context omitted.

I think you missed the point. The entire article is about specialists: astrophysicists. The problem with AI is that specialists are delegating their thinking about their specialty! The fear here is that society will stop producing specialists, and thus society will no longer progress.

You are assuming that set of specialists are fixed system! That's not the case. With change in technology, you would get more and more specialists, the same way Agricultural revolution allowed for more specialists to exist.

This comment sounds like hand-waving to me.

The author describes specifically how specialists are produced and how AI undermines their production.

No, we won't get more and more specialists literally "the same way" as the agricultural revolution. You need to be much more specific about how we'll get more specialists under the incentive structure created by AI, otherwise this sounds like some kind of religious faith in AI and progress.

Re: The threat is comfortable drift toward not understanding what you're doing

#175

Post by danielbln »

gedy wrote: post #103
stavros wrote: post #3

I see this fallacy being committed a lot these days. "Because LLMs, you will no longer need a skill you don't need any more, but which you used to need, and handwaves that's bad". Academia doesn't want to produce astrophysics (or any field) scientists just so the people who became scientists can feel warm and fuzzy inside when looking at the stars, it wants to produce scientists who can produce useful results. Bob pr…

We aren't talking pocket calculators here (I see the irony of phone app in pocket), LLMs are hugely expensive things made and controlled behind costly commercial subscriptions. And likely in the middle of a huge investment bubble and stability is uncertain. So we all need to be careful about "gee we don't need that skill or person anymore", etc.

Open weight models that run under your desk are not frontier model level, but they are getting closer. Improvements in agentic post training and things like TurboQuant mean that even if all frontier labs pull the plug tomorrow, we will still have agents to work with.

Re: The threat is comfortable drift toward not understanding what you're doing

#177

Post by simianwords »

lapcat wrote: post #174
simianwords wrote: post #164

Earlier quoted context omitted.

You are assuming that set of specialists are fixed system! That's not the case. With change in technology, you would get more and more specialists, the same way Agricultural revolution allowed for more specialists to exist.

This comment sounds like hand-waving to me. The author describes specifically how specialists are produced and how AI undermines their production. No, we won't get more and more specialists literally "the same way" as the agricultural revolution. You need to be much more specific about how we'll get more specialists under the incentive structure created by AI, otherwise this sounds like some kind of religious faith i…

I can't tell what specialists we will get the same way you wouldn't be able to tell me we will have Linux Kernel specialists at the year 1945.

People do more things with AI.

More things = more inventions = the field growing.

The field grows and people become specialists on what used to be a small or trivial.

A mathematician in 1500's wouldn't think algebraic topology would be a specialisation.

Re: The threat is comfortable drift toward not understanding what you're doing

#178

Post by Lerc »

mbbutler wrote: post #79
oncallthrow wrote: post #16

I think this article is largely, or at least directionally, correct. I'd draw a comparison to high-level languages and language frameworks. Yes, 99% of the time, if I'm building a web frontend, I can live in React world and not think about anything that is going on under the hood. But, there is 1% of the time where something goes wrong, and I need to understand what is happening underneath the abstraction. Similarly,…

In my personal experience, the rate at which Claude Code produces suboptimal Rust is way higher than 1%.

That is dependent upon the quality of the AI. The argument is not about the quality of the components but the method used.

It's trivial to say using an inadequate tool will have an inadequate result.

It's only an interesting claim to make if you are saying that there is no obtainable quality of the tool that can produce an adequate result (In this argument, the adequate result in question is a developer with an understanding of what they produce)

Re: The threat is comfortable drift toward not understanding what you're doing

#179

Post by lukev »

NiloCK wrote: post #162
jacquesm wrote: post #21

Earlier quoted context omitted.

Precisely. The first 10 rungs of the ladder will be removed, but we still expect you to be able to get to the roof. The AI won't get you there and you won't have the knowledge you'd normally gain on those first 10 rungs to help you move past #10.

People would have said the same about graphing calculators or calculators before that. Socrates said the same thing about the written word. The determining factor is always "did I come up with this tool". Somehow, subsequent generations always manage to find their own competencies (which, to be fair, may be different). This isn't guaranteed to play out, but it should be the default expectation until we actually see g…

I think that's too easy an analogy, though.

Calculators are deterministically correct given the right input. It does not require expert judgement on whether an answer they gave is reasonable or not.

As someone who uses LLMs all day for coding, and who regularly bumps against the boundaries of what they're capable of, that's very much not the case. The only reason I can use them effectively is because I know what good software looks like and when to drop down to more explicit instructions.

Post reply on HN