Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

161–170 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#161

Post by lapcat »

simianwords wrote: post #54

> Frank Herbert (yeah, I know I'm a nerd), in God Emperor of Dune, has a character observe: "What do such machines really do? They increase the number of things we can do without thinking. Things we do without thinking; there's the real danger." Herbert was writing science fiction. I'm writing about my office. The distance between those two things has gotten uncomfortably small. The author is a bit naive here: 1. Soc…

I think you missed the point. The entire article is about specialists: astrophysicists. The problem with AI is that specialists are delegating their thinking about their specialty! The fear here is that society will stop producing specialists, and thus society will no longer progress.

Re: The threat is comfortable drift toward not understanding what you're doing

#162

Post by NiloCK »

jacquesm wrote: post #21
djaro wrote: post #14

Earlier quoted context omitted.

> So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade…

Precisely. The first 10 rungs of the ladder will be removed, but we still expect you to be able to get to the roof. The AI won't get you there and you won't have the knowledge you'd normally gain on those first 10 rungs to help you move past #10.

People would have said the same about graphing calculators or calculators before that. Socrates said the same thing about the written word.

The determining factor is always "did I come up with this tool". Somehow, subsequent generations always manage to find their own competencies (which, to be fair, may be different).

This isn't guaranteed to play out, but it should be the default expectation until we actually see greatly diminishing outputs at the frontier of science, engineering, etc.

Re: The threat is comfortable drift toward not understanding what you're doing

#163

Post by zozbot234 »

djaro wrote: post #14
sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

> So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade…

Real-world cooks don't exactly avoid those newfangled microwave ovens though. They use them as a professional tool for simple tasks where they're especially suitable (especially for quick defrosting or reheating), which sometimes allows them to cook even better meals.

Re: The threat is comfortable drift toward not understanding what you're doing

#164

Post by simianwords »

lapcat wrote: post #161
simianwords wrote: post #54

> Frank Herbert (yeah, I know I'm a nerd), in God Emperor of Dune, has a character observe: "What do such machines really do? They increase the number of things we can do without thinking. Things we do without thinking; there's the real danger." Herbert was writing science fiction. I'm writing about my office. The distance between those two things has gotten uncomfortably small. The author is a bit naive here: 1. Soc…

I think you missed the point. The entire article is about specialists: astrophysicists. The problem with AI is that specialists are delegating their thinking about their specialty! The fear here is that society will stop producing specialists, and thus society will no longer progress.

You are assuming that set of specialists are fixed system! That's not the case. With change in technology, you would get more and more specialists, the same way Agricultural revolution allowed for more specialists to exist.

Re: The threat is comfortable drift toward not understanding what you're doing

#165

Post by FrojoS »

Every PhD program I'm aware of has a final hurdle known as the defence. You have to present your thesis while standing in front of a committee, and often the local community and public. They will asks questions and too many "I don't know" or false answers would make you fail. So, there is already a system in place that should stop Bob from graduating if he indeed learned much less than Alice. A similar argument can be made for conference publications. If Bob publishes his first year project at a conference but doesn't actually understand "his own work" it will show.

The difficulty of passing the defence vary's wildly between Universities, departments and committees. Some are very serious affairs with a decent chance of failure while others are more of a show event for friends and family. Mine was more of the latter, but I doubt I would have passed that day if I had spend the previous years prompting instead of doing the grunt work.

Re: The threat is comfortable drift toward not understanding what you're doing

#166

Post by Lionga »

roenxi wrote: post #105
b00ty4breakfast wrote: post #104

Earlier quoted context omitted.

>But there is also a more subtle thing, which is we're trending towards superintelligence with these AIs do you have any evidence for that, though? Besides marketing claims, I mean.

I've always quite liked https://ourworldindata.org/grapher/test-scores-ai-capabiliti... to show that once AIs are knocking at the door of a human capability they tend to overshoot in around a decade.

This is just trash, like almost any AI benchmark. E.g. it says since around 2015 speech recognition is above human yet any any speech input today has more errors than any human would have.

If I would not type but speak this comment maybe 2 to 5 words would be wrong. For a human it is maybe 10% of that.

Re: The threat is comfortable drift toward not understanding what you're doing

#167

Post by BobBagwill »

Try giving this problem to different AI LLM chatbots:

If I could make a rocket that could accelerate at 3 Gs for 10 years, how long would it take to travel from Earth to Alpha Centauri by accelerating at 3 Gs for half the time, then decelerating at 3 Gs for half the time?

Hint: They don't all get it right. Some of them never got it right after hints, corrections, etc.

Re: The threat is comfortable drift toward not understanding what you're doing

#168

Post by sd9 »

KronisLV wrote: post #160
sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

> I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I dread the flip side of this which is dealing with obtuse bullshit like trying to understand why Oracle ADF won’t render forms properly, or how to optimize some codebase with a lot of N+1 calls when there’s looming deadlines and the original devs never made it scalable, or needing to dig into…

Oh yes, it’s an entirely privileged position to be able to enjoy your work. But it’s a privilege I have enjoyed and not one I want to give up unless I have to. We spend an extraordinary amount of our waking life at work.

Re: The threat is comfortable drift toward not understanding what you're doing

#169

Post by wongarsu »

jacquesm wrote: post #21
djaro wrote: post #14

Earlier quoted context omitted.

> So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade…

Precisely. The first 10 rungs of the ladder will be removed, but we still expect you to be able to get to the roof. The AI won't get you there and you won't have the knowledge you'd normally gain on those first 10 rungs to help you move past #10.

There are a lot of people in academia who are great at thinking about complex algorithms but can't write maintainable code if their life depended on it. There are ways to acquire those skills that don't go the junior developer route. Same with debugging and profiling skills

But we might see a lot more specialization as a result

Re: The threat is comfortable drift toward not understanding what you're doing

#170

Post by lelanthran »

sd9 wrote: post #152
lelanthran wrote: post #137

Earlier quoted context omitted.

> The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. "Being able to deliver using AI" wasn't the point of the article. If it was the point, your comment would make sense. The point of the program referred to in the article is not to deliver results, but to deliver an Alice. Delivering a Bob is a failure of the program. Whether you think that a Bob+AI delivers the same resul…

I am aware of that - I was adding something along the lines of: I don’t think people care if we deliver Alices any more.

> I am aware of that - I was adding something along the lines of: I don’t think people care if we deliver Alices any more.

That's irrelevant to the goal of the program - they care. Once they stop caring, they'd shut that program down.

Maybe it would be replaced with a new program that has the goal of delivering Bobs+AI, but what would be the point? I mean, the article explained in depth that there is no market for the results currently, so what would be the point of efficiently generating those results?

The market currently does not want the results, so replacing the current program with something that produces Bobs+AI would be for... what, exactly?

Post reply on HN