Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

41–50 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#41

Post by rglullis »

stavros wrote: post #5
hirako2000 wrote: post #4

Earlier quoted context omitted.

I was reading in the article that what matters is the process that leads to the (typically useless) result, what the people get out of it. Once I realized that this white on black contrast was hurting my eyes, I decided to stop as I didn't want to see stripes for too long when looking away. Some activity has outcomes that aren't strictly in the results.

Yeah, it was saying that what matters is the process of training people to be good scientists, so they can produce other, more useful, results. That's literally what training is, everywhere. This argument boils down to "don't use tools because you'll forget how to do things the hard way", which nobody would buy for any other tool, but with LLMs we seem to have forgotten that line of reasoning entirely.

> so they can produce other, more useful, results

But to even *know* what is more useful, it is crucial to have walked the walk. Otherwise we will all end up with a bunch of people trying to reinvent the wheel, over and over again, like JavaScript "developers" who keep reinventing frameworks every six months.

> which nobody would buy for any other tool

I don't know about you, but I wasn't allowed to use calculators in my calculus classes precisely to learn the concepts properly. "Calculators are for those who know how to do it by hand" was something I heard a lot from my professors.

Re: The threat is comfortable drift toward not understanding what you're doing

#42

Post by stavros »

troupo wrote: post #35
stavros wrote: post #12

Earlier quoted context omitted.

Then test Bob on what you actually want him to produce, ie novel problems, instead of trivial things that won't tell you how good he is. Why is it a problem of the LLM if your test is unrelated to the performance you want?

How can Bob produce novel things when he lacks the skills to do even trivial things? I didn't get to be a senior engineer by immediately being able to solve novel problems. I can now solve novel problems because I spent untold hours solving trivial ones.

Because trivial things aren't a prerequisite for novel things, as any theoretical mathematician who can't do long division will tell you.

Re: The threat is comfortable drift toward not understanding what you're doing

#43

Post by sd9 »

nidnogg wrote: post #36
sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

I don't like it either. But what is really guaranteeing other markets from flunking similarly later on? What's to say other jobs are going to be any better? Back in college, most of my peers would say "I'm not cut out for anything else. This is it". They were, sure enough, computer and/or math people at heart from an early age. More importantly, what's gonna be the next stable category of remote-first jobs that a per…

I don’t know, it’s only since about December that I felt things really start to shift, and February when my job started to become very different.

Personally I’m looking at more physical domains, but it’s early days in my exploration. I think if I wanted to stick to remote work (which I have enjoyed since 2020), then the AI story would just keep playing out.

I’m also totally open to taking a big pay cut to do something I actually enjoy day to day, which I guess makes it easier.

Re: The threat is comfortable drift toward not understanding what you're doing

#44

Post by b112 »

djaro wrote: post #14
sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

> So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade…

Worse, soon fewer and fewer people will taste good food, including even higher and higher scale restaurants just using pre-made.

As fewer know what good food tastes like, the entire market will enshitify towards lower and lower calibre food.

We already see this with, for example, fruits in cold climates. I've known people who have only ever bought them from the supermarket, then tried them at a farmers when they're in season for 2 weeks. The look of astonishment on their faces, at the flavour, is quite telling. They simply had no idea how dry, flavourless supermarket fruit is.

Nothing beats an apple picked just before you eat it.

(For reference, produce shipped to supermarkets is often picked, even locally, before being entirely ripe. It last longer, and handles shipping better, than a perfectly ripe fruit.)

The same will be true of LLMs. They're already out of "new things" to train on. I question that they'll ever learn new languages, who will they observe to train on? What does it matter if the code is unreadable by humans regardless?

And this is the real danger. Eventually, we'll have entire coding languages that are just weird, incomprehensible, tailored to LLMs, maybe even a language written by an LLM.

What then? Who will be able to decipher such gibberish?

Literally all true advancement will stop, for LLMs never invent, they only mimic.

Re: The threat is comfortable drift toward not understanding what you're doing

#45

Post by troupo »

stavros wrote: post #42
troupo wrote: post #35

Earlier quoted context omitted.

How can Bob produce novel things when he lacks the skills to do even trivial things? I didn't get to be a senior engineer by immediately being able to solve novel problems. I can now solve novel problems because I spent untold hours solving trivial ones.

Because trivial things aren't a prerequisite for novel things, as any theoretical mathematician who can't do long division will tell you.

Ah yes. The famous theoretical mathematicians who immediately started on novel problems in theoretical mathematics without first learning and understanding a huge number of trivial things like how division works to begin with, what fractions are, what equations are and how they are solved etc.

Edit: let's look at a paper like Some Linear Transformations on Symmetric Functions Arising From a Formula of Thiel and Williams https://ecajournal.haifa.ac.il/Volume2023/ECA2023_S2A24.pdf and try and guess how many of trivial things were completely unneeded to write a paper like this.

Re: The threat is comfortable drift toward not understanding what you're doing

#46

Post by sd9 »

troupo wrote: post #34
sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

> The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. Can he? If he outsources all his thinking and understanding to agents, can he then fix things he doesn't know how to fix without agents? Any skill is practice first and foremost. If Bob has had no practice, what then?

My point is it doesn’t matter whether he can fix things without agents. The real world isn’t an exam hall where your boss tells you “no naughty AI!”, you just get stuff done, and if Bob can do that with agents, nobody cares how he did it.

Re: The threat is comfortable drift toward not understanding what you're doing

#47

Post by troupo »

sd9 wrote: post #46
troupo wrote: post #34

Earlier quoted context omitted.

> The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. Can he? If he outsources all his thinking and understanding to agents, can he then fix things he doesn't know how to fix without agents? Any skill is practice first and foremost. If Bob has had no practice, what then?

My point is it doesn’t matter whether he can fix things without agents. The real world isn’t an exam hall where your boss tells you “no naughty AI!”, you just get stuff done, and if Bob can do that with agents, nobody cares how he did it.

> The real world isn’t an exam hall where your boss tells you “no naughty AI!”, you just get stuff done, and if Bob can do that with agents, nobody cares.

Indeed. That's why Anthropic had to hire real engineers to make sure their vibe-coded shit doesn't consume 68GB of RAM. Because real world: https://x.com/jarredsumner/status/2026497606575398987

Re: The threat is comfortable drift toward not understanding what you're doing

#48

Post by Folcon »

stavros wrote: post #42
troupo wrote: post #35

Earlier quoted context omitted.

How can Bob produce novel things when he lacks the skills to do even trivial things? I didn't get to be a senior engineer by immediately being able to solve novel problems. I can now solve novel problems because I spent untold hours solving trivial ones.

Because trivial things aren't a prerequisite for novel things, as any theoretical mathematician who can't do long division will tell you.

There's a difference between needing no trivial skills to do novel things and not needing specific prerequisite trivial skills to do a novel thing

Re: The threat is comfortable drift toward not understanding what you're doing

#49

Post by stavros »

troupo wrote: post #45
stavros wrote: post #42

Earlier quoted context omitted.

Because trivial things aren't a prerequisite for novel things, as any theoretical mathematician who can't do long division will tell you.

Ah yes. The famous theoretical mathematicians who immediately started on novel problems in theoretical mathematics without first learning and understanding a huge number of trivial things like how division works to begin with, what fractions are, what equations are and how they are solved etc. Edit: let's look at a paper like Some Linear Transformations on Symmetric Functions Arising From a Formula of Thiel and Willi…

Seems that teaching Bob trivial things would be a simple solution to this predicament.
Post reply on HN