Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

251–260 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#251
post #5

Earlier quoted context omitted.

Yeah, it was saying that what matters is the process of training people to be good scientists, so they can produce other, more useful, results. That's literally what training is, everywhere. This argument boils down to "don't use tools because you'll forget how to do things the hard way", which nobody would buy for any other tool, but with LLMs we seem to have forgotten that line of reasoning entirely.

People say this in a very large number of other contexts. Mathematica has been able to do many integrals for decades and yet we still make students learn all the tricks to integrate by hand. This pattern is very common.

Yes. But to be fair to your specific point, symbolic solving of integrals used to be a huge skill in the engineering education. Nowadays, it is not a focus anymore, because numerical solutions are either sufficiently accurate or, more importantly, the only feasible approach anyway.

Re: The threat is comfortable drift toward not understanding what you're doing

#253
post #162

Earlier quoted context omitted.

Precisely. The first 10 rungs of the ladder will be removed, but we still expect you to be able to get to the roof. The AI won't get you there and you won't have the knowledge you'd normally gain on those first 10 rungs to help you move past #10.

People would have said the same about graphing calculators or calculators before that. Socrates said the same thing about the written word. The determining factor is always "did I come up with this tool". Somehow, subsequent generations always manage to find their own competencies (which, to be fair, may be different). This isn't guaranteed to play out, but it should be the default expectation until we actually see g…

If you hand a broken calculator to someone who knows how to do math, and they entered 123 + 765 which produced an answer of 6789; they should instantly know something is wrong. Hand that calculator to someone who never understood what the tool actually did but just accepted whatever answer appeared; and they would likely think the answer was totally reasonable.

Catching an LLM hallucinating often takes a basic understanding of what the answer should look like before asking the question.

Re: The threat is comfortable drift toward not understanding what you're doing

#254
post #106

I like this article and it reads well, but I have to say, that to me it really reads as something written by an LLM. Probably under supervision by a human that knew what it should say. I don't know if I mind. Example. This paragraph, to me, has a eerily perfect rhythm. The ending sentence perfectly delivers the twist. Like, why would you write in perfect prose an argument piece in the science realm? > Unlike Alice, w…

> The questions were similar. The progress was similar. The trajectory, from the outside, was identical.

LLM speak. But the rest of that quote doesn't look LLM-generated, it's too fiddly and complex of an argument. I think this was edited with AI, but the underlying argument at least is human.

Re: The threat is comfortable drift toward not understanding what you're doing

#255
post #162

Earlier quoted context omitted.

Precisely. The first 10 rungs of the ladder will be removed, but we still expect you to be able to get to the roof. The AI won't get you there and you won't have the knowledge you'd normally gain on those first 10 rungs to help you move past #10.

People would have said the same about graphing calculators or calculators before that. Socrates said the same thing about the written word. The determining factor is always "did I come up with this tool". Somehow, subsequent generations always manage to find their own competencies (which, to be fair, may be different). This isn't guaranteed to play out, but it should be the default expectation until we actually see g…

We still expect high school students to learn to use graph paper before they use their TI-83, grade school students to do arithmetic by hand before using a calculator. This is essentially the post's point, that LLMs are a useful tool only after you have learned to do the work without them.

Re: The threat is comfortable drift toward not understanding what you're doing

#256
post #152

Earlier quoted context omitted.

I am aware of that - I was adding something along the lines of: I don’t think people care if we deliver Alices any more.

People never cared about delivering Alices; they were an implementation detail. I think the article argues that they're still an important one, but one that isn't produced automatically anymore

The article is talking about science research in the context of astrophysics, not coding sweatshops.

Re: The threat is comfortable drift toward not understanding what you're doing

#258
post #179
post #162

Earlier quoted context omitted.

People would have said the same about graphing calculators or calculators before that. Socrates said the same thing about the written word. The determining factor is always "did I come up with this tool". Somehow, subsequent generations always manage to find their own competencies (which, to be fair, may be different). This isn't guaranteed to play out, but it should be the default expectation until we actually see g…

I think that's too easy an analogy, though. Calculators are deterministically correct given the right input. It does not require expert judgement on whether an answer they gave is reasonable or not. As someone who uses LLMs all day for coding, and who regularly bumps against the boundaries of what they're capable of, that's very much not the case. The only reason I can use them effectively is because I know what good…

> Calculators are deterministically correct

Calculators are deterministic, but they are not necessarily correct. Consider 32-bit integer arithmetic:

  30000000 * 1000 / 1000
  30000000 / 1000 * 1000
Mathematically, they are identical. Computationally, the results are deterministic. On the other hand, the computer will produce different results. There are many other cases where the expected result is different from what a computer calculates.

Re: The threat is comfortable drift toward not understanding what you're doing

#259
post #14
post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

> So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade…

The correct distinction is: if you can't do something without the agent, then you can't do it.

The problem that the author describes is real. I have run into it hundreds of times now. I will know how to do something, I tell AI to do it, the AI does not actually know how to do it at a fundamental level and will create fake tests to prove that it is done, and you check the work and it is wrong.

You can describe to the AI to do X at a very high-level but if you don't know how to check the outcome then the AI isn't going to be useful.

The story about the cook is 100% right. McDonald's doesn't have "chefs", they have factory workers who assemble food. The argument with AI is that working in McDonald's means you are able to cook food as well as the best chef.

The issue with hiring is that companies won't be able to distinguish between AI-driven humans and people with knowledge until it is too late.

If you have knowledge and are using AI tools correctly (i.e. not trying to zero-shot work) then it is a huge multiplier. That the industry is moving towards agent-driven workflows indicates that the AI business is about selling fake expertise to the incompetent.

Re: The threat is comfortable drift toward not understanding what you're doing

#260
post #220

Earlier quoted context omitted.

> I can't tell what specialists we will get the same way you wouldn't be able to tell me we will have Linux Kernel specialists at the year 1945. How about addressing astrophysics specifically. What are you claiming about it? Are you claiming that in the future, we won't need astrophysicists at all, AI can do all of our astrophysics for us, freeing humans to specialize in... other subjects? And doesn't the same proble…

> The specific subjects have changed over time, but the production of specialist mathematicians hasn't really changed. It takes hard work, grunt work, struggling, making mistakes and learning from them, as well as expert supervision. The problem with AI is that it encourages and incentivizes intellectual laziness, the opposite of what is required to produce specialists Let's take the example of economics. Economists…

> I'm simply claiming that not everyone needs to be Linux Kernel specialists.

This is an uninteresting and indeed silly claim, because nobody has ever asserted the opposite.

The point is that society needs some Linux kernel specialists, and some astrophysicists, but AI is undermining their production.

> This is precisely what AI is allowing: it automates things I don't need to know deeply so that I can focus on things I do need to understand deeply.

The submitted article is about how AI is automating the things that a specialist does need to understand deeply. It's about so-called astrophysicists using AI to produce astrophysics papers, not about how non-astrophysicists use AI to produce astrophysics papers so that they can focus on whatever their non-astrophysics specialty may be, if they have any specialty.

Post reply on HN