Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

381–390 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#381

Post by thepasch »

atoav wrote: post #171
sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

The thing is Bob can use HammerAsAService™ to put in a nail. It is so cheap! Way cheaper than buying an actual hammer. The problem with unlearning generic tools and relying on ones you rent by big corporations is that it is unreliable in the long term. The prices will be rising. The conditions will worsen. Oh nice that Bob made a thing using HammerAsAService™, but the terms of conditions (changing once a week) he acc…

The majority of nails people might want to rent a HammerAsAService for these days can already easily be put in by open source hammers you can run on consumer, uh… workbenches.

Re: The threat is comfortable drift toward not understanding what you're doing

#382

Post by Wowfunhappy »

yoyohello13 wrote: post #379
Wowfunhappy wrote: post #344

Earlier quoted context omitted.

> People would have said the same about graphing calculators or calculators before that. As it happens, we generally don't let people use calculators while learning arithmetic. We make children spend years using pencil and paper to do what a calculator could in seconds.

This is why I don’t understand the calculator analogy. Letting beginners use LLMs is like if we gave kids calculators in 1st grade and told Timmy he never needs to learn 2 + 2. That’s not how education works today.

I think this is exactly why calculators are a great analogy, and a hint toward how we should probably treat LLMs.

Re: The threat is comfortable drift toward not understanding what you're doing

#383

Post by atomicnumber3 »

cmiles74 wrote: post #317
Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

I think we already know what we need to do: encourage people to do the work themselves, discourage beginners from immediately asking an LLM for help and re-introducing some kind of oral exam. As the article mentions, banning LLMs is impractical and what we really need are people who can tell when the LLM is confidently wrong; not people who don't know how to work with an LLM. I hope it will encourage people to think…

I think we'll get there. We need to get at least some AI bust going first though. It's impossible to talk sense into people who think AI is about to completely replace engineers, or even those who think that, while it might not replace engineers, it's going to be doing 100% of all coding within a year. Or even that it can do 100% of coding right now.

There's a couple unfortunate truths going on all at the same time:

- People with money are trying to build the "perfect" business: SaaS without software eng headcount. 100% margin. 0 Capex. And finally near-0 opex and R&D cost. Or at least, they're trying to sell the idea of this to anyone who will buy. And unfortunately this is exactly what most investors want to hear, so they believe every word and throw money at it. This of course then extends to many other business and not just SaaS, but those have worse margins to start with so are less prone to the wildfire.

- People who used to code 15 years ago but don't now, see claude generating very plausible looking code. Given their job is now "C suite" or "director", they don't perceive any direct personal risk, so the smell test is passed and they're all on board, happily wreaking destruction along the way.

- People who are nominally software engineers but are bad at it are truly elevated 100x by claude. Unfortunately, if their starting point was close to 0, this isn't saying a lot. And if it was negative, it's now 100x as negative.

- People who are adjacent to software engineering, like PMs, especially if they dabble in coding on the side, suddenly also see they "can code" now.

Now of course, not all capital owners, CTOs, PMs, etc exhibit this. Probably not even most. But I can already name like 4 example per category above from people I know. And they're all impossible to explain any kind of nuance to right now. There's too many people and articles and blog posts telling them they're absolutely right.

We need some bust cycle. Then maybe we can have a productive discussion of how we can leverage LLMs (we'll stop calling it "AI"...) to still do the team sport known as software engineering.

Because there's real productivity gains to be had here. Unfortunately, they don't replace everyone with AGI or allow people who don't know coding or software engineering to build actual working software, and they don't involve just letting claude code stochastically generate a startup for you.

Re: The threat is comfortable drift toward not understanding what you're doing

#384

Post by ThrowawayR2 »

AnimalMuppet wrote: post #271
sd9 wrote: post #64

Earlier quoted context omitted.

Lovely story, thanks for sharing. I wonder how many assembly programmers got over it and retrained, versus moved on to do something totally different. I find the agentic way of working simultaneously more exhausting and less stimulating. I don’t know if that’s something I’m going to get over, or whether this is the end of the line for me.

I wasn't there at the time, but I believe that most assembly programmers learned higher-level languages. My mother actually started programming in octal . I don't remember her exact words, but she said something to the effect that her life got so much better when she got an assembler. I suspect that going from assembly to compilers was much the same - you no longer had to worry about register allocations and building…

It was a trade-off for a very long time (late 1960s to late 1990s IMO): the output of the early compilers was much less efficient than hand writing assembly language but it enabled less skilled programmers to produce working programs. Compilers pulled ahead when eventually processor ISAs evolved to optimize executing compiler generated code (e.g. the CISC -> RISC transition) and optimizing compilers became practical because of more powerful hardware. It definitely was not an overnight transformation.

Re: The threat is comfortable drift toward not understanding what you're doing

#385

Post by nathan_compton »

FrojoS wrote: post #340
nathan_compton wrote: post #313

Earlier quoted context omitted.

There is much more to life than engineering.

Sorry, I should have quoted properly in my reply. My first sentence ("Yes.") was in general agreement with you, the second sentence was specifically about > Mathematica has been able to do many integrals for decades and yet we still make students learn all the tricks to integrate by hand But maybe, integrating by hand is still as big as ever in other parts of academia. Or were you thinking about high school? I'm fair…

I studied physics and mathematics and finding analytic solutions to problems is still useful and enlightening.

Re: The threat is comfortable drift toward not understanding what you're doing

#386

Post by Wowfunhappy »

atomicnumber3 wrote: post #383
cmiles74 wrote: post #317

Earlier quoted context omitted.

I think we already know what we need to do: encourage people to do the work themselves, discourage beginners from immediately asking an LLM for help and re-introducing some kind of oral exam. As the article mentions, banning LLMs is impractical and what we really need are people who can tell when the LLM is confidently wrong; not people who don't know how to work with an LLM. I hope it will encourage people to think…

I think we'll get there. We need to get at least some AI bust going first though. It's impossible to talk sense into people who think AI is about to completely replace engineers, or even those who think that, while it might not replace engineers, it's going to be doing 100% of all coding within a year. Or even that it can do 100% of coding right now. There's a couple unfortunate truths going on all at the same time:…

> Or even that [AI] can do 100% of coding right now.

I don't actually think the article refutes this. But the AI needs to be in the hands of someone who can review the code (or astrophysics paper), notice and understand issues, and tell the AI what changes to make. Rinse, repeat. It's still probably faster than writing all the code yourself (but that doesn't mean you can fire all your engineers).

The question is, how do you become the person who can effectively review AI code without actually writing code without an AI? I'd argue you basically can't.

Re: The threat is comfortable drift toward not understanding what you're doing

#387

Post by Planktonne »

thepasch wrote: post #212
lelanthran wrote: post #158

Earlier quoted context omitted.

> The question is whether it’s more important to be able to do things, or more important to have a good sense and a keen eye for what to do at any given moment. Those aren't mutually exclusive. "People who do things" can do both, and doing the latter is a function of doing the former, so they tend to do the latter sufficiently well. "People who prompt things" can only do the latter, and they routinely do it poorly.

> “People who prompt things” can only do the latter, and they routinely do it poorly. Right, but what I don’t agree with here is the idea that this category of people will never be able to improve into the first category of people. The value of an experienced anything is that they realize there is a big chasm between something that works now and something that will continue to work long into the future. I don’t agree…

> the idea that this category of people will never be able to improve into the first category of people

The fundamental difference between the categories is that the first is filled with people who put the effort in to learning/understanding, and the second is filled with people who take the shortcut around learning/understanding.

Changing from the second category to the first is something that would require already being in the first.

Re: The threat is comfortable drift toward not understanding what you're doing

#388

Post by thepasch »

sgarland wrote: post #108
stavros wrote: post #42

Earlier quoted context omitted.

Because trivial things aren't a prerequisite for novel things, as any theoretical mathematician who can't do long division will tell you.

I would love to see someone attempt to do multiplication who never learned addition, or exponentiation without having learned multiplication. There is a vast difference between “never learned the skill,” and “forgot the skill from lack of use.” I learned how to do long division in school, decades ago. I sat down and tried it last year, and found myself struggling, because I hadn’t needed to do it in such a long time.

> There is a vast difference between “never learned the skill,” and “forgot the skill from lack of use.”

This sentence contains the entire point, and the easiest way to get there, as with many, many things, is to ask “why?”

Re: The threat is comfortable drift toward not understanding what you're doing

#389

Post by ChrisMarshallNY »

cmiles74 wrote: post #321
ChrisMarshallNY wrote: post #315

This is not wrong, but the "Bob and Alice" conundrum is not simple, either. In academia, understanding is vital. The same for research. But in production , results are what matters. Alice would be a better researcher, but Bob would be a better producer. He knows how to wrangle the tools. Each has its value. Many researchers develop marvelous ideas, but struggle to commercialize them, while production-oriented enginee…

I have to disagree that Bob will be a better producer, although I do agree that Bob will produce more . In this scenario, Bob isn't clear on which LLM output is valid and important and which is erroneous and misleading; I think that's a pretty critical distinction. It's the kind of thing that might go undetected for a long time, until a particular paper turns out to be important and it's discovered that it's also ent…

Sounds like you're still thinking of Bob as a researcher.

In production, there would be no "paper"; just some software/hardware product.

If there was a problem, that would be fairly obvious, with testing (we are going to be testing our products, right?).

I have been wrestling all morning, with an LLM. It keeps suggesting stuff that doesn't work, and I need to keep resetting the context.

I am often able to go in, and see what the issue is, but that's almost worthless. The most productive thing that I can do, is tell the LLM what is the problem, on the output end, and ask it to review and fix. I can highlight possible causes, but it often finds corner cases that I miss. I have to be careful not to be too dictatorial.

It's frustrating, as the LLM is like a junior programmer, but I can make suggestions that radically improve the result, and the total time is reduced drastically. I have gotten done, in about two hours, what might have taken all day.

Re: The threat is comfortable drift toward not understanding what you're doing

#390

Post by thepasch »

sumeno wrote: post #189
stavros wrote: post #49

Earlier quoted context omitted.

Seems that teaching Bob trivial things would be a simple solution to this predicament.

That's what the program he just took was supposed to be for, learning not output. You've just reinvented the article from first principles, congrats

> That’s what the program he just took was supposed to be for, learning not output.

If you send a kid to an elementary school, and they come back not having learned anything, do you blame the concept of elementary schools, or do you blame that particular school - perhaps a particular teacher _within_ that school?

Post reply on HN