Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

271–280 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#271

Post by AnimalMuppet »

sd9 wrote: post #64
QuantumNomad_ wrote: post #57

Earlier quoted context omitted.

> if Bob can do things with agents, he can do things I’ve been reminded lately of a conversation I had with a guy at hacker space cafe around ten years ago in Berlin. He had been working as a programmer for a significantly longer time than me. Long enough that for many years of his career, he had been programming in assembly. He was lamenting that these days, software was written in higher level languages, and that m…

Lovely story, thanks for sharing. I wonder how many assembly programmers got over it and retrained, versus moved on to do something totally different. I find the agentic way of working simultaneously more exhausting and less stimulating. I don’t know if that’s something I’m going to get over, or whether this is the end of the line for me.

I wasn't there at the time, but I believe that most assembly programmers learned higher-level languages.

My mother actually started programming in octal. I don't remember her exact words, but she said something to the effect that her life got so much better when she got an assembler. I suspect that going from assembly to compilers was much the same - you no longer had to worry about register allocations and building stack frames.

Re: The threat is comfortable drift toward not understanding what you're doing

#272

Post by torben-friis »

Almondsetat wrote: post #145
torben-friis wrote: post #61

Earlier quoted context omitted.

Can you run an industry level LLM at home? If not, you're changing learning to cook for Uber only meals. And since the alternative is starving, Uber will boil the pot. Don't give up your self sufficiency.

Can you build a computer at home? There is absolutely nothing self-sufficient about computer hardware

No, and that's the reason we're now paying twice what we paid a couple years ago. But I can write software at home.

We're already vulnerable to enshittification in so many areas, why increase the list? How does that work in my favor at all?

Re: The threat is comfortable drift toward not understanding what you're doing

#273

Post by gedy »

danielbln wrote: post #175
gedy wrote: post #103

Earlier quoted context omitted.

We aren't talking pocket calculators here (I see the irony of phone app in pocket), LLMs are hugely expensive things made and controlled behind costly commercial subscriptions. And likely in the middle of a huge investment bubble and stability is uncertain. So we all need to be careful about "gee we don't need that skill or person anymore", etc.

Open weight models that run under your desk are not frontier model level, but they are getting closer. Improvements in agentic post training and things like TurboQuant mean that even if all frontier labs pull the plug tomorrow, we will still have agents to work with.

I'm definitely looking forward to that, as I really want people to control their own tools.

Re: The threat is comfortable drift toward not understanding what you're doing

#274

Post by ipaddr »

FrojoS wrote: post #165

Every PhD program I'm aware of has a final hurdle known as the defence. You have to present your thesis while standing in front of a committee, and often the local community and public. They will asks questions and too many "I don't know" or false answers would make you fail. So, there is already a system in place that should stop Bob from graduating if he indeed learned much less than Alice. A similar argument can b…

In the future the llms can answer those questions for you by listening and feeding you answers into your headset.

The process you describe is a gate keeping exercise which will change to include llm judges at somepoint.

Re: The threat is comfortable drift toward not understanding what you're doing

#275

Post by [deleted] »

Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

[deleted]

Re: The threat is comfortable drift toward not understanding what you're doing

#276

Post by spacechild1 »

mchaver wrote: post #229
sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

> The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. Following the model of how startups have worked for the last 20 years or so, I expect agents to eventually be locked-down/nerfed/ad-infested for higher payments. We are enjoying the fruits of VC money at the moment and they are getting everyone addicted to agents. Eventually they need to turn a profit. Not sure how this p…

That's one of several reasons why I'm trying not to rely too much on LLMs. The prospect of only being able to code with a working internet connection and a subscription to some megacorp service is not particularly appealing to me.

Re: The threat is comfortable drift toward not understanding what you're doing

#277

Post by lukev »

threatofrain wrote: post #269
lukev wrote: post #250

Earlier quoted context omitted.

That's not what I mean. If I use a calculator to find a logarithm, and I know what a logarithm is, then the answer the calculator gives me is perfectly useful and 100% substitutable for what I would have found if I'd calculated the logarithm myself. If I use Claude to "build a login page", it will definitely build me a login page. But there's a very real chance that what it generated contains a security issue. If I'm…

Those two tasks are just very different. In one world you have provided a complete specification, such as 1 + 1, for which the calculator responds with some answer and both you and the machine have a decidable procedure for judging answers. In another world you have engaged in a declaration for which the are many right and wrong answers, and thus even the boundaries of error are in question. It's equivalent to asking…

Yes. That is the point I was making.

Calculators provide a deterministic solution to a well-defined task. LLMs don't.

Re: The threat is comfortable drift toward not understanding what you're doing

#278

Post by lxgr »

sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

> if Bob can do things with agents, he can do things.

This point is directly addressed in the paper: Bob will ultimately not be able to do the things Alice can, with or without agents, because he didn't build the necessary internal deep structure and understanding of the problem space.

And if Alice later on ends up being a better scientist (using agents!) than Bob will ever be, would you not say there was something lost to the world?

Learning needs a hill to climb, and somebody to actually climb it. Bob only learned how to press an elevator button.

Re: The threat is comfortable drift toward not understanding what you're doing

#279

Post by zozbot234 »

danielbln wrote: post #175
gedy wrote: post #103

Earlier quoted context omitted.

We aren't talking pocket calculators here (I see the irony of phone app in pocket), LLMs are hugely expensive things made and controlled behind costly commercial subscriptions. And likely in the middle of a huge investment bubble and stability is uncertain. So we all need to be careful about "gee we don't need that skill or person anymore", etc.

Open weight models that run under your desk are not frontier model level, but they are getting closer. Improvements in agentic post training and things like TurboQuant mean that even if all frontier labs pull the plug tomorrow, we will still have agents to work with.

TurboQuant is not a step change, it's more of a smaller incremental improvement to KV quantization, and possibly (unsure) to quantization more generally. I'm actually more positive about SSD weights offload, which opens up very large local models for slow inference (good enough for slow chat) to virtually any hardware or amount of RAM.

Re: The threat is comfortable drift toward not understanding what you're doing

#280

Post by ahussain »

> When his supervisor sent him a paper to read, Bob asked the agent to summarize it. When he needed to understand a new statistical method, he asked the agent to explain it. When his Python code broke, the agent debugged it. When the agent's fix introduced a new bug, it debugged that too. When it came time to write the paper, the agent wrote it. Bob's weekly updates to his supervisor were indistinguishable from Alice's.

In my experience, doing these things with the right intentions can actually improve understanding faster than not using them. When studying physics I would sometimes get stuck on small details - e.g. what algebraic rule was used to get from Eq 2.1 to 2.2? what happens if this was d^2 instead of d^3 etc. Textbooks don't have space to answer all these small questions, but LLMs can, and help the student continue making progress.

Also, it seems hard to imagine that Alice and Bob's weekly updates would be indistinguishable if Bob didn't actually understand what he was working on.

Post reply on HN