Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

51–60 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#51

I think this article is largely, or at least directionally, correct. I'd draw a comparison to high-level languages and language frameworks. Yes, 99% of the time, if I'm building a web frontend, I can live in React world and not think about anything that is going on under the hood. But, there is 1% of the time where something goes wrong, and I need to understand what is happening underneath the abstraction. Similarly,…

So we already have this problem and things are "fine"?

Re: The threat is comfortable drift toward not understanding what you're doing

#52

I think this article is largely, or at least directionally, correct. I'd draw a comparison to high-level languages and language frameworks. Yes, 99% of the time, if I'm building a web frontend, I can live in React world and not think about anything that is going on under the hood. But, there is 1% of the time where something goes wrong, and I need to understand what is happening underneath the abstraction. Similarly,…

> LLMs will get _so_ good that I don't feel the need to do this, in the same way that I don't think about the transistors my code is ultimately running on.

The problem is, they're nothing like transistors, and never will be. Those are simple. Work or don't, consistently, in an obvious, or easily testable, way.

LLM are more akin to biological things. Complex. Not well understood. Unpredictable behavior. To be safely useful, they need something like a lion tamer, except every individual LLM is its own unique species.

I like working on computers because it minimizes the amount of biological-like things I have to work with.

Re: The threat is comfortable drift toward not understanding what you're doing

#53
post #40
post #14

Earlier quoted context omitted.

> So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade…

That doesn't sound like much of an issue. Bob was already going to encounter problems that are too large and complex for him to solve, agents or otherwise. Life throws us hard problems. I don't recall if we even assumed Bob was unusually capable, he might be one of life's flounderers. I'd give good odds that if he got through a program with the help of agents he'll get through life achieving at least a normal level o…

"Things that have never been done before in software" has been my entire career. A lot of it requires specific knowledge of physics, modelling, computer science, and the tradeoffs involved in parsimony and efficiency vs accuracy and fidelity.

Do you have a solution for me? How does the market value things that don't yet exist in this brave new world?

Re: The threat is comfortable drift toward not understanding what you're doing

#54
> Frank Herbert (yeah, I know I'm a nerd), in God Emperor of Dune, has a character observe: "What do such machines really do? They increase the number of things we can do without thinking. Things we do without thinking; there's the real danger." Herbert was writing science fiction. I'm writing about my office. The distance between those two things has gotten uncomfortably small.

The author is a bit naive here:

1. Society only progresses when people are specialised and can delegate their thinking

2. Specialisation has been happening for millenia. Agriculture allowed people to become specialised due to abundance of food

3. We accept delegation of thinking in every part of life. A manager delegates thinking to their subordinates. I delegate some thinking to my accountant

4. People will eventually get the hang of using AI to do the optimum amount of delegation such that they still retain what is necessary and delegate what is not necessary. People who don't do this optimally will get outcompeted

The author just focuses on some local problems like skill atrophy but does not see the larger picture and how specific pattern has been repeating a lot in humanity's history.

Re: The threat is comfortable drift toward not understanding what you're doing

#55
post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

>The thing is, agents aren’t going away...

Aren't they currently propped up by investor money?

What happens when the investors realize the scam that it is and stop investing or start investing less...

Re: The threat is comfortable drift toward not understanding what you're doing

#57
post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

> if Bob can do things with agents, he can do things

I’ve been reminded lately of a conversation I had with a guy at hacker space cafe around ten years ago in Berlin.

He had been working as a programmer for a significantly longer time than me. Long enough that for many years of his career, he had been programming in assembly.

He was lamenting that these days, software was written in higher level languages, and that more and more programmers no longer had the same level of knowledge about the lower level workings of computers. He had a valid point and I enjoyed talking to him.

I think about this now when I think about agentic coding. Perhaps over time most software development will be done without the knowledge of the higher level programming languages that we know today. There will still be people around that work in the higher level programming languages in the future, and are intimately familiar with the higher level languages just like today there are still people who work in assembly even if the percentage of people has gotten lower over time relative to those that don’t.

And just like there are areas where assembly is still required knowledge, I think there will be areas where knowledge of the programming languages we use today will remain necessary and vibe coding alone wont cut it. But the percentage of people working in high level languages will go down, relative to the number of people vibe coding and never even looking at the code that the LLM is writing.

Re: The threat is comfortable drift toward not understanding what you're doing

#58
The exciting and interesting to me is that we'll probably need to engage "chaos engineering" principles, and encode intentional fallibility into these agents to keep us (and them) as good collaborators, and specifically on our toes, to help all minds stay alert and plastic

If that comes to pass, we'll be rediscovering the same principles that biological evolution stumbled upon: the benefits of the imperfect "branch" or "successive limited comparison" approach of agentic behaviour, which perhaps favours heuristics (that clearly sometimes fail), interaction between imperfect collaborators with non-overlapping biases, etc etc

https://contraptions.venkateshrao.com/p/massed-muddler-intel...

> Lindblom’s paper identifies two patterns of agentic behavior, “root” (or rational-comprehensive) and “branch” (or successive limited comparisons), and argues that in complicated messy circumstances requiring coordinated action at scale, the way actually effective humans operate is the branch method, which looks like “muddling through” but gradually gets there, where the root method fails entirely.

Re: The threat is comfortable drift toward not understanding what you're doing

#59
post #47
post #46

Earlier quoted context omitted.

My point is it doesn’t matter whether he can fix things without agents. The real world isn’t an exam hall where your boss tells you “no naughty AI!”, you just get stuff done, and if Bob can do that with agents, nobody cares how he did it.

> The real world isn’t an exam hall where your boss tells you “no naughty AI!”, you just get stuff done, and if Bob can do that with agents, nobody cares. Indeed. That's why Anthropic had to hire real engineers to make sure their vibe-coded shit doesn't consume 68GB of RAM. Because real world: https://x.com/jarredsumner/status/2026497606575398987

If your job has been totally unaffected by AI, then I am jealous.

I’m not trying to argue that AI can do everything today. I acknowledge that there are many things that it is not good at.

Re: The threat is comfortable drift toward not understanding what you're doing

#60
post #3

I see this fallacy being committed a lot these days. "Because LLMs, you will no longer need a skill you don't need any more, but which you used to need, and handwaves that's bad". Academia doesn't want to produce astrophysics (or any field) scientists just so the people who became scientists can feel warm and fuzzy inside when looking at the stars, it wants to produce scientists who can produce useful results. Bob pr…

You missed the argument. When we are talking about faculty, yes their result is the only thing that matters, so if it was produced quicker with a LLM, that's great. But when we are talking about the student, there is a drastic difference in the student in the with LLM vs without LLM cases. In the latter they have much better understanding. And that matters in the system when we are educating future physicists.
Post reply on HN