Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

241–250 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#241

Post by pbw »

There's certainly a risk that an individual will rely too much on AI, to the detriment of their ability to understand things. However, I think there are obvious counter-measures. For example, requiring that the student can explain every single intermediate step and every single figure in detail.

A two-hour thesis defense isn't enough to uncover this, but a 40-hour deep probing examination by an AI might be. And the thesis committee gets a "highlight reel" of all the places the student fell short.

The general pattern is: "Suppose we change nothing but add extensive use of AI, look how everything falls apart." When in reality, science and education are complex adaptive systems that will change as much as needed to absorb the impact of AI.

Re: The threat is comfortable drift toward not understanding what you're doing

#242

Post by steveBK123 »

maplethorpe wrote: post #226
steveBK123 wrote: post #211

For the people arguing that the output is the code and the faster we generate it the better.. I do wonder where all the novel products produced by 10x devs who are now 100x with LLMs, the “idea guys” who can now produce products from whole clothe without having to hire pesky engineers.. where is the one-man 10 billion dollar startups, etc? We are 3-4 years into this mania and all I see on the other end of it is the L…

I'm waiting for Anthropic to realise they can just set a few thousand agents loose to do just that, and monopolize the entire software market overnight. I'm not sure why they haven't done this yet.

Because a lot of valuable software is the implicit / organizational / human domain knowledge .. not the trillions of lines of code LLms all scraped and trained on.

Re: The threat is comfortable drift toward not understanding what you're doing

#243

Post by jappgar »

Almondsetat wrote: post #145
torben-friis wrote: post #61

Earlier quoted context omitted.

Can you run an industry level LLM at home? If not, you're changing learning to cook for Uber only meals. And since the alternative is starving, Uber will boil the pot. Don't give up your self sufficiency.

Can you build a computer at home? There is absolutely nothing self-sufficient about computer hardware

Or generate electricity? Or grow enough food to survive? Medicines?

"Self-sufficiency" arguments coming from tech nerds are so tiring.

Re: The threat is comfortable drift toward not understanding what you're doing

#244

Post by ndriscoll »

bigfishrunning wrote: post #200
CuriouslyC wrote: post #115

Earlier quoted context omitted.

Just because Bob doesn't know e.g. Rust syntax and library modules well, doesn't mean that Bob can't learn an algorithm to solve a difficult problem. The AI might suggest classes of algorithms that could be applicable given the real world constraints, and help Bob set up an experimental plan to test different algorithms for efficacy in the situation, but Bob's intuition is still in the drivers's seat. Of course, that…

But if Bob doesn't know rust syntax and library modules well, how can he be expected to evaluate the output generating Rust code? Bugs can be very subtle and not obvious, and Rust has some constructs that are very uncommon (or don't exist) in other languages. Human nature says that Bob will skim over and trust the parts that he doesn't understand as long as he gets output that looks like he expects it to look, and th…

Then perhaps Bob should have it use functional Scala, where my experience is that if it compiles and looks like what you expect, it's almost certainly correct.

Re: The threat is comfortable drift toward not understanding what you're doing

#245

Post by simianwords »

lapcat wrote: post #220
simianwords wrote: post #177

Earlier quoted context omitted.

I can't tell what specialists we will get the same way you wouldn't be able to tell me we will have Linux Kernel specialists at the year 1945. People do more things with AI. More things = more inventions = the field growing. The field grows and people become specialists on what used to be a small or trivial. A mathematician in 1500's wouldn't think algebraic topology would be a specialisation.

> I can't tell what specialists we will get the same way you wouldn't be able to tell me we will have Linux Kernel specialists at the year 1945. How about addressing astrophysics specifically. What are you claiming about it? Are you claiming that in the future, we won't need astrophysicists at all, AI can do all of our astrophysics for us, freeing humans to specialize in... other subjects? And doesn't the same proble…

> The specific subjects have changed over time, but the production of specialist mathematicians hasn't really changed. It takes hard work, grunt work, struggling, making mistakes and learning from them, as well as expert supervision. The problem with AI is that it encourages and incentivizes intellectual laziness, the opposite of what is required to produce specialists

Let's take the example of economics. Economists use ideas in Mathematics like integrals, statistics, PDE's and so on. They know that these concepts exist. They know how to apply them. They don't know these concepts deep enough to make progress here.

Do you think that Economists should deeply learn integrals, PDE's, Functional Analysis and Differential Geometry and all other concepts they use? Or do you think its better for them to focus just on their specific domain while learning just enough from other domains?

You keep coming back to AI replacing mathematicians. I'm not making that claim. I'm not saying Linux kernel specialists will be replaced by AI. I'm simply claiming that not everyone needs to be Linux Kernel specialists. This is precisely what AI is allowing: it automates things I don't need to know deeply so that I can focus on things I do need to understand deeply.

Re: The threat is comfortable drift toward not understanding what you're doing

#246

Post by Jensson »

ipaddr wrote: post #225
sgarland wrote: post #108

Earlier quoted context omitted.

I would love to see someone attempt to do multiplication who never learned addition, or exponentiation without having learned multiplication. There is a vast difference between “never learned the skill,” and “forgot the skill from lack of use.” I learned how to do long division in school, decades ago. I sat down and tried it last year, and found myself struggling, because I hadn’t needed to do it in such a long time.

Most people learn multiplication by memorizing a series of cards 2x2,2x3.. 9x9. Later this gets broken down to addition in higher grades.

Most people learn multiplication by counting, it has been in basic mathbooks since forever. "1 box has 4 cookies. Jenny ha 4 boxes of cookies. How many cookies do Jenny have?" and so on, the kids solve that by counting 4 cookies in every of the 4 boxes and reaching 16. Only later do you learn those tables.

Re: The threat is comfortable drift toward not understanding what you're doing

#247

Post by coldtea »

sd9 wrote: post #6

The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. I mourn the loss of working on intellectually stimulating programming problems, but that’s a part of my job that’s fading. I need to decide if the remaining work - understanding requirements, managing teams, what have you - is still enjoyable enough to continue. To be honest, I’m looking at leaving software because the job…

>The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things.

He'll get things (papers, code, etc) which he can't evaluate. And the next round of agents will be trained on the slop produced by the previous ones. Both successive Bob's and successive agents will have less understanding.

Re: The threat is comfortable drift toward not understanding what you're doing

#248

Post by bluedino »

Look at how bad the auto industry has gotten when it comes to quality and recalls.

A combination of beancounters running the show and the old, experienced engineers dying, retiring, and going through buyouts has pretty much left things in a pretty sad state.

Re: The threat is comfortable drift toward not understanding what you're doing

#249

Post by jnovek »

wizzwizz4 wrote: post #88
roenxi wrote: post #40

Earlier quoted context omitted.

That doesn't sound like much of an issue. Bob was already going to encounter problems that are too large and complex for him to solve, agents or otherwise. Life throws us hard problems. I don't recall if we even assumed Bob was unusually capable, he might be one of life's flounderers. I'd give good odds that if he got through a program with the help of agents he'll get through life achieving at least a normal level o…

From the article: > There's a common rebuttal to this, and I hear it constantly. "Just wait," people say. "In a few months, in a year, the models will be better. They won't hallucinate. They won't fake plots. The problems you're describing are temporary." I've been hearing "just wait" since 2023. We're not trending towards superintelligence with these AIs. We're trending towards (and, in fact, have already reached) s…

The rate of hallucination has gone down drastically since 2023. As LLM coding tools continue to pare that rate down, eventually we’ll hit a point where it is comparable to the rate we naturally introduce bugs as humans programmers.

Re: The threat is comfortable drift toward not understanding what you're doing

#250

Post by lukev »

yunwal wrote: post #230
lukev wrote: post #179

Earlier quoted context omitted.

I think that's too easy an analogy, though. Calculators are deterministically correct given the right input. It does not require expert judgement on whether an answer they gave is reasonable or not. As someone who uses LLMs all day for coding, and who regularly bumps against the boundaries of what they're capable of, that's very much not the case. The only reason I can use them effectively is because I know what good…

Determinism just means you don't have to use statistics to approach the right answer. It's not some silver bullet that magically makes things understandable and it's not true that if it's missing from a system you can't possibly understand it.

That's not what I mean.

If I use a calculator to find a logarithm, and I know what a logarithm is, then the answer the calculator gives me is perfectly useful and 100% substitutable for what I would have found if I'd calculated the logarithm myself.

If I use Claude to "build a login page", it will definitely build me a login page. But there's a very real chance that what it generated contains a security issue. If I'm an experienced engineer I can take a quick look and validate whether it does or whether it doesn't, but if I'm not, I've introduced real risk to my application.

Post reply on HN