Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The threat is comfortable drift toward not understanding what you're doing

ergosphere.blog

Post reply on HN
Print view View original

521–530 of 668 posts

Re: The threat is comfortable drift toward not understanding what you're doing

#521

Post by bluefirebrand »

lukev wrote: post #277
threatofrain wrote: post #269

Earlier quoted context omitted.

Those two tasks are just very different. In one world you have provided a complete specification, such as 1 + 1, for which the calculator responds with some answer and both you and the machine have a decidable procedure for judging answers. In another world you have engaged in a declaration for which the are many right and wrong answers, and thus even the boundaries of error are in question. It's equivalent to asking…

Yes. That is the point I was making. Calculators provide a deterministic solution to a well-defined task. LLMs don't.

Furthermore, it is possible to build a precise mathematical formula to produce a desired solution

It is not possible to be nearly as precise when describing a desired solution to an LLM, because natural languages are simply not capable of that level of precision... Which is the entire reason coding languages exist in the first place

Re: The threat is comfortable drift toward not understanding what you're doing

#522

Post by Wowfunhappy »

pxc wrote: post #453
Wowfunhappy wrote: post #329

Earlier quoted context omitted.

Then the social paradigm needs to change. Is everyone just going to roll over and die while AI destroys academia (and possibly a lot more)? Last September, Tyler Austin Harper published a piece for The Atlantic on how he thinks colleges should respond to AI. What he proposes is radical —but, if you've concluded that AI really is going to destroy everything these institutions stand for, I think you have to at least co…

I was pretty interested until I got to this part: > Another reason that a no-exceptions policy is important: If students with disabilities are permitted to use laptops and AI, a significant percentage of other students will most likely find a way to get the same allowances, rendering the ban useless. I witnessed this time and again when I was a professor—students without disabilities finding ways to use disability ac…

You don't have to agree with his precise solution, and in fact I'm not sure whether I do. However, I found the article useful because it got me thinking about the universe of things we could be considering, if we really do think AI is poised to destroy education as we know it.

Re: The threat is comfortable drift toward not understanding what you're doing

#523

Post by defrost »

Kon5ole wrote: post #143
defrost wrote: post #17

Earlier quoted context omitted.

> This argument boils down to "don't use tools because you'll forget how to do things the hard way", which nobody would buy for any other tool, This is false. There absolutely are people that fall back on older tools when fancy tools fail. You will find such people in the military, in emergency services, in agriculture, generally in areas where getting the job done matters. Perhaps you're unfamiliar. They other week…

>This is false. There absolutely are people that fall back on older tools when fancy tools fail. >They other week I finished putting holes in fence posts with a bit and brace as there was no fuel for the generator to run corded electric drills and the rechargable batteries were dead. It depends on the task though. If you are in a similar scenario as with your fence posts and want to edit computer programs, you can't.…

> If you are in a similar scenario as with your fence posts and want to edit computer programs, you can't.

Solar panels charging old thinkpads suddenly doesn't work, or are we reliant upon software that requires cloud services to function in your scenario?

> How many people in the world can recreate a ASML lithography machine ..

Some .. are you advocating the answer should be none, in your future?

Re: The threat is comfortable drift toward not understanding what you're doing

#524

Post by xnzakg »

spacechild1 wrote: post #276
mchaver wrote: post #229

Earlier quoted context omitted.

> The thing is, agents aren’t going away. So if Bob can do things with agents, he can do things. Following the model of how startups have worked for the last 20 years or so, I expect agents to eventually be locked-down/nerfed/ad-infested for higher payments. We are enjoying the fruits of VC money at the moment and they are getting everyone addicted to agents. Eventually they need to turn a profit. Not sure how this p…

That's one of several reasons why I'm trying not to rely too much on LLMs. The prospect of only being able to code with a working internet connection and a subscription to some megacorp service is not particularly appealing to me.

Local/open LLMs are a thing though. You can build a server for hosting decent sized (100-200B) models at home for a few k$. They may not be Opus-level, but hopefully we can get something matching current SOTA, but that we can run locally, before the megacorps get too greedy.

Alternatively you could find some other people to share the HW cost and run some larger models (like Kimi-K2.5 at 1.1T params).

Re: The threat is comfortable drift toward not understanding what you're doing

#525

Post by xnzakg »

vinceguidry wrote: post #468
Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

I've been using ChatGPT to re-bootstrap my coding hobby. After the initial honeymoon wore off, I realized I was staring down the barrel of a dilemma. If I use AI to "just handle" the parts of the system I don't want to understand, I invariably end up in a situation where I gotta throw a whole bunch of work out. But I can't supervise without an understanding of what it's supposed to be doing, and if I knew what it was…

I suppose using an LLM-powered autocomplete might be something for you then?

Then you can be writing the code, with the LLM doing the "boring" parts, in chunks small enough you can review them on the fly.

Re: The threat is comfortable drift toward not understanding what you're doing

#527

Post by Wowfunhappy »

doug_durham wrote: post #462
Wowfunhappy wrote: post #227

> Schwartz's experiment is the most revealing, and not for the reason he thinks. What he demonstrated is that Claude can, with detailed supervision, produce a technically rigorous physics paper. What he actually demonstrated, if you read carefully, is that the supervision is the physics. Claude produced a complete first draft in three days. It looked professional. The equations seemed right. The plots matched expecta…

The article is a thought experiment. The author hypothesizes that Bob isn't getting the same benefit that Alice is getting. That hypothesis could be wrong. I don't know and the author doesn't know. It could be that Bob is going to have a very successful career and will deeply know the field because he is able to traverse a wider set of problems more quickly. At this point, it's just hypothesis. I don't think that we…

Do you think you learn as much from reading summaries of papers as reading the full thing? Do you think you learn as much from asking a friend to write a paper as when you write it yourself?

Re: The threat is comfortable drift toward not understanding what you're doing

#528

Post by suzzer99 »

jnovek wrote: post #249
wizzwizz4 wrote: post #88

Earlier quoted context omitted.

From the article: > There's a common rebuttal to this, and I hear it constantly. "Just wait," people say. "In a few months, in a year, the models will be better. They won't hallucinate. They won't fake plots. The problems you're describing are temporary." I've been hearing "just wait" since 2023. We're not trending towards superintelligence with these AIs. We're trending towards (and, in fact, have already reached) s…

The rate of hallucination has gone down drastically since 2023. As LLM coding tools continue to pare that rate down, eventually we’ll hit a point where it is comparable to the rate we naturally introduce bugs as humans programmers.

I wonder how much of the decrease in hallucination is because the models are getting better, and how much is because these massively over-funded companies are adding a bunch of one-off shims at breakneck speed. IE - are they truly improving the cognition, or just monkey-patching the hell out of it?

The recent article where the AI companies are paying experts in the field to help train the models makes me wonder if they're also manually fixing a bunch of post-processing errors as they come up.

Re: The threat is comfortable drift toward not understanding what you're doing

#529

Post by suzzer99 »

b112 wrote: post #44
djaro wrote: post #14

Earlier quoted context omitted.

> So if Bob can do things with agents, he can do things. The problem arrises when Bob encounters a problem too complex or unique for agents to solve. To me, it seems a bit like the difference between learning how to cook versus buying microwave dinners. Sure, a good microwave dinner can taste really good, and it will be a lot better than what a beginning cook will make. But imagine aspiring cooks just buying premade…

Worse, soon fewer and fewer people will taste good food, including even higher and higher scale restaurants just using pre-made. As fewer know what good food tastes like, the entire market will enshitify towards lower and lower calibre food. We already see this with, for example, fruits in cold climates. I've known people who have only ever bought them from the supermarket, then tried them at a farmers when they're i…

> As fewer know what good food tastes like, the entire market will enshitify towards lower and lower calibre food.

This happened a long time ago in the US. Drive through California's Central Valley sometime and sample the fruit sold fresh along the side of the road. It's a completely different experience than the version you get at Safeway.

Re: The threat is comfortable drift toward not understanding what you're doing

#530

Post by rglullis »

thepasch wrote: post #408
rglullis wrote: post #41

Earlier quoted context omitted.

> so they can produce other, more useful, results But to even *know* what is more useful, it is crucial to have walked the walk. Otherwise we will all end up with a bunch of people trying to reinvent the wheel, over and over again, like JavaScript "developers" who keep reinventing frameworks every six months. > which nobody would buy for any other tool I don't know about you, but I wasn't allowed to use calculators i…

> But to even know what is more useful, it is crucial to have walked the walk. I feel like people tend to forget that among the many things LLMs can do these days, “using a search engine” is among them. In fact, they use them better than the majority of people do! The conversation people think they’re having here and the conversation that actually needs to be had are two entirely different conversations. > I don’t kn…

> I’ve now ostensibly understood what a derivative does and what it’s used for, yet I have zero idea how to mathematically do it. Does that make any results I gain from this intuitive understanding any less valuable?

From a science standpoint, I'd say whatever "results" you got are completely worthless.

> I’ll generally have some sort of hypothesis of what kind of result I’m expecting, given that my understanding is correct

And how do you know if your understanding is correct, if you are only taking what the LLM gives to you and you are not able to verify independently?

> Science is what happens when you expect something, test something, and get a result.

Right, but has any LLM come up with any hypothesis on its own? Has any AI said "given all this literature that I read, I'd expect ?".

Post reply on HN