Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beliefs that are true for regular software but false when applied to AI

boydkane.com

Post reply on HN
Print view View original

401–410 of 461 posts

Re: Beliefs that are true for regular software but false when applied to AI

#401

Post by lazide »

rldjbpin wrote: post #400
lazide wrote: post #333

Earlier quoted context omitted.

Personally, I’ve never seen an AI flow of any kind that meets what would meet the quality of a typical ‘corporate’ acceptable flow. As in, reliably works, doesn’t go crazy randomly, etc. I’ve seen a lot of things that look like they’re working for a demo, but shortly after starting to use it? Trash. Not every time (and it’s getting a little better), but often enough that personally I’ve found them a net drain on prod…

> Personally, I find apples hesitation here a breath of fresh air i does not appear to me as hesitation but rather an example of how they were unable to recently deliver on their marketing promises. calling a suite of incomplete features as "Apple Intelligence" means that they had much higher expectations internally, similar to how they refined as second-movers in other instances. they have a similar situation with X…

Everyone else just ships anyway?

Re: Beliefs that are true for regular software but false when applied to AI

#402

Post by saghm »

adastra22 wrote: post #51
criddell wrote: post #21

Earlier quoted context omitted.

Right away my mind went to "well, are people more reliable than they used to be?" and I'm not sure they are. Of course LLMs aren't people, but an AGI might behave like a person.

Older people are generally more reliable than younger people.

On the other hand, within 120 years (and sometimes decades earlier), every human so far also has reached reach a point where they can't do anything anymore, reliably or otherwise. I don't think humans provide much of a precedent of sustained growth in reliable over anything other than a fixed period, and it's not clear to me that we have any idea whether or not we'll reach a ceiling in how reliable we can make LLMs before they outperform humans in tasks like programming.

Re: Beliefs that are true for regular software but false when applied to AI

#403

Post by wrs »

ako wrote: post #399
wrs wrote: post #397

Earlier quoted context omitted.

Being able to make it and knowing how it works are just not the same thing. I can make bread of consistent quality, and know how to improve certain aspects of it. To know how it works , I’d need to get a doctorate in biochemistry and then still have a fairly patchy understanding. I know plenty of people who can drive a car very successfully but would never claim they know how it works.

But with LLMs is there really more to understand? They’re just large functions that take numerical input and transform it into numerical output based on trained weights. There is nothing behind the scenes doing things we don’t understand. The magic is in the weights, and we know how to create these based on training data. Regarding the car, if you know how to build a car, you understand how a car works. A driver is m…

I’ve never seen the word “just” take on more load than it did in that second sentence!

Sorry to inform everybody doing their Ph.D. on LLM interpretability that they’re just wasting their time.

Re: Beliefs that are true for regular software but false when applied to AI

#404

Post by codebra »

freetime2 wrote: post #122

For a real world example of the challenges of harnessing LLMs, look at Apple. Over a year ago they had a big product launch focused on "Apple Intelligence" that was supposed to make heavy use of LLMs for agentic workflows. But all we've really gotten since then are a couple of minor tools for making emojis, summarizing notifications, and proof reading. And they even had to roll back the notification summaries for a w…

Apple’s experience has almost nothing to do with “harnessing” LLMs, and everything to do with their wildly misjudged assumption they could run a viable model on a phone. Useful LLMs require their own power plants and can only be feasibly run in the cloud, or in a limited manner on powerful equipment like a 5090. Apple seems to have misunderstood that the “large” in large language model isn’t just a metaphor.

Re: Beliefs that are true for regular software but false when applied to AI

#405

Post by beyarkay »

AdieuToLogic wrote: post #162

I found this statement particularly relevant: While it’s possible to demonstrate the safety of an AI for a specific test suite or a known threat, it’s impossible for AI creators to definitively say their AI will never act maliciously or dangerously for any prompt it could be given. This possibility is compounded exponentially when MCP[0] is used. 0 - https://github.com/modelcontextprotocol

Thanks! I'm very interested in mechanistic intepretability, specifically Anthropic and Neel Nanda's work, so this impossibility of proving safety is a core concept for me.

Re: Beliefs that are true for regular software but false when applied to AI

#406

Post by beyarkay »

jvanderbot wrote: post #319
nedt wrote: post #316

Earlier quoted context omitted.

Yeah in that regard we should always treat it like a junior something. Very much like you can't expect your own kids to never do something dangerous even if tell it for years to be careful. I got used to getting my kid from the Kindergarten with a new injury at least once a month.

We should move well beyond human analogies. I have never met a human that would straight up lie about something, or build up so much deceptive tests that it might as well be lying. Granted this is not super common in these tools, but it is essentially unheard of in junior devs.

> Granted this is not super common in these tools, but it is essentially unheard of in junior devs.

I wonder if it's unheard of in junior devs because they're all saints, or because they're not talented enough to get away with it?

Re: Beliefs that are true for regular software but false when applied to AI

#407

Post by beyarkay »

erichocean wrote: post #330
AdieuToLogic wrote: post #162

I found this statement particularly relevant: While it’s possible to demonstrate the safety of an AI for a specific test suite or a known threat, it’s impossible for AI creators to definitively say their AI will never act maliciously or dangerously for any prompt it could be given. This possibility is compounded exponentially when MCP[0] is used. 0 - https://github.com/modelcontextprotocol

> it’s impossible for AI creators to definitively say their AI will never act maliciously or dangerously for any prompt it could be given This is false, AI doesn't "act" at all unless you, the developer, use it for actions. In which case it is you, the developer, taking the action. Anthropomorphizing AI with terms like "malicious" when they can literally be implemented with a spreadsheet—first-order functional progra…

> AI doesn't "act" at all unless you, the developer, use it for actions

This seems like a pointless definition of "act"? someone else could use the AI for actions which affect me, in which case I'm very much worried about those actions being dangerous, regardless of precisely how you're defining the word "act".

> when they can literally be implemented with a spreadsheet

The financial system that led to 2008 basically was one big spreadsheet, and yet it would have been correct to be worried about it. "Malicious" maybe is a bit evocative, I'll grant you that, but if I'm about to be eaten by a lion, I'm less concerned about not mistakenly athropomorphizing the lion, and more about ensuring I don't get eaten. It _doesn't matter_ whether the AI has agency or is just a big spreadsheet or wants to do us harm or is just sitting there. If it can do harm, it's dangerous.

Re: Beliefs that are true for regular software but false when applied to AI

#408

Post by beyarkay »

derf_ wrote: post #335

There are four main levers for improving an ML system: 1. You can change the training data. 2. You can change the objective function. 3. You can change the network topology. 4. You can change various hyperparameters (learning rate, etc.). From there, I think it is better to look at the process as one of scientific discovery rather than a software debugging task. You form hypotheses and you try to work out how test th…

Just want to say that you're the only person I've read who's come up with "ways to improve ML system" that I've agreed with. Thank you.

Re: Beliefs that are true for regular software but false when applied to AI

#409

Post by beyarkay »

themanmaran wrote: post #18

> Because eventually we’ll iron out all the bugs so the AIs will get more reliable over time Honestly this feels like a true statement to me. It's obviously a new technology, but so much of the "non-deterministic === unusable" HN sentiment seems to ignore the last two years where LLMs have become 10x as reliable as the initial models.

I'll totally grant that thing will get better over time. But a point that I was (mostly failing) to make, is that software is discrete, NNs are continuous. No matter how buggy your program is, it's got a countable number of lines, so you can have some notion of "all the bugs" (ignoring distributed systems).

But NNs are fundamentally continuous, I don't think it even makes sense to "count" bugs. You can have a list of prompts to which the model gives unwanted output, but it's a completely different ball game compared to regular software.

Re: Beliefs that are true for regular software but false when applied to AI

#410

Post by beyarkay »

lmc wrote: post #242

> With AI systems, almost all bad behaviour originates from the data that’s used to train them Careful with this - even with perfect data (and training), models will still get stuff wrong.

Indeed, this has been the most contentious line in the whole piece :D

How do you define "perfect" data and training? I'd argue that if you trained a small NN to play tic-tac-toe perfectly, it'd quickly memorise all the possible scenarios, and since the world state is small, you could exhaustively prove that it's correct for every possible input. So at the very least, there's a counter example showing that with perfect data and training, models will not get stuff wrong.

Post reply on HN