Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beliefs that are true for regular software but false when applied to AI

boydkane.com

Post reply on HN
Print view View original

381–390 of 461 posts

Re: Beliefs that are true for regular software but false when applied to AI

#381

Post by alecbz »

ben_w wrote: post #127
alecbz wrote: post #87

Earlier quoted context omitted.

Regularly trying to use LLMs to debug coding issues has convinced me that we're _nowhere_ close to the kind of AGI some are imagining is right around the corner.

Sure, but also the METR study showed the rate of change is t doubles every 7 months where t ~= «duration of human time needed to complete a task, such that SOTA AI can complete same with 50% success»: https://arxiv.org/pdf/2503.14499 I don't know how long that exponential will continue for, and I have my suspicions that it stops before week-long tasks, but that's the trend-line we're on.

Only skimmed the paper, but I'm not sure how to think about "length of task" as a metric here.

The cases I'm thinking about are things that could be solved in a few minutes by someone who knows what the issue is and how to use the tools involved. I spent around two days trying to debug one recent issue. A coworker who was a bit more familiar with the library involved figured it out in an hour or two. But in parallel with that, we also asked the library's author, who immediately identified the issue.

I'm not sure how to fit a problem like that into this "duration of human time needed to complete a task" framework.

Re: Beliefs that are true for regular software but false when applied to AI

#382

Post by krackers »

zitterbewegung wrote: post #142
freetime2 wrote: post #122

For a real world example of the challenges of harnessing LLMs, look at Apple. Over a year ago they had a big product launch focused on "Apple Intelligence" that was supposed to make heavy use of LLMs for agentic workflows. But all we've really gotten since then are a couple of minor tools for making emojis, summarizing notifications, and proof reading. And they even had to roll back the notification summaries for a w…

Now their strategy is to allow for Apple Events to work with the MCP. https://9to5mac.com/2025/09/22/macos-tahoe-26-1-beta-1-mcp-i...

Article says app intents, not apple events. Apple Events would be the natural thing but it's an abandoned ecosystem that would require them to walk back the past decade so of course they won't do that.

Re: Beliefs that are true for regular software but false when applied to AI

#383

Post by vel0city »

ponector wrote: post #314
vel0city wrote: post #214

Earlier quoted context omitted.

If you have faulty hardware no amount of software is going to solve your problems (other than software that just completely deactivates said faulty hardware). The fact you continued to have BSOD issues after a full reinstall is pretty strong evidence you probably had some kind of hardware failure.

But there was no reinstall in my case. Years goes by and further in time there are less and less bsod. My point is if you are using the same "old" modern hardware, bsod is very rare.

Ah, sorry, I misread your comment. Glad you're getting a better experience with your device over time!

Re: Beliefs that are true for regular software but false when applied to AI

#384

Post by pipes »

beyarkay wrote: post #361
lazide wrote: post #333

Earlier quoted context omitted.

Personally, I’ve never seen an AI flow of any kind that meets what would meet the quality of a typical ‘corporate’ acceptable flow. As in, reliably works, doesn’t go crazy randomly, etc. I’ve seen a lot of things that look like they’re working for a demo, but shortly after starting to use it? Trash. Not every time (and it’s getting a little better), but often enough that personally I’ve found them a net drain on prod…

The regular ChatGPT 5 seems pretty reliable to me? I ~never get crazy output unless I'm pasting a jailbreak prompt I saw on twitter. It might not always meet my standards, but that's true of a lot of things.

Maybe not the same thing, but chatgpt 5 was driving me insane in visual studio co pilot last week. I seemingly could stop it from randomly changing bits of code, to the point where it was apologising then doing the same in next change even when told not to.

I've now changed to asking where things are in the code base and how they work then making changes myself.

Re: Beliefs that are true for regular software but false when applied to AI

#385

Post by mrkmarron »

AdieuToLogic wrote: post #211
mrkmarron wrote: post #180

Earlier quoted context omitted.

[flagged]

> The goal is to build a language and system model that allows us to reliably sandbox and support agents in constructing "Trustworthy-by-Construction AI Agents." 1 - Reliability implies predictable behavior. 2 - Predictable behavior implies determinism. 3 - LLM's are non-deterministic algorithms. In the link you kindly provided are phrases such as, "increases the likelihood of successful correct use" and "structure f…

The key capability that Bosque has for API specs is the ability to provide pre/post conditions with arbitrary expressions. This is particularly useful once you can do temporal conditions involving other API calls (as discussed in the blog post and part of the 2.0 push).

Bosque also has a number of other niceties[0] -- like ReDOS free pattern regex checking, newtype support for primitives, support for more primitives than JSON (RAML) such as Char vs. Unicode strings, UUIDs, and ensures unambiguous (parsable) representations.

Also the spec and implementation are very much not proprietary. Everything is MIT licensed and is being developed in the open by our group at the U. of Kentucky.

[0] https://dl.acm.org/doi/pdf/10.1145/3689492.3690054

Re: Beliefs that are true for regular software but false when applied to AI

#386

Post by mannykannot »

erichocean wrote: post #330
AdieuToLogic wrote: post #162

I found this statement particularly relevant: While it’s possible to demonstrate the safety of an AI for a specific test suite or a known threat, it’s impossible for AI creators to definitively say their AI will never act maliciously or dangerously for any prompt it could be given. This possibility is compounded exponentially when MCP[0] is used. 0 - https://github.com/modelcontextprotocol

> it’s impossible for AI creators to definitively say their AI will never act maliciously or dangerously for any prompt it could be given This is false, AI doesn't "act" at all unless you, the developer, use it for actions. In which case it is you, the developer, taking the action. Anthropomorphizing AI with terms like "malicious" when they can literally be implemented with a spreadsheet—first-order functional progra…

You are right about 'malicious'. 'Dangerous', however, is a different matter.

Re: Beliefs that are true for regular software but false when applied to AI

#387

Post by wrs »

ako wrote: post #377
wrs wrote: post #36

My current method for trying to break through this misconception is informing people that nobody knows how AI works. Literally. Nobody knows. (Note that knowing how to make something is not the same as knowing how it works. Take humans as an obvious example.)

We don’t know how to build humans from scratch (other than letting nature do it) so that’s not really a relevant example. We only know how to fix certain aspects of defect humans, and how to minimize the chance of future defects, but that’s far from building a human from scratch.

We don't know how to build LLMs from scratch either, in that sense. We know how to build a machine that you run trillions of tokens through and hope something good happens. What happens, we only know at a vague and low level. You just end up with a bag of hundreds of billions of floating point numbers with some amazing emergent behaviors.

Similarly, two adult humans know what to do to start the process that makes another human, and we know a few of the very low-level details about what happens, but that is a far cry from knowing how adult humans do what they do.

Re: Beliefs that are true for regular software but false when applied to AI

#388

Post by leogout »

I'm a bit troubled with the phrasing

> most AI companies will slightly change the way their AIs respond, so that they say slightly different things to the same prompt. This helps their AIs seem less robotic and more natural.

To my understanding this is managed by the temperature of the next token prediction which is picked more or less randomly based on this value. This temperature plays a role in the variability of the output.

I wasn't under the impression that it was to give the user a feeling of "realism", but rather that it produced better results with a slightly random prediction.

Re: Beliefs that are true for regular software but false when applied to AI

#389

Post by conception »

alecbz wrote: post #381
ben_w wrote: post #127

Earlier quoted context omitted.

Sure, but also the METR study showed the rate of change is t doubles every 7 months where t ~= «duration of human time needed to complete a task, such that SOTA AI can complete same with 50% success»: https://arxiv.org/pdf/2503.14499 I don't know how long that exponential will continue for, and I have my suspicions that it stops before week-long tasks, but that's the trend-line we're on.

Only skimmed the paper, but I'm not sure how to think about "length of task" as a metric here. The cases I'm thinking about are things that could be solved in a few minutes by someone who knows what the issue is and how to use the tools involved. I spent around two days trying to debug one recent issue. A coworker who was a bit more familiar with the library involved figured it out in an hour or two. But in parallel…

This is an excellent example of human “context windows” though and it could be the llm could have solved the easy problem with better context engineering. Despite 1M token windows, things still start to get progressively worse after 100k. LLMs would overnight be amazingly better with a reliable 1M window.

Re: Beliefs that are true for regular software but false when applied to AI

#390

Post by ben_w »

alecbz wrote: post #381
ben_w wrote: post #127

Earlier quoted context omitted.

Sure, but also the METR study showed the rate of change is t doubles every 7 months where t ~= «duration of human time needed to complete a task, such that SOTA AI can complete same with 50% success»: https://arxiv.org/pdf/2503.14499 I don't know how long that exponential will continue for, and I have my suspicions that it stops before week-long tasks, but that's the trend-line we're on.

Only skimmed the paper, but I'm not sure how to think about "length of task" as a metric here. The cases I'm thinking about are things that could be solved in a few minutes by someone who knows what the issue is and how to use the tools involved. I spent around two days trying to debug one recent issue. A coworker who was a bit more familiar with the library involved figured it out in an hour or two. But in parallel…

Fair comment.

While I think they're trying to cover that by getting experts to solve problems, it is definitely the case that humans learn much faster than current ML approaches, so "expert in one specific library" != "expert in writing software".

Post reply on HN