Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beliefs that are true for regular software but false when applied to AI

boydkane.com

Post reply on HN
Print view View original

451–460 of 461 posts

Re: Beliefs that are true for regular software but false when applied to AI

#451

Post by surgical_fire »

beyarkay wrote: post #419
surgical_fire wrote: post #80

Earlier quoted context omitted.

"AI will take over the world". I hear that. Then I try to use AI for simple code task, writing unit tests for a class, very similar to other unit tests. If fails miserably. Forgets to add an annotation and enters in a death loop of bullshit code generation. Generates test classes that tests failed test classes that test failed test classes and so on. Fascinating to watch. I wonder how much CO2 it generated while fryi…

Given that AI couldn't even speak English 6 years ago, do you really think it's going to struggle with unit tests for the next 20 years? It's well worth looking at https://progress.openai.com/ , here's a snippet: > human: Are you actually conscious under anesthesia? > GPT-1 (2018): i did n't . " you 're awake . > GPT-3 (2021): There is no single answer to this question since anesthesia can be administered [...]

> Given that AI couldn't even speak English 6 years ago, do you really think it's going to struggle with unit tests for the next 20 years?

Yes.

LLM is a very interesting technology for machines to understand and generate natural language. It is a difficult problem that it sort of solves.

It does not understand things beyond that. Developing software is not simply a natural language problem.

Re: Beliefs that are true for regular software but false when applied to AI

#452

Post by 5- »

beyarkay wrote: post #438
5- wrote: post #92

Earlier quoted context omitted.

the author (either of the blog or its software) would do well to consult https://www.petefreitag.com/blog/rss-autodiscovery/

consider it consulted! Autodiscovery should work when I publish changes (in a few hours)

great, thanks!

Re: Beliefs that are true for regular software but false when applied to AI

#453

Post by alganet »

beyarkay wrote: post #440
alganet wrote: post #5

> here are some example ideas that are perfectly true when applied to regular software Hm, I'm listening, let's see. > Software vulnerabilities are caused by mistakes in the code That's not exactly true. In regular software, the code can be fine and you can still end up with vulnerabilities. The platform in which the code is deployed could be vulnerable, or the way it is installed make it vulnerable, and so on. > Bug…

> Ok, I'm starting to feel this is a troll post. This guy can't be serious. Did you read the footnote about writing regression tests to catch bugs before they come back in production?

Discussed here:

https://news.ycombinator.com/item?id=45583970

Thought I might just skip the repetition. You can continue the conversation within that thread.

Re: Beliefs that are true for regular software but false when applied to AI

#454

Post by GuB-42 »

beyarkay wrote: post #433
GuB-42 wrote: post #395

This made me think about a conversation I had recently with a friend who is a researcher in Natural Language Processing. Obviously what we now call LLMs have taken her field by storm, which now mostly consists of trying to understand how the fuck they work. I mean, we know they work, and they work unreasonably well, but no one knows how, no one even knows why they work!

Woah, I didn't realise NLP was still a field, but cool that they're working on interpretability now!

It still is, but at least in her case, she is doing "AI" now. It is still NLP, but it is easier to get funding if you call it AI.

That's a weird situation, LLMs are language models, the very core of NLP, and yet the field tends to be overlooked. And by the way, she doesn't like the term "LLM": a language model that is large? what kind of model? what is "large"?

Re: Beliefs that are true for regular software but false when applied to AI

#455

Post by alganet »

beyarkay wrote: post #442
alganet wrote: post #33

Earlier quoted context omitted.

Does that really matter? He is trying to lax the general public perception around AIs shortcomings. He's giving AI a break, at the expense of regular developers. This is wrong on two fronts: First, because many people foresaw the AI shortcomings and warned about them. This "we can't fix a bug like in regular software" theatre hides the fact that we can design better benchmarks, or accountability frameworks. Again, lo…

> He is trying to lax the general public perception around AIs shortcomings This is not at all what I'm trying to do. This same essay is cross-posted on LessWrong[1] because I think ASI is the most dangerous problem of our time. > This "we can't fix a bug like in regular software" theatre hides the fact that we can design better benchmarks, or accountability frameworks I'm not sure how I can say "your intuitions are…

The word intuition here have power. Surely you must have noticed that vibe coding and intuition have a strong connection.

There are people (definitely not me) who buy 100% of anything AI they read. To that beginner enthusiastic audience, your text looks like "regular software is old and unintuitive, AI is better because you can just vibe", and the text reinforces that sentiment.

Re: Beliefs that are true for regular software but false when applied to AI

#456

Post by freediver »

5- wrote: post #92
meonkeys wrote: post #79

Earlier quoted context omitted.

There's... something at https://boydkane.com/index.xml I guessed the URL based on the Quartz docs. It seems to work but only has a few items from https://boydkane.com/essays/

the author (either of the blog or its software) would do well to consult https://www.petefreitag.com/blog/rss-autodiscovery/

And then that very blog does not have RSS autodiscovery :)

Re: Beliefs that are true for regular software but false when applied to AI

#457

Post by mxgrn »

lazide wrote: post #333
dingdingdang wrote: post #306

Earlier quoted context omitted.

>> to perform up to Apple's typical standards of polish and control. >i no longer believe they have kept on to the standards in general. One 100% agree with this, if I compare AI's ability to speed up the baseline for me in terms of programming Golang (hard/tricky tasks clearly still require human input - watch out for I/O ops) with Apple's lack of ability to integrate it in even the simplest of ways.. things are jus…

Personally, I’ve never seen an AI flow of any kind that meets what would meet the quality of a typical ‘corporate’ acceptable flow. As in, reliably works, doesn’t go crazy randomly, etc. I’ve seen a lot of things that look like they’re working for a demo, but shortly after starting to use it? Trash. Not every time (and it’s getting a little better), but often enough that personally I’ve found them a net drain on prod…

> I’ve never seen an AI flow of any kind that meets what would meet the quality of a typical ‘corporate’ acceptable flow. As in, reliably works, doesn’t go crazy randomly, etc.

Jump [1] built a multi-million dollar business exactly on this, a service used by corporations in financial consultancy.

[1] https://jump.ai/

Re: Beliefs that are true for regular software but false when applied to AI

#458

Post by kees99 »

beyarkay wrote: post #410
lmc wrote: post #242

> With AI systems, almost all bad behaviour originates from the data that’s used to train them Careful with this - even with perfect data (and training), models will still get stuff wrong.

Indeed, this has been the most contentious line in the whole piece :D How do you define "perfect" data and training? I'd argue that if you trained a small NN to play tic-tac-toe perfectly, it'd quickly memorise all the possible scenarios, and since the world state is small, you could exhaustively prove that it's correct for every possible input. So at the very least, there's a counter example showing that with perfec…

Having too much parameters in your model, so that all of sample/training data is preserved perfectly, is usually considered a bad thing (overfitting).

But you're right - if dataset is exhaustive and finite, and model is large enough to preserve it perfectly - such overfitted model would work just fine, even if it's unlikely to be a particularly efficient way to build it.

Re: Beliefs that are true for regular software but false when applied to AI

#459

Post by alecbz »

conception wrote: post #389
alecbz wrote: post #381

Earlier quoted context omitted.

Only skimmed the paper, but I'm not sure how to think about "length of task" as a metric here. The cases I'm thinking about are things that could be solved in a few minutes by someone who knows what the issue is and how to use the tools involved. I spent around two days trying to debug one recent issue. A coworker who was a bit more familiar with the library involved figured it out in an hour or two. But in parallel…

This is an excellent example of human “context windows” though and it could be the llm could have solved the easy problem with better context engineering. Despite 1M token windows, things still start to get progressively worse after 100k. LLMs would overnight be amazingly better with a reliable 1M window.

What does "better context engineering" mean here? How/why are the existing token windows "unreliable"?

Re: Beliefs that are true for regular software but false when applied to AI

#460

Post by dkersten »

beyarkay wrote: post #441
dkersten wrote: post #20

Earlier quoted context omitted.

Sure, but: > these claims mostly hold, but they break down when applied to distributed systems, parallel code, or complex interactions between software systems and human processes The claims the GP quoted DON’T mostly hold, they’re just plain wrong. At least the last two, anyway.

Say more? I stand by my statement, but you're not specific enough for me to explain why I believe I'm correct.

Focusing on the last two, which I called out specifically:

> Once a bug is fixed, it won’t come back again

Regressions are extremely common, which is why regression tests are so important. It is definitely not uncommon for bugs that were fixed once to come back again. This statement doesn't "mostly hold".

> If you give specifications beforehand, you can get software that meets those specifications

In theory, maybe, but in practice its messy. It depends on your acceptance testing. It depends on whether stakeholders change their mind during implementation (not uncommon). It depends on whether the specification was complete and nothing new is learned during development (almost never the case). Providing a specification in advance does not necessarily mean what you get out the other end meets that specification, unless its a relatively small, trivial, non-changing piece and the stakeholders have the discipline to not try change things part-way through. I mean, sure, it may be true that if you throw a spec over the wall, then you will get something thrown back that meets the spec, but the real world isn't so simple.

Post reply on HN