Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some thoughts on LLMs and software development

martinfowler.com

Post reply on HN
Print view View original

261–270 of 422 posts

Re: Some thoughts on LLMs and software development

#261

Post by aitchnyu »

nine_k wrote: post #78
sebnukem2 wrote: post #6

> hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. Nice.

I'd rather say that LLMs live in a world that consists entirely of stories, nothing but words and their combinations. Thy have no other reality. So they are good at generating more stories that would sit well with the stories they already know. But the stories are often imprecise, and sometimes contradictory, so they have to guess. Also, LLMs don't know how to count, but they know that two usually follows one, and th…

I've asked LLMs to modify 5 files at a time and mark a checklist. Also image generators no longer draw 6 fingered hands.

Re: Some thoughts on LLMs and software development

#262

Post by wk_end »

colordrops wrote: post #227
FromTheFirstIn wrote: post #172

Earlier quoted context omitted.

It’s funny how nothing seems to be AI’s fault.

How could a tool be at fault? If an airplane crashes is the plane at fault or the designers, engineers, and/or pilot?

Designers, engineers, and/or pilots aren't tools, so that's a strange rhetorical question.

At any rate, it depends on the crash. The NTSB will investigate and release findings that very well may assign fault to the design of the plane and/or pilot or even tools the pilot was using, and will make recommendations about how to avoid a similar crash in the future, which could include discontinuing the use of certain tools.

Re: Some thoughts on LLMs and software development

#263

Post by lagrange77 »

jstrieb wrote: post #214
oo0shiny wrote: post #122

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. What a great way of framing it. I've been trying to explain this to people, but this is a succinct version of what I was stumbling to convey.

I have been explaining this to friends and family by comparing LLMs to actors. They deliver a performance in-character, and are only factual if it happens to make the performance better. https://jstrieb.github.io/posts/llm-thespians/

I'll steal that.

Re: Some thoughts on LLMs and software development

#264

Post by lagrange77 »

aitchnyu wrote: post #256
oo0shiny wrote: post #122

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. What a great way of framing it. I've been trying to explain this to people, but this is a succinct version of what I was stumbling to convey.

All models are wrong, some are merely useful - 1976/1933/earlier adage.

Right, all models are inherently wrong. It's up to the user know about its limits / uncertainty.

But i think this 'being wrong' is kind of confusing when talking about LLMs (in contrast to systems/scientific modelling). In what they model (language), the current LLMs are really good and acurate, except for example the occasional chinese character in the middle of a sentence.

But what we mean by LLMs 'being wrong' most of the time is being factually wrong in answering a question, that is expressed as language. That's a layer on top of what the model is designed to model.

EDITS:

So saying 'the model is wrong' when it's factually wrong above the language level isn't fair.

I guess this is essentially the same thought as 'all they do is hallucinate'.

Re: Some thoughts on LLMs and software development

#265

Post by dragonwriter »

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful.

This is an example of my least favorite style of feigned insight: redefining a term into meaninglessness just so you can say something that sounds different while not actually saying anything new.

Yes, if you redefine "hallucination" from "produce output containing detailed information despite that information not being grounded in external reality, in a manner distantly analogous to a human reporting sense data produced by a literal hallucination rather than the external inputs that are presumed normally to ground sense data" to "produce output", its true that all LLMs do is "hallucinate", and that "hallucinating" is not a undesirable behavior.

But you haven't said anything new about the thing that was called "hallucination" by everyone else, or about the thing--LLM output in general--that you have called "hallucination". Everyone already knew that producing output wasn't undesirable. You've just taken the label conventionally attached to a bad behavior, attached it to a broader category that includes all behavior, and used the power of equivocation to make something that sounds novel without saying anything new.

Re: Some thoughts on LLMs and software development

#266

Post by scott_w »

dragonwriter wrote: post #265

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. This is an example of my least favorite style of feigned insight: redefining a term into meaninglessness just so you can say something that sounds different while not actu…

Fowler is not really redefining "hallucination." He's using a form of irony that emphasises how fundamental "hallucinations" are to the operation of the system. One might also say "you can't get rid of collateral damage from bombs. Indeed, collateral damage is the feature, it's just some of that is what we want to blow up."

You're not meant to take it literally.

Re: Some thoughts on LLMs and software development

#267

Post by utyop22 »

computerex wrote: post #219
utyop22 wrote: post #157

Earlier quoted context omitted.

"but quality is definitely taking a hit now that no-one is able to keep up." And its going to get worse! So please explain to me how in the net, you are going to be better off? You're not. I think most people haven't taken a decent economics class and don't deeply understand the notion of trade offs and the fact there is no free lunch.

Technology has always helped people. Are you one of the people that say optimizing compilers are bad? Do you not use the intellisense? Or IDEs? Do you not use higher level languages? Why not write in assembly all the time? No free lunch right. Yes there are trade offs, but at this point if you haven’t found a way to significantly amplify and scale yourself using llms, and your plan is to instead pretend that they are…

"Technology has always helped people. Are you one of the people that say optimizing compilers are bad? Do you not use the intellisense? Or IDEs? Do you not use higher level languages? Why not write in assembly all the time? No free lunch right."

Actually I am not a software engineer for a living so I have zero vested interest or bias lmao. That said, I studied Comp Sci at a top institution and really loved defining algorithms and had no interest in actual coding as it didn't give my brain the variety it needs.

If you are employed as a software engineer, I am probably more open to realizing the problems that only become obvious to you later on.

Re: Some thoughts on LLMs and software development

#268

Post by [deleted] »

oo0shiny wrote: post #122

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. What a great way of framing it. I've been trying to explain this to people, but this is a succinct version of what I was stumbling to convey.

[deleted]

Re: Some thoughts on LLMs and software development

#269

Post by jama211 »

globular-toast wrote: post #239
utyop22 wrote: post #157

Earlier quoted context omitted.

"but quality is definitely taking a hit now that no-one is able to keep up." And its going to get worse! So please explain to me how in the net, you are going to be better off? You're not. I think most people haven't taken a decent economics class and don't deeply understand the notion of trade offs and the fact there is no free lunch.

Yep, my strong feeling is that the net benefit of all of this will be zero. The time you have to spend holding the LLM hand is almost equal to how much time you would have spent writing it yourself. But then you've got yourself a codebase that you didn't write yourself, and we all know hunting bugs in someone else's code is way harder than code you had a part in designing/writing. People are honestly just drunk on th…

Only when purely vibe coding. AI currently saves a LOT of time if you get it to generate boilerplate, diagnose bugs, or assist with sandboxed issues.

The proof is in the pudding. The work I do takes me half as long as it used to and is just as high in quality, even though I manage and carefully curate the output.

Re: Some thoughts on LLMs and software development

#270

Post by xyzzy123 »

scott_w wrote: post #266
dragonwriter wrote: post #265

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. This is an example of my least favorite style of feigned insight: redefining a term into meaninglessness just so you can say something that sounds different while not actu…

Fowler is not really redefining "hallucination." He's using a form of irony that emphasises how fundamental "hallucinations" are to the operation of the system. One might also say "you can't get rid of collateral damage from bombs. Indeed, collateral damage is the feature, it's just some of that is what we want to blow up." You're not meant to take it literally.

You might as well say it's interpolating or extrapolating. That's what people are usually doing too, even when recalling situations that they were personally involved in.

I think we call it "hallucinating" when the machine does this in an un-human-like way.

Post reply on HN