Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some thoughts on LLMs and software development

martinfowler.com

Post reply on HN
Print view View original

321–330 of 422 posts

Re: Some thoughts on LLMs and software development

#321

Post by diegolas »

bolobo wrote: post #244
brabel wrote: post #236

Earlier quoted context omitted.

I don’t think anyone needs to be convinced at this point. Every developer is using LLM and I really can’t believe someone who has made a career out of automating things wouldn’t be immediately drawn to trying them at least. Every single company seems convinced and using it too. The comparison to crypto makes no sense.

> Every developer is using LLM Citation needed. In my circles, Senior engineer are not using them a lot, or in very specific use cases. My company is blocking LLMs use apart from a few pilots (which I am part of, and while claude code is cool, its effectiveness on a 10-year old distributed codebase is pretty low). You can't make sweeping statements like this, software engineering is a large field. And I use claude co…

> You can't make sweeping statements like this, software engineering is a large field.

that goes both ways

Re: Some thoughts on LLMs and software development

#322

Post by cess11 »

dragonwriter wrote: post #265

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. This is an example of my least favorite style of feigned insight: redefining a term into meaninglessness just so you can say something that sounds different while not actu…

The point, though awkwardly stated, is that there is no difference between 'hallucination' output and any other output from these compressed databases, just like there is no such difference between two queries on a typical RDBMS.

It's a good point.

Re: Some thoughts on LLMs and software development

#323

Post by the_af »

_heimdall wrote: post #311
jychang wrote: post #297

Earlier quoted context omitted.

I dislike the term “stochastic parrot”, because there’s plenty of evidence that LLMs do have an understanding of at least some things that they are saying. We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. The correct features are active for the sentence “the word parrots is plural”. If you stop assuming the LLMs have no internal represen…

> We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. Seeing which parts of a model (they aren't neurons) light up when shown a face doesn't necessarily indicate understanding. The model is a complex web of numbers representing a massively compressed data space. It could easily be that what you see light up when shown a face only indicates…

I think this could be seen as a proxy for evidence that there's some degree of reasoning, if we think we can identify specialized features that always become involved in some kinds of outputs. It's not proof, but it's not nothing either. It has some parallels about research on human brains are conducted, right?

Re: Some thoughts on LLMs and software development

#324

Post by aatd86 »

dapperdrake wrote: post #279
xyzzy123 wrote: post #270

Earlier quoted context omitted.

You might as well say it's interpolating or extrapolating. That's what people are usually doing too, even when recalling situations that they were personally involved in. I think we call it "hallucinating" when the machine does this in an un-human-like way.

The longer term for this is "stochastic parrot". See another HN comment here comparing LLMs to theater actors or movie actors. LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often. Might even be a useful case of pareidolia (a term I dislike, because a world without any pattern matching whatsoever would not necessarily be "better")…

But..but...humans do the same thing? This is input/output with the output being formed from applying the function of our life experiences (training) to the input? They just don't have the hormonal circuitry to optimise for whatever our bodies are trying to optimise for when we take decisions.

Re: Some thoughts on LLMs and software development

#325

Post by cmiles74 »

aatd86 wrote: post #324
dapperdrake wrote: post #279

Earlier quoted context omitted.

The longer term for this is "stochastic parrot". See another HN comment here comparing LLMs to theater actors or movie actors. LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often. Might even be a useful case of pareidolia (a term I dislike, because a world without any pattern matching whatsoever would not necessarily be "better")…

But..but...humans do the same thing? This is input/output with the output being formed from applying the function of our life experiences (training) to the input? They just don't have the hormonal circuitry to optimise for whatever our bodies are trying to optimise for when we take decisions.

People actually understand the input and the output. An LLM understand neither, it's generating output that is statistically likely, within some bounds. As Fowler said, it's a pleasant coincidence that some of this output has value to us.

(For sure, arguments can be made that the relationship betweens the terms that the model has encoded could maybe be called "model thinking" or "model understanding" but it's not how people work.)

Re: Some thoughts on LLMs and software development

#326

Post by Jensson »

Cthulhu_ wrote: post #249
FromTheFirstIn wrote: post #172

Earlier quoted context omitted.

It’s funny how nothing seems to be AI’s fault.

That's because it's software / an application. I don't blame my editor for broken code either. You can't put blame on software itself, it just does what it's programmed to do. But also, blameless culture is IMO important in software development. If a bug ends up in production, whose fault is it? The developer that wrote the code? The LLM that generated it? The reviewer that approved it? The product owner that decided…

> You can't put blame on software itself, it just does what it's programmed to do.

This isn't what AI enthusiasts say about AI though, they only bring that up when they get defensive but then go around and say it will totally replace software engineers and is not just a tool.

Re: Some thoughts on LLMs and software development

#327

Post by Filligree »

sarchertech wrote: post #318
jama211 wrote: post #269

Earlier quoted context omitted.

Only when purely vibe coding. AI currently saves a LOT of time if you get it to generate boilerplate, diagnose bugs, or assist with sandboxed issues. The proof is in the pudding. The work I do takes me half as long as it used to and is just as high in quality, even though I manage and carefully curate the output.

I use AI for most of those things. And I think it probably saves me a bit of time. But in that study that came out a few weeks ago where they actually looked at time saved, every single developer overestimated their time saved. To the point where even the ones who lost time thought they saved time. LLMs are very good at making you feel like you’re saving time even when you aren’t. That doesn’t mean they can’t be a ne…

That study predates Claude Code though.

I’m not surprised by the contents. I had the same feeling; I made some attempts at using LLMs for coding prior to CC, and with rare exceptions it never saved me any time.

CC changed that situation hugely, at least in my subjective view. It’s of course possible that it’s not as good as I feel it is, but I would at least want a new study.

Re: Some thoughts on LLMs and software development

#328

Post by neonspark »

_heimdall wrote: post #311
jychang wrote: post #297

Earlier quoted context omitted.

I dislike the term “stochastic parrot”, because there’s plenty of evidence that LLMs do have an understanding of at least some things that they are saying. We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. The correct features are active for the sentence “the word parrots is plural”. If you stop assuming the LLMs have no internal represen…

> We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. Seeing which parts of a model (they aren't neurons) light up when shown a face doesn't necessarily indicate understanding. The model is a complex web of numbers representing a massively compressed data space. It could easily be that what you see light up when shown a face only indicates…

> Seeing which parts of a model (they aren't neurons)…

I thought models were composed of neural network layers, among other things. Are these data structures called something different?

Re: Some thoughts on LLMs and software development

#329

Post by neonspark »

ACCount37 wrote: post #308
jychang wrote: post #297

Earlier quoted context omitted.

I dislike the term “stochastic parrot”, because there’s plenty of evidence that LLMs do have an understanding of at least some things that they are saying. We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. The correct features are active for the sentence “the word parrots is plural”. If you stop assuming the LLMs have no internal represen…

This matches what we know about LLMs and hallucination-avoidance behavior in LLMs. "Wrong answers on SAT" is also the leading hypothesis on why o3 was such an outlier - far more prone to hallucinations than either prior or following OpenAI models. On SAT, giving a random answer is right 20% of the time - more if you ruled at least one obviously wrong answer out. Saying "I don't know" and not answering is right 0% of…

> But another cause of hallucinations is limited self-awareness of modern LLMs… Humans have some awareness of the limits of their knowledge

Until you said that I didn’t realize just how much humans “hallucinate“ in just the same ways that AI does. I have a friend who is fluent in Spanish, a native speaker, but got a pretty weak grammar education when he was in high school. Also, he got no education at all in Critical thinking, at least not formally. So this guy is really, really fluent in his native language, but can often have a very difficult time explaining why he uses whatever grammar he uses. I think the whole world is realizing how little our brains can correctly explain and identify the grammar we use flawlessly.

He helps me to improve my Spanish a lot, he can correct me with 100% accuracy of course, but I’ve noticed on many occasions, including this week, that when I ask a question about why he said something one way or another in Spanish, he will just make up some grammar rule that doesn’t actually exist, and is in fact not true.

He said something like “you say it this way when you really know the person and you’re saying that the other way when it’s more formal“, but I think really it was just a slangy way to mis-stress something and it didn’t have to do with familiar/formal or not. I’ve learned not to challenge him on any of these grammar rules that he makes up, because he will dig his heels in, and I’ve learned just to ignore him because he won’t have remembered this made up grammar rule in a week anyway.

This really feels like a very tight analogy with what my LLM does to me every day, except that when I challenge the LLM it will profusely apologize and declare itself incorrect even if it had been correct after all. Maybe LLMs are a little bit too humble.

I imagine this is a very natural tendency in humans, and I imagine I do it much more than I’m aware of. So how do humans use self-awareness to reduce the odds of this happening?

I think we mostly get trained in higher education to not trust the first thought that comes into our head, even if it feels self consistent and correct. We eventually learn to say “I don’t know” even if it’s about something that we are very, very good at.

Re: Some thoughts on LLMs and software development

#330

Post by sceptic123 »

dragonwriter wrote: post #265

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. This is an example of my least favorite style of feigned insight: redefining a term into meaninglessness just so you can say something that sounds different while not actu…

I actually found that comment interesting. It's pointing towards something I've struggled with around LLMs. They are (currently) incapable of knowing if what they output is correct, so the idea that "it's all hallucinations" acknowledges that point and gives useful context for anyone using LLMs for software development.
Post reply on HN