Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some thoughts on LLMs and software development

martinfowler.com

Post reply on HN
Print view View original

311–320 of 422 posts

Re: Some thoughts on LLMs and software development

#311

Post by _heimdall »

jychang wrote: post #297
dapperdrake wrote: post #279

Earlier quoted context omitted.

The longer term for this is "stochastic parrot". See another HN comment here comparing LLMs to theater actors or movie actors. LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often. Might even be a useful case of pareidolia (a term I dislike, because a world without any pattern matching whatsoever would not necessarily be "better")…

I dislike the term “stochastic parrot”, because there’s plenty of evidence that LLMs do have an understanding of at least some things that they are saying. We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. The correct features are active for the sentence “the word parrots is plural”. If you stop assuming the LLMs have no internal represen…

> We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face.

Seeing which parts of a model (they aren't neurons) light up when shown a face doesn't necessarily indicate understanding.

The model is a complex web of numbers representing a massively compressed data space. It could easily be that what you see light up when shown a face only indicates what specific part of the model is housing the compresses data related to recognizing specific facial features.

Re: Some thoughts on LLMs and software development

#312

Post by jeltz »

jstummbillig wrote: post #288
whstl wrote: post #253

Earlier quoted context omitted.

LLMs are amazing at producing boilerplate, which removes the incentive to get rid of it. Boilerplate sucks to review. You just see a big mass of code and can't fully make sense of it when reviewing. Also, Github sucks for reviewing PRs with too many lines. So junior/mid devs are just churning boilerplate-rich code and don't really learn. The only outcome here is code quality is gonna go down very very fast.

I envy the people working at mystical places where humans were on average writing code of high quality prior LLMs. I'll never know you now.

I am working at one right now and I have worked at such in the past. One of the main tricks is to treat code reviews very seriously so people are not incentived to write lazy code. You need to build a cultire which cares about quality of both product and code. You also need decent developers, but not necessarily great developers.

Re: Some thoughts on LLMs and software development

#313

Post by ACCount37 »

red75prime wrote: post #287
jstrieb wrote: post #214

Earlier quoted context omitted.

I have been explaining this to friends and family by comparing LLMs to actors. They deliver a performance in-character, and are only factual if it happens to make the performance better. https://jstrieb.github.io/posts/llm-thespians/

The analogy goes down the drain when a criterion for good performance is being objectively right. Like with Reinforcement Learning from Verifiable Rewards.

RLVR can also encourage hallucinations quite easily. Think of SAT: giving a random answer is right 20% of the time, giving "I don't know" is right 0% of the time. If you only reward for test score, you encourage guesswork. So good RL reward design is as important as ever.

That being said, there are methods to train LLMs against hallucinations, and they do improve hallucination-avoidance. But anti-hallucination capabilities are fragile and do not fully generalize. There's no (known) way to train full awareness of its own capabilities into an LLM.

Re: Some thoughts on LLMs and software development

#314

Post by raziel2p »

jeltz wrote: post #312
jstummbillig wrote: post #288

Earlier quoted context omitted.

I envy the people working at mystical places where humans were on average writing code of high quality prior LLMs. I'll never know you now.

I am working at one right now and I have worked at such in the past. One of the main tricks is to treat code reviews very seriously so people are not incentived to write lazy code. You need to build a cultire which cares about quality of both product and code. You also need decent developers, but not necessarily great developers.

It's very easy to go from what you're describing to a place hamstrung by nitpicking, though. The code review becomes more important than the code itself and appearances start mattering more than results.

Re: Some thoughts on LLMs and software development

#315

Post by raziel2p »

TomBers wrote: post #292

One aspect of using LLM's to code that I have not seen mentioned is the "loss of attachment" to code in my projects. For example, working on my project ( https://mudg.fly.dev ) I wanted to experiment with a new FE graph library. I asked an llm the fantastic ( https://tidewave.ai ) and it completely implemented a solution, which turned out to be worse than the current solution. If I had spent many hours working on a s…

Have you considered that LLMs are also biased against new languages and libraries, so the code quality will be worse compared to something more established regardless of what you personally think/feel?

Re: Some thoughts on LLMs and software development

#316

Post by scott_w »

bartread wrote: post #278
dragonwriter wrote: post #265

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. This is an example of my least favorite style of feigned insight: redefining a term into meaninglessness just so you can say something that sounds different while not actu…

I’ve never liked that this behaviour is described using the term “hallucination”. If a human being talked confidently about something that they were just making up out of thin air by synthesizing based (consciously or unconsciously) on other information they know you wouldn’t call it “hallucination”: you’d call it “bullshit”. And, honestly, “bullshit” is a much more helpful way of thinking about this behaviour becaus…

> If a human being talked confidently about something that they were just making up out of thin air by synthesizing based (consciously or unconsciously) on other information they know you wouldn’t call it “hallucination”: you’d call it “bullshit”.

I'd recommend you watch https://www.youtube.com/watch?v=u9CE6a5t59Y&t=2134s&pp=ygUYc... which covers the topic of bullshit. I don't think we can call LLM output "bullshit" because someone spewing bullshit has to not care about whether what they're saying is true or false. LLMs don't "care" about anything because they're not human. It's better to give it an alternative term to differentiate it from the human behaviour, even if the observed output is recognisable.

Re: Some thoughts on LLMs and software development

#317

Post by bo1024 »

jstrieb wrote: post #214
oo0shiny wrote: post #122

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. What a great way of framing it. I've been trying to explain this to people, but this is a succinct version of what I was stumbling to convey.

I have been explaining this to friends and family by comparing LLMs to actors. They deliver a performance in-character, and are only factual if it happens to make the performance better. https://jstrieb.github.io/posts/llm-thespians/

This is also related to the philosophical definition of bullshit[1]: speech intended to persuade or influence without any active intention to be either true or false.

[1] https://en.wikipedia.org/wiki/On_Bullshit

Re: Some thoughts on LLMs and software development

#318

Post by sarchertech »

jama211 wrote: post #269
globular-toast wrote: post #239

Earlier quoted context omitted.

Yep, my strong feeling is that the net benefit of all of this will be zero. The time you have to spend holding the LLM hand is almost equal to how much time you would have spent writing it yourself. But then you've got yourself a codebase that you didn't write yourself, and we all know hunting bugs in someone else's code is way harder than code you had a part in designing/writing. People are honestly just drunk on th…

Only when purely vibe coding. AI currently saves a LOT of time if you get it to generate boilerplate, diagnose bugs, or assist with sandboxed issues. The proof is in the pudding. The work I do takes me half as long as it used to and is just as high in quality, even though I manage and carefully curate the output.

I use AI for most of those things. And I think it probably saves me a bit of time.

But in that study that came out a few weeks ago where they actually looked at time saved, every single developer overestimated their time saved. To the point where even the ones who lost time thought they saved time.

LLMs are very good at making you feel like you’re saving time even when you aren’t. That doesn’t mean they can’t be a net productivity benefit.

But I’d be very very very surprised if you have real hard data to back up your feelings about your work taking you half as long and being equal quality.

Re: Some thoughts on LLMs and software development

#319

Post by mexicocitinluez »

jama211 wrote: post #275
rootusrootus wrote: post #158

For a hot second I thought LLMs were coming for our jobs. Then I realized they were just as likely to end up creating mountains of things for us to fix later. And as things settle down, I find good use cases for Claude Code that augment me but are in no danger of replacing me. It certainly has its moments.

Finally, an opinion on here that’s reasonable and isn’t “AI is perfect” or “AI is useless”.

One of the things that has struck me as odd is just how little self-awareness devs have when talking about "skin in the game" with regard to CEO's hawking AI products.

Like, we have just as much to lose as they have to gain. Of course a part of us doesn't want these tools to be as good as some people say they are because it directly affects our future and livelihood.

No, they can't do everything. Yes, they can do some things. It's that simple.

Re: Some thoughts on LLMs and software development

#320

Post by sarchertech »

xtracto wrote: post #301
FromTheFirstIn wrote: post #172

Earlier quoted context omitted.

It’s funny how nothing seems to be AI’s fault.

If your toaster burns your breakfast bread, Do you ultimately blame "it"? You gdt mad, swear at it, maybe even throw it to the wall on a git of rage but, at the end of the day, deep inside you still know you screwed.

If I bought an AI powered toaster that allows me to select a desired shade of toast, I select light golden brown, and it burns my toast, I certainly do blame “it”.

I wouldn’t throw it against a wall because I’m not a psychopath, but I would demand my money back.

Post reply on HN