Some thoughts on LLMs and software development • Hacker News Board

Re: Some thoughts on LLMs and software development

#351

Post by socksy » Fri, Aug 29, 2025, 3:35 PM UTC

scott_w wrote: post #316

bartread wrote: post #278

Earlier quoted context omitted.

I’ve never liked that this behaviour is described using the term “hallucination”. If a human being talked confidently about something that they were just making up out of thin air by synthesizing based (consciously or unconsciously) on other information they know you wouldn’t call it “hallucination”: you’d call it “bullshit”. And, honestly, “bullshit” is a much more helpful way of thinking about this behaviour becaus…

> If a human being talked confidently about something that they were just making up out of thin air by synthesizing based (consciously or unconsciously) on other information they know you wouldn’t call it “hallucination”: you’d call it “bullshit”. I'd recommend you watch https://www.youtube.com/watch?v=u9CE6a5t59Y&t=2134s&pp=ygUYc... which covers the topic of bullshit. I don't think we can call LLM output "bullshit"…

It's precisely because they can't care that they are by definition bullshit machines. See https://link.springer.com/article/10.1007/s10676-024-09775-5

Re: Some thoughts on LLMs and software development

#352

Post by jimbokun » Fri, Aug 29, 2025, 3:35 PM UTC

Filligree wrote: post #327

sarchertech wrote: post #318

Earlier quoted context omitted.

I use AI for most of those things. And I think it probably saves me a bit of time. But in that study that came out a few weeks ago where they actually looked at time saved, every single developer overestimated their time saved. To the point where even the ones who lost time thought they saved time. LLMs are very good at making you feel like you’re saving time even when you aren’t. That doesn’t mean they can’t be a ne…

That study predates Claude Code though. I’m not surprised by the contents. I had the same feeling; I made some attempts at using LLMs for coding prior to CC, and with rare exceptions it never saved me any time. CC changed that situation hugely, at least in my subjective view. It’s of course possible that it’s not as good as I feel it is, but I would at least want a new study.

> That study predates Claude Code though.

Is there a study demonstrating Claude Code improves productivity?

Re: Some thoughts on LLMs and software development

#353

Post by dghlsakjg » Fri, Aug 29, 2025, 3:42 PM UTC

Eridrus wrote: post #337

sceptic123 wrote: post #330

Earlier quoted context omitted.

I actually found that comment interesting. It's pointing towards something I've struggled with around LLMs. They are (currently) incapable of knowing if what they output is correct, so the idea that "it's all hallucinations" acknowledges that point and gives useful context for anyone using LLMs for software development.

Humans are also incapable of knowing whether their output is correct. We merely convince ourselves that it is and then put our thoughts in contact with the external world and other people to see if we actually are.

Humans are capable of using formal logic to reach a provably correct conclusion from valid premises. That conclusion can then be used as premises for a further conclusion. Follow the logic the other way and using our powers of observation, and sensing, we can gather base level truths and know that what we are saying is correct in an absolute sense.

Computers use formal logic all the time to output truth, LLMs, however, do not.

Re: Some thoughts on LLMs and software development

#354

Post by sarchertech » Fri, Aug 29, 2025, 3:48 PM UTC

Filligree wrote: post #327

sarchertech wrote: post #318

Earlier quoted context omitted.

I use AI for most of those things. And I think it probably saves me a bit of time. But in that study that came out a few weeks ago where they actually looked at time saved, every single developer overestimated their time saved. To the point where even the ones who lost time thought they saved time. LLMs are very good at making you feel like you’re saving time even when you aren’t. That doesn’t mean they can’t be a ne…

That study predates Claude Code though. I’m not surprised by the contents. I had the same feeling; I made some attempts at using LLMs for coding prior to CC, and with rare exceptions it never saved me any time. CC changed that situation hugely, at least in my subjective view. It’s of course possible that it’s not as good as I feel it is, but I would at least want a new study.

I don’t believe that CC is so much better than cursor using Claude models that it moves the needle enough to flip the results of that study.

The key thing to look at is that even the participants that did objectively save time, overestimated time saved by a huge amount.

But also you’re always likely to be at least one model ahead of any studies that come out.

Re: Some thoughts on LLMs and software development

#355

Post by thrawa8387336 » Fri, Aug 29, 2025, 3:54 PM UTC

neonspark wrote: post #329

ACCount37 wrote: post #308

Earlier quoted context omitted.

This matches what we know about LLMs and hallucination-avoidance behavior in LLMs. "Wrong answers on SAT" is also the leading hypothesis on why o3 was such an outlier - far more prone to hallucinations than either prior or following OpenAI models. On SAT, giving a random answer is right 20% of the time - more if you ruled at least one obviously wrong answer out. Saying "I don't know" and not answering is right 0% of…

> But another cause of hallucinations is limited self-awareness of modern LLMs… Humans have some awareness of the limits of their knowledge Until you said that I didn’t realize just how much humans “hallucinate“ in just the same ways that AI does. I have a friend who is fluent in Spanish, a native speaker, but got a pretty weak grammar education when he was in high school. Also, he got no education at all in Critical…

Spanish in particular has more connotations per word than English. It's not even the grammar or spelling, those have rules and that's that. But choosing appropriate words, is more like every word has it right place and time and context. Some close examples would be the N-word or the R-word in English, as they are steeped in meanings far beyond the literal.

Re: Some thoughts on LLMs and software development

#356

Post by thrawa8387336 » Fri, Aug 29, 2025, 3:56 PM UTC

naasking wrote: post #336

dapperdrake wrote: post #279

Earlier quoted context omitted.

The longer term for this is "stochastic parrot". See another HN comment here comparing LLMs to theater actors or movie actors. LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often. Might even be a useful case of pareidolia (a term I dislike, because a world without any pattern matching whatsoever would not necessarily be "better")…

> LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often. This sentence is inherently contradictory. If LLM output is meaningful more than chance, then it's literally not "just spewing words". Therefore whatever model it is using to generate that meaning must contain some semantic content, even if it's not semantic content that's as…

It's a sufficiently large N shannonizer. Nothing more

Re: Some thoughts on LLMs and software development

#357

Post by thrawa8387336 » Fri, Aug 29, 2025, 4:04 PM UTC

jeppester wrote: post #107

In my company I feel that we getting totally overrun with code that's 90% good, 10% broken and almost exactly what was needed. We are producing more code, but quality is definitely taking a hit now that no-one is able to keep up. So instead of slowly inching towards the result we are getting 90% there in no time, and then spending lots and lots of time on getting to know the code and fixing and fine-tuning everything…

Imagine someones add 10 UTs carefully devised and someone notices they need 1 more during the PR.

Scenario B, you add 40 with an LLM, that look good on paper but only cover 6 of the original ones. Besides, who's going to pay careful attention to a PR with 40.

"Must be so thorough!".

Re: Some thoughts on LLMs and software development

#358

Post by jonnycomputer » Fri, Aug 29, 2025, 4:06 PM UTC

>All an LLM does is produce hallucinations, it’s just that we find some of them useful.

Nice callback to "All models are wrong, but some of them are useful."

Re: Some thoughts on LLMs and software development

#359

Post by thwarted » Fri, Aug 29, 2025, 4:07 PM UTC

dragonwriter wrote: post #265

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. This is an example of my least favorite style of feigned insight: redefining a term into meaninglessness just so you can say something that sounds different while not actu…

It's meant to contrast/correct the claim that

"LLMs produce either truth or they produce hallucinations."

Claims worded like that give the impression that if we can just reduce/eliminate the hallucinations, all that will remain will be the truth.

But that's not the case. What is the case is that all the output is the same thing, hallucination, and that some of those hallucinations just so happen to reflect reality (or expectations) so appears to embody truth.

It's like rolling a die and wanting one to come up, and when it does saying "the die knew what I wanted".

An infinite number of monkeys typing on an infinite number of typewriters will eventually produce the script for Hamlet.

That doesn't mean the monkeys know about Shakespeare, or Hamlet, or even words, or even what they are doing being chained to typewriters.

We've found a way to optimize the infinite typing monkeys to output something that passes for Hamlet much sooner than infinity.

Re: Some thoughts on LLMs and software development

#360

Post by seunosewa » Fri, Aug 29, 2025, 4:15 PM UTC

cmiles74 wrote: post #325

aatd86 wrote: post #324

Earlier quoted context omitted.

But..but...humans do the same thing? This is input/output with the output being formed from applying the function of our life experiences (training) to the input? They just don't have the hormonal circuitry to optimise for whatever our bodies are trying to optimise for when we take decisions.

People actually understand the input and the output. An LLM understand neither, it's generating output that is statistically likely, within some bounds. As Fowler said, it's a pleasant coincidence that some of this output has value to us. (For sure, arguments can be made that the relationship betweens the terms that the model has encoded could maybe be called "model thinking" or "model understanding" but it's not how…

We do the same thing. We pick words that are statistically likely to get us what we want. And much of it is unconscious. You don't formally reason about every word you speak. You are focused on your objective, and your brain fills in the gaps.