Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some thoughts on LLMs and software development

martinfowler.com

Post reply on HN
Print view View original

331–340 of 422 posts

Re: Some thoughts on LLMs and software development

#331

Post by kaosoaksh »

computerex wrote: post #241
johnnienaked wrote: post #237

Earlier quoted context omitted.

I need to be convinced. Go ahead, convince me. Please describe clearly and concisely in one or two sentences the clear economic value/advantage of LLMs.

The company I work for uses LLM's for digital marketing, the company has over 100M ARR selling products build on top of LLM's with real life measurable impact as measured by KPIs.

> real life measurable impact as measured by KPIs

This is making me even more skeptical of your claims. Individual metrics are often very poor at tracking reality.

Re: Some thoughts on LLMs and software development

#332

Post by neonspark »

ACCount37 wrote: post #313
red75prime wrote: post #287

Earlier quoted context omitted.

The analogy goes down the drain when a criterion for good performance is being objectively right. Like with Reinforcement Learning from Verifiable Rewards.

RLVR can also encourage hallucinations quite easily. Think of SAT: giving a random answer is right 20% of the time, giving "I don't know" is right 0% of the time. If you only reward for test score, you encourage guesswork. So good RL reward design is as important as ever. That being said, there are methods to train LLMs against hallucinations, and they do improve hallucination-avoidance. But anti-hallucination capabi…

I think what you say is true, and I think that this is exactly true for humans as well. There is no known way to completely eliminate unintentional bullshit coming from a human’s mouth. We have many techniques for reducing it, including critical thinking, but we are all susceptible to it and I imagine we do it many times a day without too much concern.

We need to make these models much much better, but it’s going to be quite difficult to reduce the levels to even human levels. And the BS will always be there with us. I suppose BS is the natural side effect of any complex system, artificial or biological, that tries to navigate the problem space of reality and speak on it. These systems, sometimes called “minds”, are going to produce things that sound right but just are not true.

Re: Some thoughts on LLMs and software development

#333

Post by sodapopcan »

bongodongobob wrote: post #195
achierius wrote: post #161

Earlier quoted context omitted.

I used to be one of those people, then I started saving these scripts in a folder and realized just how much time it saved me. Especially for nontrivial scripts which require a lot of typing or input specification.

This was me before LLMs. Had tons of scripts saved. I just don't see the point anymore "Take this list of users, remove from x group and reset their passwords" takes less time than trying to remember what I named that file and where it's saved. Anything that can be done in under 100 lines isn't worth saving anymore.

Usually one would save this type of thing as a build tool task.

Re: Some thoughts on LLMs and software development

#334

Post by kaosoaksh »

mexicocitinluez wrote: post #319
jama211 wrote: post #275

Earlier quoted context omitted.

Finally, an opinion on here that’s reasonable and isn’t “AI is perfect” or “AI is useless”.

One of the things that has struck me as odd is just how little self-awareness devs have when talking about "skin in the game" with regard to CEO's hawking AI products. Like, we have just as much to lose as they have to gain. Of course a part of us doesn't want these tools to be as good as some people say they are because it directly affects our future and livelihood. No, they can't do everything. Yes, they can do som…

> doesn’t want these tools to be as good as some people say they are

No, this is because that would mean AGI. And it’s obviously not that.

Re: Some thoughts on LLMs and software development

#335

Post by jstrieb »

red75prime wrote: post #287
jstrieb wrote: post #214

Earlier quoted context omitted.

I have been explaining this to friends and family by comparing LLMs to actors. They deliver a performance in-character, and are only factual if it happens to make the performance better. https://jstrieb.github.io/posts/llm-thespians/

The analogy goes down the drain when a criterion for good performance is being objectively right. Like with Reinforcement Learning from Verifiable Rewards.

Nobody that I'd be using this analogy with is currently using LLMs for tasks that are covered by RLVF. They're asking models for factual information about the real world (Google replacement), or to generate text (write a cover letter), not the type of outputs that are verifiable within formal systems—by definition the type of output that RLVF is intended to improve. The actor analogy is still helpful for providing intuition to non-technical people who don't know how to think about LLMs, but do use them.

Also, unless I am mistaken, RLVF changes the training to make LLMs less likely to hallucinate, but in no way does it make hallucination impossible. Under the hood, the models still work the same way (after training), and the analogy still applies, no?

Re: Some thoughts on LLMs and software development

#336

Post by naasking »

dapperdrake wrote: post #279
xyzzy123 wrote: post #270

Earlier quoted context omitted.

You might as well say it's interpolating or extrapolating. That's what people are usually doing too, even when recalling situations that they were personally involved in. I think we call it "hallucinating" when the machine does this in an un-human-like way.

The longer term for this is "stochastic parrot". See another HN comment here comparing LLMs to theater actors or movie actors. LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often. Might even be a useful case of pareidolia (a term I dislike, because a world without any pattern matching whatsoever would not necessarily be "better")…

> LLMs just spew words. It just so happens that human beings can decode them into something related, useful, and meaningful surprisingly often.

This sentence is inherently contradictory. If LLM output is meaningful more than chance, then it's literally not "just spewing words". Therefore whatever model it is using to generate that meaning must contain some semantic content, even if it's not semantic content that's as rich as humans are capable of. The "stochastic parrot" term is thus silly.

Re: Some thoughts on LLMs and software development

#337

Post by Eridrus »

sceptic123 wrote: post #330
dragonwriter wrote: post #265

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. This is an example of my least favorite style of feigned insight: redefining a term into meaninglessness just so you can say something that sounds different while not actu…

I actually found that comment interesting. It's pointing towards something I've struggled with around LLMs. They are (currently) incapable of knowing if what they output is correct, so the idea that "it's all hallucinations" acknowledges that point and gives useful context for anyone using LLMs for software development.

Humans are also incapable of knowing whether their output is correct. We merely convince ourselves that it is and then put our thoughts in contact with the external world and other people to see if we actually are.

Re: Some thoughts on LLMs and software development

#338

Post by threecheese »

utyop22 wrote: post #157
jeppester wrote: post #107

In my company I feel that we getting totally overrun with code that's 90% good, 10% broken and almost exactly what was needed. We are producing more code, but quality is definitely taking a hit now that no-one is able to keep up. So instead of slowly inching towards the result we are getting 90% there in no time, and then spending lots and lots of time on getting to know the code and fixing and fine-tuning everything…

"but quality is definitely taking a hit now that no-one is able to keep up." And its going to get worse! So please explain to me how in the net, you are going to be better off? You're not. I think most people haven't taken a decent economics class and don't deeply understand the notion of trade offs and the fact there is no free lunch.

Fast feedback is one benefit, given the 90% is releasable - even if only to a segment of users. This might be anathema to good engineering, but a benefit to user experience research and to organizations that want to test their market for demand.

Fast feedback is also great for improving release processes; when you have a feedback loop with Product, UX, Engineering, Security etc, being able to front load some % of a deliverable can help you make better decisions that may end up being a time saver net/net.

Re: Some thoughts on LLMs and software development

#339

Post by naasking »

sillyfluke wrote: post #309
jychang wrote: post #297

Earlier quoted context omitted.

I dislike the term “stochastic parrot”, because there’s plenty of evidence that LLMs do have an understanding of at least some things that they are saying. We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. The correct features are active for the sentence “the word parrots is plural”. If you stop assuming the LLMs have no internal represen…

>I dislike the term “stochastic parrot”, because there’s plenty of evidence that LLMs do have an understanding of at least some things that they are saying. It's bold to use the term "understanding" in this context. You ask it something about a topic, it gives an answer like someone who understands the topic. You change the prompt slightly, where a human who understands the topic would still give the right response t…

> You change the prompt slightly, where a human who understands the topic would still give the right response trivially, the LLM outputs an answer that is both wrong/irrelevant and unpredicably and non-humanly wrong in a way that no human who exhibited understanding with the first answer could be predicted to answer the second question in the same bizarrw manner as the LLM.

I think this should make you question whether the prompt change was really as trivial as you imply. Providing an example of this would elucidate.

Re: Some thoughts on LLMs and software development

#340

Post by brabel »

skydhash wrote: post #304
brabel wrote: post #296

Earlier quoted context omitted.

Stack overflow published recently a survey in which something like 80% of developers were using AI and the rest “wants to soon”. By now I have trouble believing a competent developer is still convinced they shouldn’t use it at all , though a few ludites perhaps might hold on for a bit longer.

Stack overflow published a report about text editors and Emacs wasn’t part of the list. So I’m very sceptical about SO surveys.

I was also offended by that :D.
Post reply on HN