Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some thoughts on LLMs and software development

martinfowler.com

Post reply on HN
Print view View original

371–380 of 422 posts

Re: Some thoughts on LLMs and software development

#371

Post by roadside_picnic »

naasking wrote: post #339
sillyfluke wrote: post #309

Earlier quoted context omitted.

>I dislike the term “stochastic parrot”, because there’s plenty of evidence that LLMs do have an understanding of at least some things that they are saying. It's bold to use the term "understanding" in this context. You ask it something about a topic, it gives an answer like someone who understands the topic. You change the prompt slightly, where a human who understands the topic would still give the right response t…

> You change the prompt slightly, where a human who understands the topic would still give the right response trivially, the LLM outputs an answer that is both wrong/irrelevant and unpredicably and non-humanly wrong in a way that no human who exhibited understanding with the first answer could be predicted to answer the second question in the same bizarrw manner as the LLM. I think this should make you question wheth…

Here's an entire paper [0] showing the impact of extremely minor structural changes on the quality of the results of the model. Things as simple as not using a colon in the prompt can lead to notably degraded (or improved) performance.

0. https://arxiv.org/pdf/2310.11324

Re: Some thoughts on LLMs and software development

#372

Post by jstummbillig »

jeltz wrote: post #312
jstummbillig wrote: post #288

Earlier quoted context omitted.

I envy the people working at mystical places where humans were on average writing code of high quality prior LLMs. I'll never know you now.

I am working at one right now and I have worked at such in the past. One of the main tricks is to treat code reviews very seriously so people are not incentived to write lazy code. You need to build a cultire which cares about quality of both product and code. You also need decent developers, but not necessarily great developers.

Oh, I understand what you need to do. It's like losing weight. It's fairly simple.

And at the same time it's borderline impossible proven by the fact that people can't do it, even though everyone understands and roughly everyone agrees on how it works.

So the actual "trick" turns out to be understanding what keeps people from doing the necessary things that they all agree on are important – like treating code reviews very seriously. And getting that part right turns out to be fairly hard.

Re: Some thoughts on LLMs and software development

#373

Post by tony2016 »

The developer is supposed to check and verify the code that an LLM creates. Ask for smaller requests in the prompts so you don't get too much code. Unit tests are verified and run by the developer. I don't know what he means by an LLM runs all the tests and gives green. It can fake running tests. I always the tests myself, whether they were created by an LLM or not.

Re: Some thoughts on LLMs and software development

#374

Post by dragonwriter »

cess11 wrote: post #322
dragonwriter wrote: post #265

> My former colleague Rebecca Parsons, has been saying for a long time that hallucinations aren’t a bug of LLMs, they are a feature. Indeed they are the feature. All an LLM does is produce hallucinations, it’s just that we find some of them useful. This is an example of my least favorite style of feigned insight: redefining a term into meaninglessness just so you can say something that sounds different while not actu…

The point, though awkwardly stated, is that there is no difference between 'hallucination' output and any other output from these compressed databases, just like there is no such difference between two queries on a typical RDBMS. It's a good point.

> The point, though awkwardly stated, is that there is no difference between 'hallucination' output and any other output from these compressed databases,

But there is, except when you redefine "hallucination" so there isn't. And, when you retain the definition where there is a difference, you find there are techniques by which you can reduce hallucinations, which is important and useful. Changing the definition to eliminate the distinction is actively harmful to understanding and productive use of LLMs, for the benefit of making what superficially seems like an insightful comment.

Re: Some thoughts on LLMs and software development

#375

Post by jama211 »

sarchertech wrote: post #318
jama211 wrote: post #269

Earlier quoted context omitted.

Only when purely vibe coding. AI currently saves a LOT of time if you get it to generate boilerplate, diagnose bugs, or assist with sandboxed issues. The proof is in the pudding. The work I do takes me half as long as it used to and is just as high in quality, even though I manage and carefully curate the output.

I use AI for most of those things. And I think it probably saves me a bit of time. But in that study that came out a few weeks ago where they actually looked at time saved, every single developer overestimated their time saved. To the point where even the ones who lost time thought they saved time. LLMs are very good at making you feel like you’re saving time even when you aren’t. That doesn’t mean they can’t be a ne…

I mean, I used to average 2 hours of intense work a day and now it’s 1 hour.

Re: Some thoughts on LLMs and software development

#376

Post by _heimdall »

neonspark wrote: post #328
_heimdall wrote: post #311

Earlier quoted context omitted.

> We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. Seeing which parts of a model (they aren't neurons) light up when shown a face doesn't necessarily indicate understanding. The model is a complex web of numbers representing a massively compressed data space. It could easily be that what you see light up when shown a face only indicates…

> Seeing which parts of a model (they aren't neurons)… I thought models were composed of neural network layers, among other things. Are these data structures called something different?

That point may not have been relevant for me to include.

I was getting at the idea that a neuron is a very specific feature of a biological brain, regardless of what AI researchers may call their hardware they aren't made of neurons.

Re: Some thoughts on LLMs and software development

#377

Post by _heimdall »

the_af wrote: post #323
_heimdall wrote: post #311

Earlier quoted context omitted.

> We can trace which neurons activate for a face recognition model and see that a certain neuron does light up when it sees a face. Seeing which parts of a model (they aren't neurons) light up when shown a face doesn't necessarily indicate understanding. The model is a complex web of numbers representing a massively compressed data space. It could easily be that what you see light up when shown a face only indicates…

I think this could be seen as a proxy for evidence that there's some degree of reasoning, if we think we can identify specialized features that always become involved in some kinds of outputs. It's not proof, but it's not nothing either. It has some parallels about research on human brains are conducted, right?

It does have parallels to the human brain, absolutely. We've been studying the human brain in similar ways for much longer though and we still don't know much about it.

We do know what areas of a human brain often light up in response to various conditions. We don't know why that is though, or how it actually works. Maybe more importantly for LLMs, we don't know how human memory works, where it is stored, how to recognize or even define consciousness, etc.

Seeing what areas of a brain or an LLM light up can be interesting, but I'd be very cautious trying to read much into it.

Re: Some thoughts on LLMs and software development

#378

Post by abcde-bbq »

More details on tolerance would be nice. Examples:

1: when building games for my kids, the tolerance is high so the LLM can implement a feature in one shot and I will manually test the feature (without bothering to review the code or generating unit tests or using a type-safe language).

2: building a demo at work has lower tolerance for errors, but is still good with LLM with a brief code review.

3: for production code, other processes can be in place to meet the even lower tolerance requirements.

Re: Some thoughts on LLMs and software development

#379

Post by scott_w »

seunosewa wrote: post #360
cmiles74 wrote: post #325

Earlier quoted context omitted.

People actually understand the input and the output. An LLM understand neither, it's generating output that is statistically likely, within some bounds. As Fowler said, it's a pleasant coincidence that some of this output has value to us. (For sure, arguments can be made that the relationship betweens the terms that the model has encoded could maybe be called "model thinking" or "model understanding" but it's not how…

We do the same thing. We pick words that are statistically likely to get us what we want. And much of it is unconscious. You don't formally reason about every word you speak. You are focused on your objective, and your brain fills in the gaps.

We absolutely do not "pick words that are statistically likely to get us what we want." We use words to try to articulate (to varying levels of success) a message that we want to communicate. The words, tone, speed, pitch, etc. all convey meaning.

> And much of it is unconscious.

That does not mean we're "picking words statistically likely to get us what we want," it means "our brains do a lot of work subconsciously." Nothing more.

> You are focused on your objective, and your brain fills in the gaps.

This is a total contradiction of what you said at the start. LLMs are not focused on an objective, they are using very complex statistical algorithms to determine output strings. There is no objective to an LLM's output.

Re: Some thoughts on LLMs and software development

#380

Post by jychang »

_heimdall wrote: post #376
neonspark wrote: post #328

Earlier quoted context omitted.

> Seeing which parts of a model (they aren't neurons)… I thought models were composed of neural network layers, among other things. Are these data structures called something different?

That point may not have been relevant for me to include. I was getting at the idea that a neuron is a very specific feature of a biological brain, regardless of what AI researchers may call their hardware they aren't made of neurons.

1. They are neurons, whether you like it or not. A binary tree may not have squirrels living in them, but it's still a tree, even though the word "tree" here is defined differently than from biology. Or are you going to say a binary tree is not a tree?

2. You are about 5 years behind in terms of the research. Look into hierarchical feature representation and how MLP neurons work. (Or even in older CNNs and RNNs etc). And I'm willingly using the word "neuron" instead of "feature" here because while I know "feature" is more correct in general, there are definitely small toy models where you can pinpoint an individual neuron to represent a feature such as a face.

Post reply on HN