Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some thoughts on LLMs and software development

martinfowler.com

Post reply on HN
Print view View original

181–190 of 422 posts

Re: Some thoughts on LLMs and software development

#181

Post by tptacek »

skydhash wrote: post #180
ares623 wrote: post #67

Earlier quoted context omitted.

But there is an underlying deterministic property in the TCP example. A message is either received within a timeout or not. How can that be extralopated with LLMs? How does a system independently know that it's arrived at a correct answer within a timeout or not? Has the halting problem been solved?

> How can that be extralopated with LLMs? How does a system independently know that it's arrived at a correct answer within a timeout or not? That's the catch 22 with LLM. You're supposed to be both the asker and the verifier. Which in practice, it's not that great. LLMs will just find the snippets of code that matches somehow and just act on it (It's the "I'm feeling Lucky" button with extra steps) In traditional pr…

No? These kinds of analyses all seem to rely on the notion that the LLM-caller needs to accept whatever output the LLM provides. In practice, they discard all the outputs that don't compile, and then a further subset of the ones that don't --- those outputs that aren't instantly clear to the caller.

My intuition for the problem here is that people are fixated on the nondeterminism of the LLM itself, which is of limited importance to the actual problem domain of code generation. The LLM might spit out ancient Egyptian hieroglyphics! It's true! The LLM is completely nondeterministic. But nothing like that is ever going to get merged into `main`.

It's fine if you want to go on about how bad "vibe coding" is, with LLM-callers that don't bother to read LLM output, because they're not competent. But here we're assuming an otherwise competent developer. You can say the vibe coder is the more important phenomenon, but the viber doesn't implicate the halting problem.

Re: Some thoughts on LLMs and software development

#182

Post by positron26 »

ares623 wrote: post #25

> Other forms of engineering have to take into account the variability of the world. > Maybe LLMs mark the point where we join our engineering peers in a world on non-determinism. Those other forms of engineering have no choice due to the nature of what they are engineering. Software engineers already have a way to introduce determinism into the systems they build! We’re going backwards!

It's uncertainty, not non-determinism. I don't design a rocket with no idea about the chain of events. I have inscrutable uncertainty everywhere, from manufacture to flight.

Re: Some thoughts on LLMs and software development

#183

Post by riwsky »

utyop22 wrote: post #153
gnerd00 wrote: post #90

Earlier quoted context omitted.

this was true, but then it wasn't... the research world several years ago, had a moment when the machinery could reliably solve multi-step problems.. there had to be intermediary results; and machinery could solve problems in a domain where they were not trained specifically.. this caused a lot of excitement, and several hundred billion dollars in various investments.. Since no one actually knows how all of it works,…

"Since no one actually knows how all of it works, not even the builders, here we are." To me this is the most bizarre part. Have we ever had a technology deployed at this scale without a true understanding of its inner workings? My fear is that the general public perception of AI will be damaged since for most LLMs = AI.

Humanity used fire for like a bazillion years before figuring out thermodynamics

Re: Some thoughts on LLMs and software development

#184

Post by ants_everywhere »

skydhash wrote: post #180
ares623 wrote: post #67

Earlier quoted context omitted.

But there is an underlying deterministic property in the TCP example. A message is either received within a timeout or not. How can that be extralopated with LLMs? How does a system independently know that it's arrived at a correct answer within a timeout or not? Has the halting problem been solved?

> How can that be extralopated with LLMs? How does a system independently know that it's arrived at a correct answer within a timeout or not? That's the catch 22 with LLM. You're supposed to be both the asker and the verifier. Which in practice, it's not that great. LLMs will just find the snippets of code that matches somehow and just act on it (It's the "I'm feeling Lucky" button with extra steps) In traditional pr…

> LLMs will just find the snippets of code that matches somehow

This suggests a huge gap in your understanding of LLMs if we are to take this literally.

> LLM programming, when first started, was more about a direct english to finished code translation

There is no direct english to finished code translation. A prompt like "write me a todo app" has infinitely many codes it maps to with different tradeoffs and which appeal to different people. Even if LLMs never made any coding mistakes, there is no function that maps a statement like that to specific pieces of code unless you're making completely arbitrary choices like the axiom of choice.

So we're left with the fact that we have to specify what we want. And at that LLMs do exceptionally well.

Re: Some thoughts on LLMs and software development

#185

Post by skydhash »

tptacek wrote: post #181
skydhash wrote: post #180

Earlier quoted context omitted.

> How can that be extralopated with LLMs? How does a system independently know that it's arrived at a correct answer within a timeout or not? That's the catch 22 with LLM. You're supposed to be both the asker and the verifier. Which in practice, it's not that great. LLMs will just find the snippets of code that matches somehow and just act on it (It's the "I'm feeling Lucky" button with extra steps) In traditional pr…

No? These kinds of analyses all seem to rely on the notion that the LLM-caller needs to accept whatever output the LLM provides. In practice, they discard all the outputs that don't compile, and then a further subset of the ones that don't --- those outputs that aren't instantly clear to the caller. My intuition for the problem here is that people are fixated on the nondeterminism of the LLM itself, which is of limit…

Valid programs are almost infinite. Context free grammars (which describe valid programs) are generative. When you're programming, you are mostly restricting the set of valid program to include only the few that satisfy the specs. Adding an extra 0 to a number is valid, but put that in the context of money transactions, it's a "hell breaks loose" situation.

SO that's why "it compiles" is worthless in a business settings. Of course it should compile. That's the bare minimum of expectations. And even "it passes the tests" is not that great. That just means you have not mess things up. So review and quality (accountability for both) is paramount, so that the proper stuff get shipped (and fixed swiftly if there was a mistake).

Re: Some thoughts on LLMs and software development

#186

Post by tjwebbnorfolk »

furyofantares wrote: post #99
nicwolff wrote: post #81

> I’ve often heard, with decent reason, an LLM compared to a junior colleague. No, they're like an extremely experienced and knowledgeable senior colleague – who drinks heavily on the job. Overconfident, forgetful, sloppy, easily distracted. But you can hire so many of them, so cheaply, and they don't get mad when you fire them!

These metaphors all suck. Well, ok, yours is funny. But anyway, LLMs are just very different from any human. They are extremely shallow, even compared to a junior developer. But extremely broad, even compared to the most experienced developer. They type real fuckin fast compared to anyone on earth, but they need to be told what to do much more carefully than anyone on earth.

> but they need to be told what to do much more carefully than anyone on earth.

have you ever managed an offshore team. holy cow

Re: Some thoughts on LLMs and software development

#187

Post by seadan83 »

nicwolff wrote: post #81

> I’ve often heard, with decent reason, an LLM compared to a junior colleague. No, they're like an extremely experienced and knowledgeable senior colleague – who drinks heavily on the job. Overconfident, forgetful, sloppy, easily distracted. But you can hire so many of them, so cheaply, and they don't get mad when you fire them!

Like a person that has memorized every single answer in the world to every single question ever asked - but is completely dead behind the eyes.

Re: Some thoughts on LLMs and software development

#188

Post by tptacek »

skydhash wrote: post #185
tptacek wrote: post #181

Earlier quoted context omitted.

No? These kinds of analyses all seem to rely on the notion that the LLM-caller needs to accept whatever output the LLM provides. In practice, they discard all the outputs that don't compile, and then a further subset of the ones that don't --- those outputs that aren't instantly clear to the caller. My intuition for the problem here is that people are fixated on the nondeterminism of the LLM itself, which is of limit…

Valid programs are almost infinite. Context free grammars (which describe valid programs) are generative. When you're programming, you are mostly restricting the set of valid program to include only the few that satisfy the specs. Adding an extra 0 to a number is valid, but put that in the context of money transactions, it's a "hell breaks loose" situation. SO that's why "it compiles" is worthless in a business setti…

It feels like you stopped reading before "and then a further subset of those".

Again: my claim is simply that whatever else is going on, the halting problem doesn't enter into it, because the user in this scenario isn't obligated to prove arbitrary programs. Here, I can solve the halting problem right now: "only accept branchless programs with finite numbers of instructions". Where's my Field Medal? :)

It always feels like the "LLMs are nondeterministic" people are relying on the claim that it's impossible to tell whether an arbitrary program is branchless and finite. Obviously, no, that's not true.

Re: Some thoughts on LLMs and software development

#189

Post by ants_everywhere »

skydhash wrote: post #185
tptacek wrote: post #181

Earlier quoted context omitted.

No? These kinds of analyses all seem to rely on the notion that the LLM-caller needs to accept whatever output the LLM provides. In practice, they discard all the outputs that don't compile, and then a further subset of the ones that don't --- those outputs that aren't instantly clear to the caller. My intuition for the problem here is that people are fixated on the nondeterminism of the LLM itself, which is of limit…

Valid programs are almost infinite. Context free grammars (which describe valid programs) are generative. When you're programming, you are mostly restricting the set of valid program to include only the few that satisfy the specs. Adding an extra 0 to a number is valid, but put that in the context of money transactions, it's a "hell breaks loose" situation. SO that's why "it compiles" is worthless in a business setti…

I have LLMs generate Haskell. Having the code compile means everything type checks and is not worthless. That's a huge constraint on valid programs.

Re: Some thoughts on LLMs and software development

#190

Post by SillyUsername »

utyop22 wrote: post #153
gnerd00 wrote: post #90

Earlier quoted context omitted.

this was true, but then it wasn't... the research world several years ago, had a moment when the machinery could reliably solve multi-step problems.. there had to be intermediary results; and machinery could solve problems in a domain where they were not trained specifically.. this caused a lot of excitement, and several hundred billion dollars in various investments.. Since no one actually knows how all of it works,…

"Since no one actually knows how all of it works, not even the builders, here we are." To me this is the most bizarre part. Have we ever had a technology deployed at this scale without a true understanding of its inner workings? My fear is that the general public perception of AI will be damaged since for most LLMs = AI.

This is a misconception, we absolutely do know how LLMs work, that's how we can write them and publish research papers.

The idea we don't is tabloid journalism, it's simply because the output is (usually) randomised - taken to mean, by those who lack the technical chops, that programmers "don't know how it works" because the output is indeterministic.

This is not withstanding we absolutely can repeat the output by using not randomisation (temperature 0).

Post reply on HN