Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I write software with LLMs

stavros.io

Post reply on HN
Print view View original

171–180 of 544 posts

Re: How I write software with LLMs

#171

Post by loloquwowndueo »

or_am_i wrote: post #160
miguelgrinberg wrote: post #151

> One thing I’ve noticed is that different people get wildly different results with LLMs, so I suspect there’s some element of how you’re talking to them that affects the results. It's always easier to blame the prompt and convince yourself that you have some sort of talent in how you talk to LLMs that other's don't. In my experience the differences are mostly in how the code produced by the LLM is reviewed. Develope…

It's always easier to blame the model and convince yourself that you have some sort of talent in reviewing LLM's work that others don't. In my experience the differences are mostly in how the code produced by LLM is prompted and what context is given to the agent. Developers who have experience delegating their work are more likely to prevent downstream problems from happening immediately and complain their colleague…

Never takes long for the “you’re holding it wrong” crowd to pop in.

Re: How I write software with LLMs

#172

Post by cultofmetatron »

miguelgrinberg wrote: post #151

> One thing I’ve noticed is that different people get wildly different results with LLMs, so I suspect there’s some element of how you’re talking to them that affects the results. It's always easier to blame the prompt and convince yourself that you have some sort of talent in how you talk to LLMs that other's don't. In my experience the differences are mostly in how the code produced by the LLM is reviewed. Develope…

> Developers who have experience reviewing code are more likely to find problems immediately and complain they aren't getting great results without a lot of hand holding

this makes me feel better about the amount of disdain I've been feeling about the output from these llms. sometimes it popsout exactly what I need but I can never count on it to not go offrails and require a lot of manual editing.

Re: How I write software with LLMs

#173

Post by darkerside »

loloquwowndueo wrote: post #171
or_am_i wrote: post #160

Earlier quoted context omitted.

It's always easier to blame the model and convince yourself that you have some sort of talent in reviewing LLM's work that others don't. In my experience the differences are mostly in how the code produced by LLM is prompted and what context is given to the agent. Developers who have experience delegating their work are more likely to prevent downstream problems from happening immediately and complain their colleague…

Never takes long for the “you’re holding it wrong” crowd to pop in.

That's a terrible reason for a mass consumer tool to fail, and a perfectly reasonable one for a professional power tool to fail

Re: How I write software with LLMs

#174

Post by titanomachy »

arialdomartini wrote: post #157
titanomachy wrote: post #144

Earlier quoted context omitted.

If it could be done with 30 cents of Haiku calls, maybe it wasn't a complicated enough project to provide good signal?

Fair point. I could try with a harder problem. This still does not explain why Claude Code felt the need to use Opus, and why Opus felt the need to burn 12$ or such an easy task. I mean, it's 40 times the cost.

I'm a bit confused actually, you said you used Claude Code for both examples? Was that a typo, or was it (1) Claude Code instructed to use a hierarchy of agents and (2) Claude Code allowed to do whatever it wants?

Re: How I write software with LLMs

#175

Post by kasey_junk »

miguelgrinberg wrote: post #151

> One thing I’ve noticed is that different people get wildly different results with LLMs, so I suspect there’s some element of how you’re talking to them that affects the results. It's always easier to blame the prompt and convince yourself that you have some sort of talent in how you talk to LLMs that other's don't. In my experience the differences are mostly in how the code produced by the LLM is reviewed. Develope…

I think that code review experience is a big driver of success with the llms, but my take away is somewhat different. If you’ve spent a lot of time reviewing other people’s code you realize the failures you see with llms are common failures full stop. Humans make them too.

I also think reviewable code, that is code specifically delivered in a manner that makes code review more straightforward was always valuable but now that the generation costs have lowered its relative value is much higher. So structuring your approach (including plans and prompts) to drive to easily reviewed code is a more valuable skill than before.

Re: How I write software with LLMs

#177

Post by Roxxik »

mikkupikku wrote: post #168
miguelgrinberg wrote: post #151

> One thing I’ve noticed is that different people get wildly different results with LLMs, so I suspect there’s some element of how you’re talking to them that affects the results. It's always easier to blame the prompt and convince yourself that you have some sort of talent in how you talk to LLMs that other's don't. In my experience the differences are mostly in how the code produced by the LLM is reviewed. Develope…

It's not skill with talking to an LLM, it's the users skill and experience with the problem they're asking the LLM to solve. They work better for problems the prompter knows well and poorly for problems the prompter doesn't really understand. Try it yourself. Ask claude for something you don't really understand. Then learn that thing, get a fresh instance of claude and try again, this time it will work much better be…

Not only you understanding the how, but you not understanding the goal.

I often use AI successfully, but in a few cases I had, it was bad. That was when I didn't even know the end goal and regularly switched the fundamental assumptions that the LLM tried to build up.

One case was a simulation where I wanted to see some specific property in the convergence behavior, but I had no idea how it would get there in the dynamics of the simulation or how it should behave when perturbed.

So the LLM tried many fundamentally different approaches and when I had something that specifically did not work it immediately switched approaches.

Next time I get to work on this (toy) problem I will let it implement some of them, fully parametrize them and let me have a go with it. There is a concrete goal and I can play around myself to see if my specific convergence criterium is even possible.

Re: How I write software with LLMs

#178

Post by Dannymetconan »

ttanveer wrote: post #163
miguelgrinberg wrote: post #151

> One thing I’ve noticed is that different people get wildly different results with LLMs, so I suspect there’s some element of how you’re talking to them that affects the results. It's always easier to blame the prompt and convince yourself that you have some sort of talent in how you talk to LLMs that other's don't. In my experience the differences are mostly in how the code produced by the LLM is reviewed. Develope…

That seems to make sense. Any suggestions to improve this skill of reviewing code? I think especially a number of us more junior programmers lack in this regard, and don't see a clear way of improving this skill beyond just using LLMs more and learning with time?

It's "easy". You just spend a couple of years reviewing PRs and working in a professional environment getting feedback from your peers and experience the consequences of code.

There is no shortcut unfortunately.

Re: How I write software with LLMs

#179

Post by stavros »

miguelgrinberg wrote: post #151

> One thing I’ve noticed is that different people get wildly different results with LLMs, so I suspect there’s some element of how you’re talking to them that affects the results. It's always easier to blame the prompt and convince yourself that you have some sort of talent in how you talk to LLMs that other's don't. In my experience the differences are mostly in how the code produced by the LLM is reviewed. Develope…

That's what I meant, though. I didn't mean "I say the right words", I meant "I don't give them a sentence and walk away".

Re: How I write software with LLMs

#180

Post by zingar »

akhrail1996 wrote: post #22

Genuine question: what's the evidence that the architect → developer → reviewer pipeline actually produces better results than just... talking to one strong model in one session? The author uses different models for each role, which I get. But I run production agents on Opus daily and in my experience, if you give it good context and clear direction in a single conversation, the output is already solid. The ceremony…

Nitpick: I don’t think architect is a good name for this role. It’s more of a technical project kickoff function: these are the things we anticipate we need to do, these are the risks etc.

I do find it different from the thinking that one does when writing code so I’m not surprised to find it useful to separate the step into different context, with different tools.

Is it useful to tell something “you are an architect?” I doubt it but I don’t have proof apart from getting reasonable results without it.

With human teams I expect every developer to learn how to do this, for their own good and to prevent bottlenecks on one person. I usually find this to be a signal of good outcomes and so I question the wisdom of biasing the LLM towards training data that originates in spaces where “architect” is a job title.

Post reply on HN