Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Anyone struggling to get value out of coding LLMs?

news.ycombinator.com

Post reply on HN
Print view View original

211–220 of 293 posts

Re: Ask HN: Anyone struggling to get value out of coding LLMs?

#211

Post by esafak »

nottorp wrote: post #43

So you want to use a LLM on a code base. You have to feed it your code base as part of the prompt, which is limited in size. I don't suppose there's any solution where you can somehow further train a LLM on your code base to make it become part of the neural net and not part of the prompt? This could be useful on a large ish code base for helping with onboarding at the least. Of course you'd have to do both the runni…

Modern tools don't fine tune on your code base but use RAG; select the context to feed it to the LLM with each request. The better the context inference algorithm, the better the results. See if your tool tells you what files it selected.

Re: Ask HN: Anyone struggling to get value out of coding LLMs?

#212

Post by esafak »

reify wrote: post #26

An IBM study based on conversations with 2,000 global CEOs recently found that only 25% of AI initiatives have delivered their expected ROI over the last few years, and, worse still, "64% of CEOs surveyed acknowledge that the risk of falling behind drives investment in some technologies before they have a clear understanding of the value they bring to the organization." 50% of respondents also found that "the pace of…

CEOs are supposed to be able to make snap decisions without having the big picture. If everybody else is doing it, following suit will mean you won't be left behind. Worst case, you'll all make the same mistake. Of course, if you know better than the others, you can profit from that.

Re: Ask HN: Anyone struggling to get value out of coding LLMs?

#213

Post by esafak »

cies wrote: post #152

I found they work for tasks that have been done 1000s of times already. But for creative solutions in super specialized environments (lot's of the works I do it just that) they cannot help me. I expect they soon will be able to help me with basic refactoring that needs to be performed across a code base. Luckily my code uses strong types: type safety quickly shows where the LLM was tripping/forgetting.

I find it helps to give it examples of what I want it to do in such cases.

Re: Ask HN: Anyone struggling to get value out of coding LLMs?

#214

Post by FiberBundle »

loveparade wrote: post #22
gyomu wrote: post #13

There are two kinds of engineers. Those who can’t stop raving about how much of a superpower LLMs are for coding, how it’s made them 100x more productive, and is unlocking things they could’ve never done before. And those who, like you, find it to be an extremely finicky process that requires extreme amount of coddling to get average results at best. The only thing I don’t understand is why people from the former gro…

That's because the reality is somewhere in the middle. It obviously isn't 100x or even 10x except for very specific toy tasks, but for me it's probably around ~1.5-2x in everyday work. And I work on mostly long-tail scientific stuff, I imagine it's more if you do something like frontend dev with a lot of boilerplate. I'm absolutely convinced that people who say LLMs are not making them more productive either don't un…

Where's the actual evidence for the productivity boost though? Wouldn't one expect a huge increase in valuable software products or a dramatic increase in open source contributions if llms provide this kind of productivity increase?

Re: Ask HN: Anyone struggling to get value out of coding LLMs?

#215

Post by shio_desu »

Honestly I've been getting a lot of use out of LLMs for coding, and have been adjusting my approach to LLM usage over the past year and a half. The current approach I take that has been fairly effective is to spend a lot of focus energy writing out exactly what I'm looking to implement, sometimes taking 30 or more minutes creating a specs doc / implementation plan, and passing it to a agent with an architect persona to review, generate a comprehensive, phased implementation document. I then review the document, iterate with it to make sure the plan works well, then send it off to do the work.

I'm not yet a fan of Windsurf or Cursor, but honestly Roo Codes out of the box personas for architect, and orchestration to spin up focused subtasks works well for me.

I am kinda treating it how I would a junior, to guide it there, give it enough information to do the work, and check it afterwards, ensuring it didn't do things like BS test coverage or write useless tests / code.

It works pretty well for me, and I've been treating prompting these bots just as a skill I improve as I go along.

Frankly it saves me a lot of time, I knocked out some work Friday afternoon that I'd estimate was probably 5pts of effort in 3 hours. I'll take the efficiency anyday as I've had less actual coding focus time in coding implementations than I used to in my career due to other responsibilities.

Re: Ask HN: Anyone struggling to get value out of coding LLMs?

#216

Post by elfly »

Philip-J-Fry wrote: post #41
gyomu wrote: post #13

There are two kinds of engineers. Those who can’t stop raving about how much of a superpower LLMs are for coding, how it’s made them 100x more productive, and is unlocking things they could’ve never done before. And those who, like you, find it to be an extremely finicky process that requires extreme amount of coddling to get average results at best. The only thing I don’t understand is why people from the former gro…

I find LLMs 100x more productive for greenfield work. If I want to create a React app with X amount of pages, some Redux stores, Auth, etc. then it can smash that out in minutes. I can say "now add X" and it'll do it. Generally with good results. But when it comes to maintaining existing systems, or adding more complicated features, or needing to know business domain details, a LLM is usually not that great for me. T…

yeah, this is the issue. I've used Claude Code to great success to start a project. Once the basic framework is in place, it becomes less and less useful. I think it cannot handle the big context of a full project.

It is something that future versions could fix, if the context a llm can handle grows and also if you could fix it so it could handle debugging itself. Right now it can do it for short burst and it is not bad at it, but it will get distracted quickly and do other things I did not ask for

One of these problems has a technical fix that is only limited by money; the other does not

Re: Ask HN: Anyone struggling to get value out of coding LLMs?

#217

Post by bredren »

joshstrange wrote: post #51
Philip-J-Fry wrote: post #41

Earlier quoted context omitted.

I find LLMs 100x more productive for greenfield work. If I want to create a React app with X amount of pages, some Redux stores, Auth, etc. then it can smash that out in minutes. I can say "now add X" and it'll do it. Generally with good results. But when it comes to maintaining existing systems, or adding more complicated features, or needing to know business domain details, a LLM is usually not that great for me. T…

I couldn’t agree more and I’ve said the same thing many times. I have yet to see an LLM-generated app not collapse under it’s own weight after enough iterations/prompts. It gets stuck in loops (removing and adding the same code/concept), it gets hung up on simple errors, etc. For greenfield it’s amazing, no doubt, but unless you are watching it very closely and approving/reviewing the code along the way it will go of…

The way to get past the removing and adding is to have the prompt include what has already been done and what is left to do.

Then specify the need to conclude the work by a deadline.

These kinds of things cause the LLM to “finish” tasks and try to move on or say it is done when it is.

This won’t let you skip the curation of output along the way, but at least some of the stumbling around can be handled with prompting.

Re: Ask HN: Anyone struggling to get value out of coding LLMs?

#218

Post by mattfca »

jghn wrote: post #130
gyomu wrote: post #13

There are two kinds of engineers. Those who can’t stop raving about how much of a superpower LLMs are for coding, how it’s made them 100x more productive, and is unlocking things they could’ve never done before. And those who, like you, find it to be an extremely finicky process that requires extreme amount of coddling to get average results at best. The only thing I don’t understand is why people from the former gro…

> There are two kinds of engineers. Apparently there are at least three as I fit neither of these molds of yours. They are neither making me 100x more productive, nor am I putting in an extreme amount of coddling. For context, in the 30ish years I've been writing code professionally, I've *always* needed to look stuff up constantly. I know what I want, and I know what's possible. I just can never remember things like…

this describes me and my experience exactly

Re: Ask HN: Anyone struggling to get value out of coding LLMs?

#219

Post by nottorp »

esafak wrote: post #211
nottorp wrote: post #43

So you want to use a LLM on a code base. You have to feed it your code base as part of the prompt, which is limited in size. I don't suppose there's any solution where you can somehow further train a LLM on your code base to make it become part of the neural net and not part of the prompt? This could be useful on a large ish code base for helping with onboarding at the least. Of course you'd have to do both the runni…

Modern tools don't fine tune on your code base but use RAG; select the context to feed it to the LLM with each request. The better the context inference algorithm, the better the results. See if your tool tells you what files it selected.

Yes but why?

On a large code base it would probably work better if you didn't need to put it all in the context... if you even could...

Re: Ask HN: Anyone struggling to get value out of coding LLMs?

#220

Post by nottorp »

squidbeak wrote: post #206
nottorp wrote: post #32

Earlier quoted context omitted.

> LLMs are better at reading terrible documentation than the average programmer. LLMs don't go and read the terrible documentation for you when prompted. They reproduce the information posted by other people that struggled with said terrible documentation, if it was posted somewhere. It's still better than a modern web search or struggling with the terrible documentation on your own - for introductory stuff. For goin…

This is is wildly incorrect. Documentation categorically isn't excluded from LLMs' training sets, and they are very well able to summarize that documentation when asked.

I believe you missed "when prompted".
Post reply on HN