Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My LLM codegen workflow

harper.blog

Post reply on HN
Print view View original

161–168 of 168 posts

Re: My LLM codegen workflow

#161

Post by khqc »

bcoates wrote: post #85

That lonely/downtime section at the end is a giant red flag for me. It looks like the sort of nonproductive yak-shaving you do when you're stuck or avoiding an unpleasant task--coasting, fooling around incrementally with your LLM because your project's fucked and you psychologically need some sense of progress. The opposite of this is burnout--one of the things they don't tell you about successful projects with good…

I guess you're not a big fan of rubber duck debugging then? Whenever I get stuck I like to ask myself a bunch of questions and thought experiments to get a better understanding of the problem/project, and with LLMs I'm forced to spell out each one of these questions/experiments coherently, which ends up being great documentation later on. I think LLMs are great if you're actually interested in the fundamentals of your problem/project, otherwise it just turns into a sinkhole that sucks you in.

Re: My LLM codegen workflow

#162

Post by balls187 »

yoz wrote: post #96
balls187 wrote: post #34

Earlier quoted context omitted.

It's likely the end of a lot of abstractions that made programming easier. At some point, specialized code-gen transformer models should get really good at just spitting out the lowest level code required to perform the job.

Disagree. Some abstractions are still vital, and it's for the same reasons as always: communicate purpose and complexity concisely rather than hiding it. The best code is that which explains itself most efficiently and readably to Whoever Reads It Next. That's even more important with LLMs than with humans, because the LLMs probably have far less context than the humans do. Developers often fall back on standard abst…

I’ve thought about your comment, and I think we’re both right.

Fundamentally, computers are a series of high and low voltages, and everything above that is a combination of abstraction and interpretations.

Fundamentally there will always be some level of this-it’s not like an A(G)I will interface directly using electrical signals (though in some distant future it could).

However from what I believe, this current phase of AI (LLMs + Generators + Tools) are showing that computers do not need to solve problems the same way humans need to because computers face different constraints.

So abstractions that programmers utilize to manage complexity won’t be necessary (in some future time).

Re: My LLM codegen workflow

#163

Post by ziml77 »

briga wrote: post #73

Absolutely LLMs are great for greenfield projects. They can get you to a prototype for a new idea faster than any tool yet invented. Where they start to break down, I find, is when you ask it to make changes/refactors to existing code and mature projects. They usually lack context, so they doesn't hesitate to introduce lots of extra complexity, add frameworks you don't need, and in general just make the situation wor…

I've found they will also introduce subtle changes. I just used o1 recently to pull code from a Python notebook and remove all the intermediate output. It basically got it right except for one string that was used to look up info from an external source. It just dropped 2 characters from the end of the string. That issue required a bit of time to track down because I thought it was an issue with the test environment!

Eventually I ended up looking at the notebook and the extracted code side-by-side and carefully checking every line. Despite being split across dozens of cells, it would have been faster if I had started out by just manually copying the code out of each meaningful cell and pasted it all together.

Re: My LLM codegen workflow

#164

Post by cheema33 »

ziml77 wrote: post #163
briga wrote: post #73

Absolutely LLMs are great for greenfield projects. They can get you to a prototype for a new idea faster than any tool yet invented. Where they start to break down, I find, is when you ask it to make changes/refactors to existing code and mature projects. They usually lack context, so they doesn't hesitate to introduce lots of extra complexity, add frameworks you don't need, and in general just make the situation wor…

I've found they will also introduce subtle changes. I just used o1 recently to pull code from a Python notebook and remove all the intermediate output. It basically got it right except for one string that was used to look up info from an external source. It just dropped 2 characters from the end of the string. That issue required a bit of time to track down because I thought it was an issue with the test environment!…

I have seen LLMs do this. This is why I try to create a git repo, even if it is just local and then diff my changes. This allows me to catch LLM changing something that was completely unrelated to the task it was working on.

Re: My LLM codegen workflow

#165

Post by newtwilly »

ghuntley wrote: post #50
rd wrote: post #15

Has anyone who evolved from a baseline of just using Cursor chat and freestyling to a proper workflow like this got any anecdata to share on noticeable improvements? Does the time invested into the planning benefit you? Have you noticed less hallucinations? Have you saved time overall? I’d be curious to hear because my current workflow is basically 1. Have idea 2. create-next-app + ShadCN + TailwindUI boilerplate 3.…

If you steer it and build a stdlib, you get better outcomes. See https://ghuntley.com/stdlib

Hi, I appreciate you sharing. I've been starting to use this advice with a different tool. Just FYI, this sentence kind of came out of nowhere and it wasn't clear what you meant: > The foundational LLM models right now are what I'd estimate to be at circa 45% accuracy and require frequent steering

Do your rules count as frequent steering and lead to increased 'accuracy', or is that the 'accuracy' you're seeing with your current workflow, rules and all?

Re: My LLM codegen workflow

#166

Post by bcoates »

khqc wrote: post #161
bcoates wrote: post #85

That lonely/downtime section at the end is a giant red flag for me. It looks like the sort of nonproductive yak-shaving you do when you're stuck or avoiding an unpleasant task--coasting, fooling around incrementally with your LLM because your project's fucked and you psychologically need some sense of progress. The opposite of this is burnout--one of the things they don't tell you about successful projects with good…

I guess you're not a big fan of rubber duck debugging then? Whenever I get stuck I like to ask myself a bunch of questions and thought experiments to get a better understanding of the problem/project, and with LLMs I'm forced to spell out each one of these questions/experiments coherently, which ends up being great documentation later on. I think LLMs are great if you're actually interested in the fundamentals of you…

Sure, but I just use OneNote, the company wiki, or like physical sticky notes for that. I'm not seeing the value of having the LLM give feedback aside from entertainment

Re: My LLM codegen workflow

#167

Post by adr1an »

bambax wrote: post #20

This is all fine for a solo dev, but how does this work with a team / squad, working on the same code base? Having 7 different instances of an LLM analyzing the same code base and making suggestions would not just be economically wasteful, it would also be unpractical or even dangerous? Outside of RAG, which is a different thing, are there products that somehow "centralize" the context for a team, where all questions…

Cody/ sourcegraph provide workspaces for teams/ enterprise. Probably for this and other reasons.

Re: My LLM codegen workflow

#168

Post by kridsdale3 »

layer8 wrote: post #28
kridsdale3 wrote: post #26

Earlier quoted context omitted.

Well, do you as a human have the whole codebase loaded in to your memory with the ability to mentally reason with it? No, you work on a small scope at a time.

You may work in a limited scope at a time, but you are aware how it fits into the larger scope, and more often than not you actually have to connect things across different scopes.

I work on projects with hundreds of thousands of files and tens of millions of lines. I am honestly clueless how most of it fits together and working here as a human feels often not too different (just a few steps of scale higher) than LLM-based coding like Cursor.
Post reply on HN