I'm curious to see his mise tasks. He lists a few of them near the end but I'm not sure what his LLM CLI is. Is that an actual tool or is he using it as a placeholder for "insert your LLM CLI tool here"?
My LLM codegen workflow
81–90 of 168 posts
Re: My LLM codegen workflow
#82I'm curious to see his mise tasks. He lists a few of them near the end but I'm not sure what his LLM CLI is. Is that an actual tool or is he using it as a placeholder for "insert your LLM CLI tool here"?
https://github.com/simonw/llm
It is linked to in the article - a brilliant utility from Simon.
Re: My LLM codegen workflow
#83Earlier quoted context omitted.
fwiw it doesn't/that was a joke. In some cases the LLM will suggest running a command that doesn't terminate (eg npm run dev to run a webserver), so it'd get stuck running that command just waiting for user input.
Surely you can just ask the ai to make sure the code will terminate before it runs it? /s
You jest, but given something that's in it's training data, ChatGPT's able to say the code won't terminate (eg a code to give all the Fibonacci numbers), so there are subsets of the halting problem it can solve for.
Re: My LLM codegen workflow
#84Earlier quoted context omitted.
Aider + AI generated maps and user guides for internal modules has worked well for me. Just today I did my own version of a script that uses Gemini 2 Flash (1M context window) to generate maps of each module in my codebase, i.e. a short one or two sentence description of what's in every file. Aider's repo maps don't work well for me, so I disable them, and I think this will work better. I also have a scratchpad file…
LLMs forcing us to improve our documentation habits. Seriously though, many languages allow API doc generation out of comments. Maybe these docs can just be flattened into a file.
Indexing automatically generated API docs using Cursor seems to work very well. I also index any guides/mdbooks libraries have available, depending on whether I’m trying to implement something new or modifying existing code.
Re: My LLM codegen workflow
#85That lonely/downtime section at the end is a giant red flag for me.
It looks like the sort of nonproductive yak-shaving you do when you're stuck or avoiding an unpleasant task--coasting, fooling around incrementally with your LLM because your project's fucked and you psychologically need some sense of progress.
The opposite of this is burnout--one of the things they don't tell you about successful projects with good tools is they induce much more burnout than doomed projects. There's a sort of Amdahl's Law in effect, where all the tooling just gives you more time to focus on the actual fundamentals of the product/project/problem you’re trying to address, which is stressful and mentally taxing even when it works.
Fucking around with LLM coding tools, otoh, is very fun, and like constantly clean-rebuilding your whole (doomed) project, gives you both some downtime and a sense of forward momentum--look how much the computer is chugging!
The reality testing to see if the tool is really helping is to sit down with a concrete goal and a (near) hard deadline. Every time I've tried to use an LLM under these conditions it just fails catastrophically--I don't just get stuck, I realize how basically every implicit decision embedded in the LLM output has an unacceptably high likelihood of being wrong, and I have an amount of debug cycles ahead of me exceeding the time to throw it all away and do it without the LLM by, like, an order of magnitude.
I'm not an LLM-coding hater and I've been doing AI stuff that's worked for decades, but current offerings I've tried aren't even close to productive compared to searching for code that already exists on the web.
Re: My LLM codegen workflow
#86Question to folks with good workflows: Are you using tools like DSPy to generate prompts? Any other tools/tips about managing prompts?
Re: My LLM codegen workflow
#87Given a 3 648 318 tokens repository (number from Repomix), I'm not sure what would be the cost of using a leading LLM to analyse it and ask improvements. Isn't the input token number way more limited than that ? This is part is unclear to me in the "non-Greenfield" part of the article. Iterating with aider on very limited scopes is easy, I've used it often. But what about understanding a whole repository and act on i…
Well, do you as a human have the whole codebase loaded in to your memory with the ability to mentally reason with it? No, you work on a small scope at a time.
On larger codebases, I use tools heavily instead of just winging it. If the LLM can't either orchestrate those tools or apply its own whole-program analysis it quite impossible for it to do anything useful.
Re: My LLM codegen workflow
#88Earlier quoted context omitted.
It's likely the end of a lot of abstractions that made programming easier. At some point, specialized code-gen transformer models should get really good at just spitting out the lowest level code required to perform the job.
This. Future programming language designers are then answering questions like: "How low-level can this language be while considering generally available models and hardware available can only generate so many tokens per second?", "Do we have the language models generate binary code directly, or is it still more efficient time-wise to generate higher level code and use a compiler?" "Do we ship this language with both…
I’m excited for my new woodworking career if this ever becomes a reality. LLMs are truly sucking the art out of everything.
Re: My LLM codegen workflow
#89That lonely/downtime section at the end is a giant red flag for me. It looks like the sort of nonproductive yak-shaving you do when you're stuck or avoiding an unpleasant task--coasting, fooling around incrementally with your LLM because your project's fucked and you psychologically need some sense of progress. The opposite of this is burnout--one of the things they don't tell you about successful projects with good…
It sounds like LLMs are the new futzing with emacs configuration.
Re: My LLM codegen workflow
#90Absolutely LLMs are great for greenfield projects. They can get you to a prototype for a new idea faster than any tool yet invented. Where they start to break down, I find, is when you ask it to make changes/refactors to existing code and mature projects. They usually lack context, so they doesn't hesitate to introduce lots of extra complexity, add frameworks you don't need, and in general just make the situation wor…
100% agree.
My experience to date across the major LLMs is that they are quick to leap to complex solutions, and I find that the code often is much harder to maintain than if I were to do it myself.
But complex code is only part of the problem. Another huge problem I see is the rapid accumulation of technical debt. LLMs will confidently generate massive amounts of code with abstractions and design patterns that may be a good fit in isolation, but are absolutely the wrong pattern for problem you're trying to solve or the system you're trying to build. You run into the "existing code pattern" problem that Sandi Metz talked about in her fantastic 2014 RailsConf talk, "All the little things" [0]:
> "We have a bargain to follow the pattern, and if the pattern is a good one then the code gets better. If the pattern is a bad one, then we exacerbate the problem."
Rapidly generating massive amounts of code with the wrong abstractions and design patterns is insidious because it feels like incredible productivity. You see it all the time in posts on e.g. Twitter or LinkedIn. People gushing about how quickly they are shipping products with minimal to zero other humans involved. But there is no shortcut to understanding or maintainability if you care about building sustainable software for the medium to long-term.
EDIT: Forgot to add link