I don't have much success with using the LLM to make changes to a big legacy codebase. Instead, I use the LLM to gripe about things I don't like in the code. Usually, it is a brilliant commiserator.
I think I have LLM burnout
21–30 of 391 posts
Re: I think I have LLM burnout
#22This is legitimately the reason I'm looking to leave programming. I got into programming because the problems of programming were interesting to me. But if the problems go from "figure out why this calculator is off by one in France" to "Get this LLM to stop spamming cutsey emojis", then maybe it's time for a career change.
(And I admit I'm salty that the "I don't give a shit about why the calculator doesn't work in France, I'm just here because they pay me to fix it" people were the ones vindicated by technological progress)
[flagged]
Re: I think I have LLM burnout
#23I do not have the burnout but I certainly operate similarly to the author. I continue to be unable to establish a workflow where allowing the LLM to generate code that I review is faster than writing the code myself. Literally the only two ways out of this dilemma is to blindly trust what was generated or to generate an uncharacteristically exhaustive suite of unit tests to validate every possible scenario. I just wr…
> generate an uncharacteristically exhaustive suite of unit tests to validate every possible scenario. This is what you want. You want comprehensive tests at every level, far more than is reasonable for a human to build or maintain, from unit, functional, to full end to end and beyond. Adversarial testing (both TDD-style "write tests to demonstrate this bug", and posthoc "prove this patch wrong with a new test") is t…
I see this get mentioned a lot but I still am skeptical that AI can generate tests we can trust more than any other code we know we cannot trust.
Yes tests are conceptually isolated and that helps, but I've personally seen unit tests get generated that are semantically incorrect - that is, they test the structure of the code (e.g. they can check function output types and values), but they can't know _why_ the unit tests need to be there, so the really really helpful tests never get generated. Not to mention the obvious issues with generated tests only testing is x = x, or needless redundant tests for the same thing, or them essentially testing basic features of the language.
Re: I think I have LLM burnout
#24> My main project right now is to establish a framework for large-scale, unsupervised code generation in our codebase Anyone else working on something like this or know of any projects attempting it?
An almost infinite supply of such modern-day alchemists.
Re: I think I have LLM burnout
#25I've started feeling slightly physically ill when I read Opus output for hours straight. This article rings very true for me. I've started complaining about it with my team; at least have a personal style guide in your agent rules that eliminates emdashes, the "it's not X, it's Y"s, the long lists of modifiers before the noun, using the word "land" to mean finish, etc. I hope this is just a phase of adolescent LLMs.
Re: I think I have LLM burnout
#26Earlier quoted context omitted.
> generate an uncharacteristically exhaustive suite of unit tests to validate every possible scenario. This is what you want. You want comprehensive tests at every level, far more than is reasonable for a human to build or maintain, from unit, functional, to full end to end and beyond. Adversarial testing (both TDD-style "write tests to demonstrate this bug", and posthoc "prove this patch wrong with a new test") is t…
I see this get mentioned a lot but I still am skeptical that AI can generate tests we can trust more than any other code we know we cannot trust. Yes tests are conceptually isolated and that helps, but I've personally seen unit tests get generated that are semantically incorrect - that is, they test the structure of the code (e.g. they can check function output types and values), but they can't know _why_ the unit te…
You have to iterate on the tests, review and validate them, just like any other code, and if you generate a whole project's tests all at once the quality is abysmal, of course. I've been using a lot of old school data-driven testing techniques, where the harness is just code I review, and the data itself is e.g. json files and drives the system.
I actually have a public (AGPL) example here: https://github.com/pgdogdev/pgdog/tree/main/integration/sql - pgdog is particularly testable since it is trying for complete transparency, so you have a perfect oracle in hand via base postgresql, but it demonstrates the concept at least.
Re: I think I have LLM burnout
#27I do not have the burnout but I certainly operate similarly to the author. I continue to be unable to establish a workflow where allowing the LLM to generate code that I review is faster than writing the code myself. Literally the only two ways out of this dilemma is to blindly trust what was generated or to generate an uncharacteristically exhaustive suite of unit tests to validate every possible scenario. I just wr…
> generate an uncharacteristically exhaustive suite of unit tests to validate every possible scenario. This is what you want. You want comprehensive tests at every level, far more than is reasonable for a human to build or maintain, from unit, functional, to full end to end and beyond. Adversarial testing (both TDD-style "write tests to demonstrate this bug", and posthoc "prove this patch wrong with a new test") is t…
100% this is what I've done. I sucked it up and adapted myself to the tool (agents) by having as many implicit guardrails (static typing, functional, no nulls, great linting) and then layering on explicit guardrails (TDD) on top. I also want my workflow to be portable because I don't really trust the frontier model providers.
It is different though. Basically a lot of what I do has changed over the last 2 years. I totally get that a lot of people won't want to adapt though.
Re: I think I have LLM burnout
#28Earlier quoted context omitted.
(And I admit I'm salty that the "I don't give a shit about why the calculator doesn't work in France, I'm just here because they pay me to fix it" people were the ones vindicated by technological progress)
[flagged]
In every industry, and we wonder why everything is being enshittified.
I'm not looking forward to using computers or technology over the next decade. There is a non-zero chance myself or a loved one is killed because of vibe coding.
Re: I think I have LLM burnout
#29Earlier quoted context omitted.
> generate an uncharacteristically exhaustive suite of unit tests to validate every possible scenario. This is what you want. You want comprehensive tests at every level, far more than is reasonable for a human to build or maintain, from unit, functional, to full end to end and beyond. Adversarial testing (both TDD-style "write tests to demonstrate this bug", and posthoc "prove this patch wrong with a new test") is t…
I used an LLM to build this https://github.com/dprkh/eventfs It has good test coverage, mostly unit tests but also a number of end-to-end tests. I also made the LLM build a benchmark, which you can find at the bottom of the readme. It is obviously slow, but I thought that it is good enough to work. When I tried to write a 1 GiB file, I found that it broke down, and after writing half the file, the speed went to under…
That's interesting because I would feed that benchmark back into the agent and loop over it, to see how much faster you could get it, and agents are really good at that kind of recursive optimization. And I would definitely add at least a simulated 1GiB write test, probably a real one honestly, if I was building something like that.
At least with agent-run tests I care about loop speed a lot, but I care about complete coverage more, so having the odd heavy weight full stack integration test is fine, I think.
Re: I think I have LLM burnout
#30I don't mind interacting with LLMs myself and find they increase my productivity a decent amount. I just can't stand dealing with other people's slop.
Getting sent IM responses that are copy pasted LLM nonsense. Getting a massive PR to review that was generated overnight and the author didn't read it first.