After months of coding with LLMs, I'm going back to using my brain

I delegate all kinds of deep analysis to the AI, but it's to create detailed plans with specific implementation steps and validation criteria, backed by data in reproducible reports (i.e. "generate a script to generate this json data and another to render this data"). Plans have a specific goal that is reflected in the report ("migrated total should be 100%"). It's still an iterative process, the generators and plans have to be refined as it misses edge cases, but that's plans in general, AI or no.

It takes a good hour or two to draw up the plans, but it's the kind of thing that would take me all day to do, possibly several as my ADHD brain rebels against the tedium. AI can do yeomans work when it just wings it, and sometimes I have just pointed at a task and did it in one shot, but they work best when they have detailed plans. Plus it's really satisfying to be able to point at the plan doc and literally just say "make it so".

So the end game for this is, to a certain extent, does the code quality matter if it does what it is needed?

In the past the quality mattered because maintenance and tech-debt was something that we had to spend time and resources to resolve and it would ultimately slow us down as a result.

But if we have LLMs do we even have "resources" any more? Should we even care if the quality is bad if it is only ever LLMs that touch the code? So long as it works, who cares?

I've heard this positioned in two different ways, from two different directions, but I think they both work as analogies to bring this home:

- do engineers care what machine code a compiler generates, so long as it works? (no, or at least very, very, very rarely does a human look at the machine code output)

- does a CEO care what code their engineers generate, so long as it works? (no)

Its a very very interesting inflection point.

The knee jerk reaction is "yes we care! of course we care about code quality!" but my intuition is that caring about code quality is based on the assumption that bad code = more human engineer time later (bugs, maintenance, refactoring etc).

If we can use a LLM to effectively get an unlimited number of engineer resources whenever we need them, does code quality matter provided it works? Instead of a team of say 5 engineers and having to pick what to prioritise etc, you can just click a button and get the equivalent of 500 engineers work on your feature for 15 minutes and churn out what you need and it works and everyone is happy, should we care about the quality of the code?

We're not there yet - I think the models we have today kinda work for smaller tasks but are still limited with fairly small context windows even for Gemini (I think we'll need at least a 20x-50x increase in context before any meaningfully complex code can be handled, not just ToDo or CRUD etc), but we'll get there one day (and probably sooner than we think)

What kind of guardrails did you give the agent? Like following SOLID, linting, 100% code coverage, templates, architectural documents before implementing, architectural rules, DRY cleanup cycles, code review guidelines (incl strict rules around consistency), review by another LLM etc?

> Some days I feel like we’re also being gaslit by LLM providers. Just look at any AI related subreddit, and you’ll see people having completely opposite experiences, with the same exact model, with the same prompt, on the same day. If you code with AI long enough you’ll be able to relate. One day it’s amazing, the next day it’s incredibly stupid.

> Are they throttling the GPUs? Are these tools just impossible to control? What the fuck is going on?

Money and dreams. As everyone knows there’s an obscene amount of money invested in these tools of course. The capitalist class is optimistic their money can finally do the work for them directly instead of having to hire workers.

But more than that, AI is something that’s been alluring to humans forever. I’m not talking about cyberpunk fantasies I’m talking about The Mechanical Turk, Automata in the Middle Ages, Talos[0]. The desire to create an artificial mind is, if not hardwired into us, at least culturally a strong driver for many. We’re at a point where the test of the computer age for determining if we built AI was so utterly destroyed it’s unclear how to go about judging what comes next.

The hype is understandable if you view if you step back and view it through that lens. Maybe we are at an inflection point and just a bit more scaling will bring us the singularity. Maybe we’ve seen a burst of progress that’ll mostly stall from a consumer perspective for another 5-10 years, like all of the major AI moments before.

If you want to use them effectively it’s the same as any tool. Understand what they are good at and where they flounder. Don’t give up your craft, use them to elevate it.

[0]: https://news.stanford.edu/stories/2019/02/ancient-myths-reve...

Not quite a source but it’s a fun read from 2019 about this.

Sadly, the employers (CEOs, managers etc.) don't think like that.