AI doesn't generate working products, that's still your job

In my LLM coding experience, I’ve had this happen numerous times, for instance when building a card game.

The LLM does a great job for a while, but adding special event cards becomes impossible because it imported a standard 52 card deck library instead of having a flexible data model with card objects.

I suppose this is knowable and you could tell an LLM to do this from the beginning. It's just that when you view making a game (or any piece of software) as just a throwaway thing and don't start with deep thought about the implementation as an experienced engineer might, you don't think about this.

I think part of the issue with AI-generated writing, code, or anything else, is that it robs us of the thinking that used to be baked into the process of making stuff.

For my game, if I had to build it from scratch, I surely would've sketched out the kind of data model it requires to not waste my time.

I do think some of this stuff is overblown though. It's true that LLMs don't create production-grade stuff, but not all software needs to be scalable, fast, and easily maintainable.

This might be required for infrastructure or apps designed to (hopefully eventually) be used by millions.

But it's not true for a minor utility, like if I want to build a meal-planning app for my family. I don't care if the app is the fastest it can be or if the data model could eventually support a feature to configure dietary preferences and allergies or to plan the meals for Google's tens of thousands of employees.

There was a viral article a few years ago around how software could be like a home-cooked meal. And AI enables exactly that. A home-cooked meal doesn't require culinary perfection, but to feed the family and, maybe more importantly, be a gift of labor to the other person.

The experiment was: given existing Experience section of Senior Software Engineer CV (manually and carefully written), write 2-4 lines of high pitch About [me].

gpt-sol-xhigh just could not do it, making a complete AI slop mess.

It became substantially better when I asked to get a sample of real Senior/Principal engineers CVs, that it would be able to connect with meaningful online presence/contributions of their authors, and draw ideas from that.

Nevertheless, the remaining issues were critical, with their classes spanning:

- word for word repetition;

- tautology (phrases mapping to the same semantic entity);

- category mistakes (combining apples with oranges);

- faulty composition of generalised and concrete terms.

Before throwing it away I decided to give it a try and asked for strict prompt following, setting low logical errors threshold, and eventually providing a formal proof that it complied.

It took spaCy, doing NER and dependency parsing; then I suggested adding stanza for constituency parsing.

After slapping together the artefacts of analysis and thinking a bit, it produced great phrase (to my taste).

-

Then when I asked to apply the framework and improve bullet points in some experience block, it did 2 of 4 well, then miserably broke down; I’m not sure if it’s harness issue (codex) or fundamental model restriction.

-

So no, without substantial investment in steering, SOTA AI doesn’t perform even remotely close to a human, in complex reasoning.