My AI skeptic friends are all nuts

At 0 temperature an LLM is a Map - a string input (key) will give you the same string output (value) every time. Hypothesis: there exists a key whose value is a complete, working, fully-tested application which meets your requirements 100% and fulfills your business need. This key is the smallest, most complete description of what your application does. It is written in natural language and represents a significantly compressed version of your application code.

My part-time obsession over the last few months has been trying to demonstrate this and come up with a method for finding these magic keys (I even tried to get the LLMs to search for me, lol). What I really want is to give the latest thinking models (200k input, 100k output) a 5-6 page design doc (4k words, 5k tokens) and have them produce a complete 5kloc (50k tokens) microservice, which would show a 10x compression. It's hard, but I haven't seen any reason to think it wouldn't work.

For better or worse, I think this will be close to what IC jobs will be like in few years. Fundamentally, our jobs are to try work with other functions to agree to some system that needs to exist, then we talk to the computers to actually implement this. If we switch kotlin+compiler for design doc+llm, it still going to be somewhat the same, but far more productive. Agents and such are somewhat of a stop-gap measure, you don't want people giving tasks to machines, you want to accurately describe some idea and then let the computers make it work. You can change your description and they can also figure out their own tasks to evolve the implementation.

> The time has come for llm users to put up or shut up - if it’s so great, stop telling us and show and use the code it generated on its own.

I’m open to that happening. I mean them showing me. I’m less open to the Nth “aww shucks, the very few doubters that are left at this point are about to get a rude awakening” FOMO concern trolling. I mean I guess it’s nice for me that you are so concerned about my well-being, soon to be suffering-being?

Now, AI can do a lot of things. Don’t get me wrong. It has probably written a million variations on the above sentiment.

> eventually get to a point where I have to re-write major pieces of the code to extract the agent from some major ditch it has driven into, leading to a failure to make forward progress.

As it stands AI can't even get out of Lt Surge's gym in Pokemon Red. When an AI manages to beat Lance I'll start to think about using it for writing my code :-)

You think that the only code that is valuable is code that is written by a professional SWE.

There are LOADS of people who need "a program" but aren't equipped to write code or hire an SWE that are empowered by this. And example: last week, I saw a PM vibe code several different applications to demo what might get built after it gets prioritized by SWEs

FWIW with proper MDC/ rules I’ve found LLM programming agents excellent at rust. There’s a lot of complex and tedious minutiae in rust that I know but forget to apply everywhere it’s helpful while a SOTA LLM agent does well, especially with proper rule guidance to remember to use it.

Generally though I find LLMs have a pretty rapidly diminishing return on what you can expect out of them. They’re like a 3-5 year senior programmer that has really learned their domain well, but doesn’t have the judgement of a principal engineer. You get to a point where you need to reach in and right things and really pay attention, and at that point the diminishing returns set it rapidly and you’re better off just doing the rest yourself. Refactors and stuff can be delegated but that’s about it.

I find this true regardless of the language. None the less, I’ve been able to improve my overall velocity dramatically completing several projects in the last few months in the span of one typically. If tooling improves I hope to continue that but I’m already getting close to the limit of how fast I can conceive of useful creative things.