Instead of trying to get LLMs from zero to 100, which is impossible to do, you should concentrate on getting them from 75% to 95%.
The idea that someone who has no knowledge of the domain and no understanding of relational data modeling can chat “like to know really good um but don’t make any mistakes” with a GPU an uncover the mysteries of the universe is imbecilic.
But… someone what knows approximately what to do and sort of how to do it could work wonders — if we had LLMs trained on a corpus with specific rules.
I don’t know how to left join and what table I need to get the aggregate of sales in each region by date and price range, but I can describe it halfway and know how to check if each step is valid.
LLMs can do this. They’re trained on English, and they are able to weight definitive rules. But instead we throw a random text at a general purpose transformer.
Parsing a response of tokens into grammatical English is the most expensive computation (after the initial scraping and catalog). Instead of wasting all the cycles doing that against the sun total of GitHub, StackOverflow, Reddit, and Wikipedia, create a fuzzy match on a simplification a rigorous specification and train it on your data (just a few million tokens) to teach it that users have primary addresses and are associated to accounts that have regions and region X has roughly 10 times the sales volume of region y.
So someone intelligent in the matter with an understanding of logical rigor and a general idea of the data shape can actually become 10x more efficient, instead of trying to lift vibe coders to the level of Spakespearean monkeys, you could be turning mid-level devs into super analysts.