How to generate SQL statements with ChatGPT

Writing tweaks to complex statements is tedious and I’m lazy? I often just feed in existing queries and scripts and say “use the below as a template, but do X,Y, and Z instead.” GPT-4 does this correctly the majority of the time and for all but a handful of circumstances, can typically be coaxed into the correct thing with a couple of back-and-forths.

I really don't see how english is a better query language than SQL. Yes, SQL has a bunch of sharp corners you have to learn, but plain english can be very ambiguous. Creating an interface that that is simple at low query complexity is easy. A good interface is the one that still makes sense when queries get complicated.

And to be fair, even SQL is not that good at that.

It sounds like your concern is that an LLM is not capable of understanding an english explanation of the query and producing the correct query. I personally have not had many issues here, and the time saved by tweaking a generated query is larger than writing the query myself from scratch. And LLMs will only improve so that these tweaks require less time.

I've spent a lot of time on this problem space for our business. My current conclusions are as follows:

Writing the actual SQL query is not the hard part for us. The hardest part is exhaustively expressing our intention in any language at all, followed closely by its apprentice - the incomplete requirements monster.

I've found that if I can coach a team member into giving me a proper natural language statement that we would use as an ideal training example for a SQL bot, then they are probably able to just go ahead and write the final query by hand. Overall, it would likely take less time & cognitive overhead.

Don't get me wrong - I see the automation potential too. I've used it for lots of really crappy bulk refactors and it's great. But, it absolutely cannot think for you and magic-in the requirements that your customer ultimately left on the floor.

Generative SQL output is the opposite of what you want when you are at the frontier of understanding and incomplete requirements. For us, this is kind of where the biggest use case would have come from. We don't need much help with the boring stuff. We need a really smart asshole to call us out when there appears to be a logical gap somewhere. ChatGPT almost seemed to be this @ launch but I think its been RLHF'd into a potato by now. I stopped using it for anything serious around late May.

This is solvable by augmenting the database schema with comments.

When you integrate your database with an LLM, you'll notice the LLM will produce flawed queries based on wrinkles in your database schema. This is because the LLM relies on conventional understanding of how the schema is probably tied together. When you see the flawed queries, you augment the schema with a comment that explains why the schema has a wrinkle. The LLM takes that into consideration and the resulting queries are improved.

A concrete example[1]: I found that when querying the Sakila movie rental database, the generated query would frequently attempt to join the `rental` table to the `film` table through a nonexistent `film_id` column on the rental table. By adding the linked comment, the LLM stopped doing that.

1. https://github.com/amoffat/HeimdaLLM/blob/dev/notebooks/saki...