An open source DuckDB text to SQL LLM
motherduck.com
An open source DuckDB text to SQL LLM
1–10 of 31 posts
Re: An open source DuckDB text to SQL LLM
#2looks great, most text-to-sql attempts i’ve tried fall short, hoping this is different
Re: An open source DuckDB text to SQL LLM
#3looks great, most text-to-sql attempts i’ve tried fall short, hoping this is different
I do this a lot with GPT4 and I can't recall it ever generating an invalid query, and I did have some complicated ones.
Re: An open source DuckDB text to SQL LLM
#4I see so many business leaders touting the promise of LLMs allowing business to "talk" to their data. The promise does sound enticing, but it's actually kind of hard to get working in practice.
A lot of our databases at work have columns with custom types and enums, and getting the LLM (Llama2) to write SQL queries to robustly answer natural language questions about the data is tough. It requires a lot of instruction prompting, context, and question-SQL examples (few-shot learning), and it still fails in unexpected ways. It's a tough ask for people to use a tool like this if they can't trust the results all the time. It's also a bit infeasible to scale this to tens or hundreds of tables across our data warehouse.
It's great that a lot of people are trying to crack this problem, I'm curious to try this model out. I'd also love to see if other people have tried solving this problem and made any headway.
Re: An open source DuckDB text to SQL LLM
#5looks great, most text-to-sql attempts i’ve tried fall short, hoping this is different
I do this a lot with GPT4 and I can't recall it ever generating an invalid query, and I did have some complicated ones.
The problem isn't the SQL generation per se, it's the mapping between natural language and the data that columns actually hold. If you have a clean schema with well named fields, LLMs are very good at generating valid SQL but if you have an old schema that has been patched and modified for years by a dozen DBAs each with their own naming quirks, short hands, and hacks, it can barely string a valid query together.
Re: An open source DuckDB text to SQL LLM
#6I see so many business leaders touting the promise of LLMs allowing business to "talk" to their data. The promise does sound enticing, but it's actually kind of hard to get working in practice. A lot of our databases at work have columns with custom types and enums, and getting the LLM (Llama2) to write SQL queries to robustly answer natural language questions about the data is tough. It requires a lot of instruction…
I've worked on a similar problem and we have pretty much the same issues as you. An idea that makes things better is having an intermediate representation that exposes the key tables of your dwh and gets compiled to SQL. This allows you to have somewhat better security (even if the model outputs garbage it can't do too much damage because nothing except exactly what you want is even representable), and somewhat better explainability (you can't guarantee the model will get it right, but you can see what SQL gets executed rather than "magic 8-ball says no").
But as you say custom types and encoded domain knowledge is extremely tough and as a result it's very tough to "transfer" the system to different databases.
Re: An open source DuckDB text to SQL LLM
#7Co-founder and Head of Produck at MotherDuck here, happy to answer any questions or go nag the amazing engineers [0] who worked on this :)
Re: An open source DuckDB text to SQL LLM
#8I see so many business leaders touting the promise of LLMs allowing business to "talk" to their data. The promise does sound enticing, but it's actually kind of hard to get working in practice. A lot of our databases at work have columns with custom types and enums, and getting the LLM (Llama2) to write SQL queries to robustly answer natural language questions about the data is tough. It requires a lot of instruction…
My biggest concern is that this will allow people to type a question and get a number back from the database, without being able to tell if the query is right or if the LLM just made up something.
It can work to support business analysts to crank out more reporters, but I wouldn’t roll it out to all my staff.
Re: An open source DuckDB text to SQL LLM
#9Co-founder and Head of Produck at MotherDuck here, happy to answer any questions or go nag the amazing engineers [0] who worked on this :) [0] https://news.ycombinator.com/user?id=tdoehmen
Congratulations! The FixIt seems a pragmatic feature.
Any other ideas about where you plan to utilize LLM? E.g.
- migrate from different SQL dialects (e.g. date functions)
- make my queries nicer (e.g., rewrite them in shorter form or use better variable names)
- warn about potential bugs (e.g. SQL fanout)
Re: An open source DuckDB text to SQL LLM
#10I see so many business leaders touting the promise of LLMs allowing business to "talk" to their data. The promise does sound enticing, but it's actually kind of hard to get working in practice. A lot of our databases at work have columns with custom types and enums, and getting the LLM (Llama2) to write SQL queries to robustly answer natural language questions about the data is tough. It requires a lot of instruction…
My biggest concern is that this will allow people to type a question and get a number back from the database, without being able to tell if the query is right or if the LLM just made up something. It can work to support business analysts to crank out more reporters, but I wouldn’t roll it out to all my staff.
I've actually done exactly what @qsort suggested and outputted the intermediate SQL query and raw data generated by that query when generating the response back to the user. That definitely helps in establishing more trust with the customer since they can verify the response. My approach right now is to just be honest with our customers in the capabilities of the tool, acknowledge its shortcomings, and keep iterating over time to make it better and better. That's what the team in charge of our company-wide custom LLM has done and it's gained a surprising amount of traction and trust over the last few months.