We did this at bit.io and people loved it - there a bunch of articles we wrote on what we found during our work: https://innerjoin.bit.io/
We’ve since shut down (acquired by databricks) but happy to answer what I can.
Vanna.ai: Chat with your SQL database
121–130 of 243 posts
Re: Vanna.ai: Chat with your SQL database
#122We did something similar for our reporting service which is based duckdb. Overall it works great, though we've ran into a few things: * Even with low temperature, GPT-4 sometimes deviates from examples or schema. For example, sometimes it forgets to check one or another field... * Our service hosts generic data, but customers ask to generate reports using their domain language (give me top 10 colors... what's a color…
Curious, as we're looking to build / use a similar setup.
> Debugging LLM prompts is just tricky... Customers can confuse the model pretty easily.
Would a RAG like how Vanna.ai uses, help?
> For example, sometimes it forgets to check one or another field
Do prompting techniques like CoT improve the outcome?
> So we need to teach customers to nudge the report generator a bit towards generic terms.
Did you folks experiment with building an Agent-like interface that asks more questions before the LLM finally answers?
Re: Vanna.ai: Chat with your SQL database
#123How about instead of making AI wrappers to over 50 years old SQL, we’d make a database query language that’s easier to read an write?
In general, if something has been around for a very long time and nobody apparently seems to have thought to improve it, then odds are the reason is it's pretty good and genuinely hard to improve on.
> pretty good and genuinely hard to improve on
I think that might be a bit more positive than I would be. Broadly speaking, I think you could say that the downsides of the legacy technology in question aren’t larger than the collective switching costs.
But I’d definitely agree that when something has been around that long, it’s prob not all bad.
Re: Vanna.ai: Chat with your SQL database
#124While I recognize the efforts in developing natural language to SQL translation systems, I remain skeptical. The core of my concern lies in the inherent nature of natural language and these models, which are approximative and lack precision. SQL databases, on the other hand, are built to handle precise, accurate information in most cases. Introducing an approximative layer, such as a language model, into a system tha…
The key here is to always have some structured intermediate layer that can allow people to evaluate what it is the system thinks they're asking, short of the full complexity of SQL. You'll need something like this for disambiguation anyway - are "Jon Favreau movies" movies starring or directed by Jon Favreau? Or do you mean the other Jon Favreau? Don't one-shot anything important end to end from language. Use language to guide users towards unambiguous and easy to reason about compositions of smaller abstractions.
Re: Vanna.ai: Chat with your SQL database
#125Earlier quoted context omitted.
A translation engine between natural language and SQL means that everyone can communicate with a SQL database now. That's huge. Soon also DB people will use it to get the response for complex questions and queries. It's just more natural and way faster. With technology like this, there is little reason to even know SQL anymore as the average developer. Just like today, the average developer doesn't know how databases…
What’s funny is SQL was supposed to be the natural language everyone uses to communicate with. Few bothered to learn it.
"Few"? I guess I must be one of the olds.
Re: Vanna.ai: Chat with your SQL database
#126We did this at bit.io and people loved it - there a bunch of articles we wrote on what we found during our work: https://innerjoin.bit.io/ We’ve since shut down (acquired by databricks) but happy to answer what I can.
Super interesting! Why did you decide to shut down?
Re: Vanna.ai: Chat with your SQL database
#127I have seen good results from just describing the schema to ChatGPT-4 and then asking it to translate English to SQL. Does this work significantly better?
That's basically what happens but the power is that in Python, you can do this:
sql = vn.generate_sql(question=...)
Which means that now the SQL can be executed and you can get the table, chart, etc in any interface.
Flask: https://github.com/vanna-ai/vanna-flask
Streamlit: https://github.com/vanna-ai/vanna-streamlit
Re: Vanna.ai: Chat with your SQL database
#128I'm curious about how this performs with more complex queries, like joins across five tables. Also, does the training phase actually involve writing SELECT queries by hand? In the age of ORMs and so on, many people have probably forgotten how to write raw SQL queries.
> In the age of ORMs and so on, many people have probably forgotten how to write raw SQL queries. I’ve heard this general sentiment repeated quite a lot - mostly by people that don’t use ORMs. In my experience pretty quickly you reach the limits of even the best ORMs and need to write some queries by hand. And these tend to be the relatively complicated queries. You need to know about all of the different join types,…
ORM abuse are absolutely rife in small-scale/volume build industries i.e. web agencies, outsourced crews
8/10 projects I look into don't have any indexes set up.
Use of ORMs with little thought into lazily loaded relations lead to 100s of queries being done per request.
It's pretty mad. Do not underestimate the propensity of a developer to stick to the only tool they know how to use. Unfortunately ORMs like Eloquent make it way too easy.
Re: Vanna.ai: Chat with your SQL database
#129The docs don't really work on mobile, the side navbar takes up half the screen and doesn't seem closeable
Re: Vanna.ai: Chat with your SQL database
#130Thanks so much for making this and making it under MIT to boot.
I’ve been thinking about how to do this for about 6 months now and just started working on a demo for querying just one table // JSON column today.
I would feel a lot more comfortable with putting this (and/or my demo) into production if the database had been set up with a schema+db user per account rather than every tenant sharing just the one set of tables.