Live data from Hacker News

Vanna.ai: Chat with your SQL database

github.com

71–80 of 243 posts

Re: Vanna.ai: Chat with your SQL database

#71
post #28

Is the architecture they use in this diagram currently the best way to train LLMs in general on custom data sets? https://raw.githubusercontent.com/vanna-ai/vanna/main/img/va... That is, store your trained custom data in vector db and then use RAG to retrieve relevant content and inject that into the prompt of the LLM the user is querying with? As opposed to fine tuning or other methods?

All the podcasts I've been listening to recommend RAG over fine-tuning. My intuition is that having the relevant knowledge in the context rather than the weights brings it closer to the outputs, thereby making it much more likely to provide accurate information and avoid hallucinations/confabulations.

> All the podcasts I've been listening to recommend RAG over fine-tuning

I'm always suspicious that is just because RAG is so much more accessible (both compute wise and in terms of expertise required). There's far more profit in selling something accessible to the masses to a lot of people than something only a niche group of users can do.

I think most people who do actual fine tuning would still probably then use RAG afterwards ...

Re: Vanna.ai: Chat with your SQL database

#72

I have been keeping track of a few products like these including some that are YC backed. Interesting space as I am looking for a solution myself: - Minds DB (YC W20) https://github.com/mindsdb/mindsdb - Buster (YC W24) https://buster.so - DB Pilot https://dbpilot.io and now this one

I don’t fully understand the use of business case after reading the documentation. Is it really a time save?

Re: Vanna.ai: Chat with your SQL database

#74
post #72

I have been keeping track of a few products like these including some that are YC backed. Interesting space as I am looking for a solution myself: - Minds DB (YC W20) https://github.com/mindsdb/mindsdb - Buster (YC W24) https://buster.so - DB Pilot https://dbpilot.io and now this one

I don’t fully understand the use of business case after reading the documentation. Is it really a time save?

Allow people that don't know SQL to query a database.

Re: Vanna.ai: Chat with your SQL database

#76
I'm curious to see if people have tried this out with their datasets and seen success? I've been using similar techniques at work to build a bot that allows employees internally to talk to our structured datasets (a couple MySQL tables). It works kind of ok in practice, but there are a few challenges:

1. We have many enums and data types specific to our business that will never be in these foundation models. Those have to be manually defined and fed into the prompt as context also (i.e. the equivalent of adding documentation in Vanna.ai).

2. People can ask many kinds of questions that are time-related like 'how much demand was there in the past year?'. If you store your data in quarters, how would you prompt engineer the model to take into account the current time AND recognize it's the last 4 quarters? This has typically broken for me.

3. It took a LOT of sample and diverse example SQL queries in order for it to generate the right SQL queries for a set of plausible user questions (15-20 SQL queries for a single MySQL table). Given that users can ask anything, it has to be extremely robust. Requiring this much context for just a single table means it's difficult to scale to tens or hundreds of tables. I'm wondering if there's a more efficient way of doing this?

4. I've been using the Llama2 70B Gen model, but curious to know if other models work significantly better than this one in generating SQL queries?

Re: Vanna.ai: Chat with your SQL database

#77
post #34

Sorry, maybe I'm just too tired to see it, but how much control do you have over the SQL query that is generated by the AI? Is there a risk that it could access unwanted portions or, worse, delete parts of your data? (the AI equivalent of Bobby Tables, so to speak)

Guessing its intended use case is business analytic queries without write permissions —- particularly for non-programmers. I don’t think it’d be advisable to use something like this for app logic

Re: Vanna.ai: Chat with your SQL database

#78
I've already done this with GPT-4.

It goes something like this:

Here's the table structure from MySQL cli `SHOW TABLE` statements for my tables I want to query.

Now given those tables, give me a query to show me my cart abandonment rate (or, some other business metric I want to know).

Seems to work pretty well.

Post reply on HN