Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vanna.ai: Chat with your SQL database

github.com

Post reply on HN
Print view View original

221–230 of 243 posts

Re: Vanna.ai: Chat with your SQL database

#221

Post by anon291 »

sagaro wrote: post #186

All these products that pitch about using AI to find insights from your data always end up looking pretty in demos and fall short in reality. This is not because the product is bad, but because there is enormous amount of nuance in DB/Tables that becomes difficult to manage. Most startups evolve too quickly and product teams generally tries to deliver by hacking some existing feature. Columns are added, some columns…

The AI needs to truly be 'listening' in in a passive way to all Slack messages, virtual meetings, code commits, etc and really be present whenever the 'team' is in order to get anything done.

Re: Vanna.ai: Chat with your SQL database

#223

Post by bongodongobob »

lysecret wrote: post #199
bongodongobob wrote: post #175

Earlier quoted context omitted.

I think a lot of people tried just asking GPT-3.5 to "Write me full stack web app no bugs please." when it first came out. When it failed to do that they threw up their hands and said "It's just a parrot." Then GPT4 came out, they tried the same thing and got the same results. I keep seeing comments regarding it not being helpful because "super big codebase, doesn't work, it doesn't know the functions and what they d…

I think the main reason people don't do something like this more often is that this turns you from a "coder" to a "manager". Your task now is to serialize the issue and to ask the right questions / keep everything moving along. I don't particularly mind because I cre more about building something than doing my craft but I can totally see how this will be different for many people.

Yeah that's exactly it. You have to build up the proper context first. Get all the documentation to be nice and coherent and it will happily write beautiful code.

Re: Vanna.ai: Chat with your SQL database

#224

Post by [deleted] »

lmeyerov wrote: post #212
eurekin wrote: post #210

Earlier quoted context omitted.

Speculating How about a chat interface, where you correct the result and provide more contextual information about those columns? Those chats could be later fed back to the model and ran a DPO optimisation on top

Agreed. Agent reasoning systems should learn based on past and future use, and both end users and maintainers should have power in how they work. So projects naturally progress on adding guard rails, heuristics, policies, customization, etc. Likewise, they first do it with simple hardcoding and then swapping in learning. As we have built out Louie.ai with these kinds of things, I've appreciated ChatGPT as its own inn…

[deleted]

Re: Vanna.ai: Chat with your SQL database

#225

Post by atomicvibe »

great tool and as someone mentioned there are quite a few AI products/libraries that accomplish this.

The main challenge for production settings

1) soon becomes optimizing the latency for each step and 2) dealing with complex queries at a reasonable cost (GPT-4 is bit expensive at scale)

We evaluated multiple options and settled on GPT-3Turbo in the short term. However we are not happy with the latency, especially if you create an agentic solution.

Re: Vanna.ai: Chat with your SQL database

#226

Post by atomicvibe »

ignoramous wrote: post #122
qiller wrote: post #98

We did something similar for our reporting service which is based duckdb. Overall it works great, though we've ran into a few things: * Even with low temperature, GPT-4 sometimes deviates from examples or schema. For example, sometimes it forgets to check one or another field... * Our service hosts generic data, but customers ask to generate reports using their domain language (give me top 10 colors... what's a color…

Curious, as we're looking to build / use a similar setup. > Debugging LLM prompts is just tricky... Customers can confuse the model pretty easily. Would a RAG like how Vanna.ai uses, help? > For example, sometimes it forgets to check one or another field Do prompting techniques like CoT improve the outcome? > So we need to teach customers to nudge the report generator a bit towards generic terms. Did you folks experi…

>Did you folks experiment with building an Agent-like interface that asks more questions before the LLM finally answers?

We built something similar to query DB. Created two versions, one of which was agent based that had a maker-checker style of generation. Basically one generates and the other checks its correctness and if objective has been acheived.

The accuracy improves in the agent driven framework, at the cost of latency.

Re: Vanna.ai: Chat with your SQL database

#227

Post by ukuina »

lysecret wrote: post #199
bongodongobob wrote: post #175

Earlier quoted context omitted.

I think a lot of people tried just asking GPT-3.5 to "Write me full stack web app no bugs please." when it first came out. When it failed to do that they threw up their hands and said "It's just a parrot." Then GPT4 came out, they tried the same thing and got the same results. I keep seeing comments regarding it not being helpful because "super big codebase, doesn't work, it doesn't know the functions and what they d…

I think the main reason people don't do something like this more often is that this turns you from a "coder" to a "manager". Your task now is to serialize the issue and to ask the right questions / keep everything moving along. I don't particularly mind because I cre more about building something than doing my craft but I can totally see how this will be different for many people.

Here's the philosophical conundrum: Do we prioritize learning new things (improve craft) or create value (build stuff)? The first used to lead to the second, but not as much anymore.

Re: Vanna.ai: Chat with your SQL database

#229

Post by eek2121 »

joshstrange wrote: post #204
sagaro wrote: post #186

All these products that pitch about using AI to find insights from your data always end up looking pretty in demos and fall short in reality. This is not because the product is bad, but because there is enormous amount of nuance in DB/Tables that becomes difficult to manage. Most startups evolve too quickly and product teams generally tries to deliver by hacking some existing feature. Columns are added, some columns…

I couldn’t agree more. I’ve hooked up things to my DB with AI in an attempt to “talk” to it but the results have been lackluster. Sure it’s impressive when it does get things right but I found myself spending a bunch of time adding to the prompt to explain how the data is organized. I’m not expecting any LLM to just understand it, heck another human would need the same rundown from me. Maybe it’s worth keeping this “…

Welcome to AI in general.

Billions wasted on a pointless endeavor.

10 years from now folks are going to be laughing at how billions of dollars and productivity was flushed down the drain to support Microsoft Word 2.0.

AI is a bubble. Do yourself a favor and short (or buy put options) the companies that only have "AI" for a business model.

Also short Intel, because Intel.

Re: Vanna.ai: Chat with your SQL database

#230

Post by eek2121 »

scoot wrote: post #202
lmeyerov wrote: post #194

Earlier quoted context omitted.

Our theory is we are having simultaneously a bit of a Google moment and a Tableau moment. There is à lot more discovery & work to pull it off, but the dam has been broken. It's been am exciting time to work through with our customers: * Google moment: AI can now watch and learn how you and your team do data. Around the time Google pagerank came around, the Yahoo-style search engines were highly curated, and the seman…

> Around the time Google pagerank came around, the Yahoo-style search engines were highly curated Hmmm, no. Altavista was the go-to search engine at the time (launched 1995), and was a crawler (i.e. not a curated catalog/directory) based search. Lycos predates that but had keyword rather than natural language search. Google didn't launch until 1998.

Hotbot was amazing back in the day. Google scored me post high school schoolin', however, so I won't complain...Shoot Google helped me make the local news back in the day!
Post reply on HN