Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tuning GPT-3.5-turbo for natural language to SQL

medium.com

Post reply on HN
Print view View original

31–40 of 77 posts

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#31

Post by shireboy »

I think natural language to SQL may not be as great as it sounds in many real-world applications. You often have permissions, tenants, rules about who-can-see-what. I love the idea of letting users use natural language to query and possibly even bulk-update data. But if the app has rules like those, this would just be a SQL injection vulnerability. You could possibly limit it - just allow query over certain views or something - but the risk of the GPT crafting a dangerous query is still high.

What would be more useful IMO is natural language to OData, GraphQL, and OpenAPI/Swagger. Then you could let users do ad-hoc query but only against data they are allowed. I did a PoC using GPT3 to query OData and it was pretty fun, but did occasionally get the query wrong. I also ran into the context window issue. It would get lost when fed larger OData schema.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#32

Post by bob1029 »

sillysaurusx wrote: post #27
bob1029 wrote: post #25

We've been chasing this rabbit since the beginning. It currently seems to be uncatchable for the use cases that would be most valuable to us - writing complex queries we've never seen before. Our use case seems to confound any notion of training or fine-tuning, since the cases we need the most help with are also the ones we have the fewest examples of. Instead of going for generative, few-shot models, I am starting t…

I think you've fallen into the trap of "AIs don't generalize, they memorize." But they do in fact generalize. The reason ChatGPT is so valuable is precisely because it can help out with situations that have never been seen before, not because it merely unlocks old preexisting knowledge. The fella who saved their dog with ChatGPT comes to mind. https://nypost.com/2023/03/27/chatgpt-saved-my-dogs-life-aft...

> I think you've fallen into the trap of "AIs don't generalize, they memorize."

Binary classifiers don't generalize?

Just because my output is not generative does not mean we are cannot learn / generalize elsewhere. Think of it as a 2-stage process.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#33

Post by bob1029 »

Nick_Molter wrote: post #30
bob1029 wrote: post #25

We've been chasing this rabbit since the beginning. It currently seems to be uncatchable for the use cases that would be most valuable to us - writing complex queries we've never seen before. Our use case seems to confound any notion of training or fine-tuning, since the cases we need the most help with are also the ones we have the fewest examples of. Instead of going for generative, few-shot models, I am starting t…

Curious to learn more about your use case. If fine-tuning is only ineffective for your most complex queries (and presumably those are less frequent as well, since you mentioned you have few examples), then couldn't you use fine-tuning to handle the simpler queries (presumably the lion's share) and thus free up excess man hours to focus on the more complex queries? Is there any benefit to AI being able to answer 90% o…

These tools are already fantastic at our 80% average case, even without fine tuning. We are seeing some value, but the real pain is in that other 20%.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#34

Post by codeulike »

I often do complicated reports in SQL or complicated transformations in SQL for system migrations. To really write a query and 'get it right' you usually need insider knowledge that you can't glean from the column names.

I see in their training set they've got comments about columns too. e.g.

    "Country" TEXT NOT NULL, - country where the singer born
But thats still not enough.

You also need a bunch of information about the real business that the data is describing. And you also need to analyse the whole database - is that field actually used? What are the common values for this picklist? What does that status actually mean in terms of business? If there are two of those rows that match the join but I want to avoid duplicates, which one do I take? - the newest, or the one with a certain status? etc.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#35

Post by ainesh93 »

h1fra wrote: post #13
beebmam wrote: post #4

Earlier quoted context omitted.

Really? I sure can. I can even imagine a single API request that triggers a job that costs $100,000. These APIs exist. They're not for individuals.

I mean yes, I have managed BigQuery jobs that cost thousand of dollars per run but the actual call is not the thing that is expensive and I'm in control of what it costs me. It's not exactly the same scenario imo

I think the focus here isn't necessarily on compute cost. When companies hire data scientists or analysts, they're niche-skilled and expensive. If those people spend 50-60% of their time courting ad-hoc questions from various people in the org, the cost of that employee's time (and the money spent on them doing menial tasks that are a waste of their skillset) is the biggest factor.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#36

Post by Zircom »

sillysaurusx wrote: post #27
bob1029 wrote: post #25

We've been chasing this rabbit since the beginning. It currently seems to be uncatchable for the use cases that would be most valuable to us - writing complex queries we've never seen before. Our use case seems to confound any notion of training or fine-tuning, since the cases we need the most help with are also the ones we have the fewest examples of. Instead of going for generative, few-shot models, I am starting t…

I think you've fallen into the trap of "AIs don't generalize, they memorize." But they do in fact generalize. The reason ChatGPT is so valuable is precisely because it can help out with situations that have never been seen before, not because it merely unlocks old preexisting knowledge. The fella who saved their dog with ChatGPT comes to mind. https://nypost.com/2023/03/27/chatgpt-saved-my-dogs-life-aft...

Did you read the article you posted?

-ChatGPT initially gives the same diagnosis the vets did, Babesiosis

-It notes that Babesiosis may have either been a misdiagnosis or there may be a secondary condition/infection causing the remaining symptoms after the Babesiosis treatment didn't resolve all of them

-It suggests such a hypothetical secondary condition could be IMHA, which the article notes is an extremely common complication of Babesiosis with this specific dog breed

-A quick Google search brings up a fair amount of literature about the association between Babesiosis and IMHA

So in fact this is the opposite of a never before seen situation, ChatGPT was just regurgitating common comorbidities of Babesiosis and the vets in question are terrible at their job.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#37

Post by MuffinFlavored »

bob1029 wrote: post #25

We've been chasing this rabbit since the beginning. It currently seems to be uncatchable for the use cases that would be most valuable to us - writing complex queries we've never seen before. Our use case seems to confound any notion of training or fine-tuning, since the cases we need the most help with are also the ones we have the fewest examples of. Instead of going for generative, few-shot models, I am starting t…

> Binary classification into deterministic query building.

I have no clue what this means.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#38

Post by capableweb »

codeulike wrote: post #34

I often do complicated reports in SQL or complicated transformations in SQL for system migrations. To really write a query and 'get it right' you usually need insider knowledge that you can't glean from the column names. I see in their training set they've got comments about columns too. e.g. "Country" TEXT NOT NULL, - country where the singer born But thats still not enough. You also need a bunch of information abou…

Why dont you collect a sample of the data so it knows what to expect then? Include 10 handpicked rows in your prompt and it should be able to pick up the patterns. If you find it doesn't handle something, append that row and restart.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#39

Post by sillysaurusx »

Zircom wrote: post #36
sillysaurusx wrote: post #27

Earlier quoted context omitted.

I think you've fallen into the trap of "AIs don't generalize, they memorize." But they do in fact generalize. The reason ChatGPT is so valuable is precisely because it can help out with situations that have never been seen before, not because it merely unlocks old preexisting knowledge. The fella who saved their dog with ChatGPT comes to mind. https://nypost.com/2023/03/27/chatgpt-saved-my-dogs-life-aft...

Did you read the article you posted? -ChatGPT initially gives the same diagnosis the vets did, Babesiosis -It notes that Babesiosis may have either been a misdiagnosis or there may be a secondary condition/infection causing the remaining symptoms after the Babesiosis treatment didn't resolve all of them -It suggests such a hypothetical secondary condition could be IMHA, which the article notes is an extremely common…

Are you suggesting that it was commonplace for machine learning models to be able to extrapolate medical case reports into an actual diagnosis? Even being able to read and understand a case report is a minor miracle in extrapolation, and it’s interesting how far the goal posts move.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#40

Post by ainesh93 »

codeulike wrote: post #34

I often do complicated reports in SQL or complicated transformations in SQL for system migrations. To really write a query and 'get it right' you usually need insider knowledge that you can't glean from the column names. I see in their training set they've got comments about columns too. e.g. "Country" TEXT NOT NULL, - country where the singer born But thats still not enough. You also need a bunch of information abou…

> You also need a bunch of information about the real business that the data is describing.

While the article focuses on finetuning GPT-3.5-turbo, how you use the text-to-SQL engine within the architecture of your overall solution is for you to decide. Providing this business context from vectorized context stores in the actual prompt would be a step in the right direction.

Post reply on HN