Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tuning GPT-3.5-turbo for natural language to SQL

medium.com

Post reply on HN
Print view View original

51–60 of 77 posts

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#51

Post by riwsky »

PheonixPharts wrote: post #49
sillysaurusx wrote: post #27

Earlier quoted context omitted.

I think you've fallen into the trap of "AIs don't generalize, they memorize." But they do in fact generalize. The reason ChatGPT is so valuable is precisely because it can help out with situations that have never been seen before, not because it merely unlocks old preexisting knowledge. The fella who saved their dog with ChatGPT comes to mind. https://nypost.com/2023/03/27/chatgpt-saved-my-dogs-life-aft...

I think you have fallen into the trap of mistaking interpolation for generalization . Working with these models every day, it's clear that they can certainly interpolate between points in latent space and generate sensible answers to unseen questions, but it's pretty clear that they don't generalize . I've seen far to many examples of models failing to display any sense of generalization to believe otherwise. That's…

I think we’ve all fallen into the trap of mistaking pithiness for evidence

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#52

Post by famouswaffles »

PheonixPharts wrote: post #49
sillysaurusx wrote: post #27

Earlier quoted context omitted.

I think you've fallen into the trap of "AIs don't generalize, they memorize." But they do in fact generalize. The reason ChatGPT is so valuable is precisely because it can help out with situations that have never been seen before, not because it merely unlocks old preexisting knowledge. The fella who saved their dog with ChatGPT comes to mind. https://nypost.com/2023/03/27/chatgpt-saved-my-dogs-life-aft...

I think you have fallen into the trap of mistaking interpolation for generalization . Working with these models every day, it's clear that they can certainly interpolate between points in latent space and generate sensible answers to unseen questions, but it's pretty clear that they don't generalize . I've seen far to many examples of models failing to display any sense of generalization to believe otherwise. That's…

>it's pretty clear that they don't generalize. I've seen far to many examples of models failing to display any sense of generalization to believe otherwise

Failing to generalize on whatever you have in mind is not evidence that the models are incapable of generalization. If you really think so, just be prepared to write off a good chunk of humans as well lol.

This is generalization. https://general-pattern-machines.github.io/

Just seems to me like you've taken examples of generalization and invented an alternate meaning just so you can't admit it generalizes.

"Oh but you see this example of answering an unseen question is "insert meaningless distinction" so it doesn't really count"

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#53

Post by totalhack »

I think natural language to SQL is a worthwhile goal and will have its use cases, but perhaps fine-tuning a model on a semantic layer and having it talk to a semantic data modeling API will yield a more useful/practical result. You may even find the semantic layer makes a natural language approach moot in many cases.

In Zillion* I added an experimental feature that uses OpenAI to form a report API call from natural language. Fine tuning (vs prompt tuning) this on the semantic model with gpt-3.5 would probably yield some notable improvement in abilities, as I view it as more of a toy feature at the moment.

* https://github.com/totalhack/zillion

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#55

Post by avereveard »

sillysaurusx wrote: post #44
lossolo wrote: post #42

Earlier quoted context omitted.

I have another example of ChatGPT not generalizing but just being a really good statistical model. I needed a solution to a problem that you can't find on Google, and a variation of a problem that also can't be found on Google. I attempted to obtain around 15 lines of code from ChatGPT that would solve the problem, but it consistently failed to produce the correct solution. I spent a few hours trying different prompt…

GPT 3.5 or 4? Surprisingly it makes a huge difference. I think a lot of peoples’ impressions are with 3.5, but many startups couldn’t have been built on it, whereas with 4 they can. If it was 4 I’d be curious about the specific problem if you’d be willing to link to the chat.

I had similar issue with gpt4, was looking for a library that given a grammar and a string, would produce a list of next valid symbols.

Gpt4 only ever suggested grammar validator to the point I had given up and was going to write a grammar generator, and so I started looking for the equivalent of antlr in python, and in three searches I find out nltk.grammar that actually solves the original problem

It's not a new library either, so I'm dumbfounded by why gpt4 couldn't make sense of my request.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#56

Post by johndough »

PheonixPharts wrote: post #49
sillysaurusx wrote: post #27

Earlier quoted context omitted.

I think you've fallen into the trap of "AIs don't generalize, they memorize." But they do in fact generalize. The reason ChatGPT is so valuable is precisely because it can help out with situations that have never been seen before, not because it merely unlocks old preexisting knowledge. The fella who saved their dog with ChatGPT comes to mind. https://nypost.com/2023/03/27/chatgpt-saved-my-dogs-life-aft...

I think you have fallen into the trap of mistaking interpolation for generalization . Working with these models every day, it's clear that they can certainly interpolate between points in latent space and generate sensible answers to unseen questions, but it's pretty clear that they don't generalize . I've seen far to many examples of models failing to display any sense of generalization to believe otherwise. That's…

With a sufficiently large dataset, every problem becomes interpolation.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#57

Post by sandGorgon »

aazo11 wrote: post #19
sandGorgon wrote: post #17

do you not use a vector db and embeddings search to get the table structure, etc ?

Most RAG approaches use a vectorDB and embeddings for schema linking. In this case the fine-tuning is handling schema linking and there is no vectorDB.

that is super interesting. Do you find that schema-linking using finetuning works better ?

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#58

Post by nl »

PheonixPharts wrote: post #49
sillysaurusx wrote: post #27

Earlier quoted context omitted.

I think you've fallen into the trap of "AIs don't generalize, they memorize." But they do in fact generalize. The reason ChatGPT is so valuable is precisely because it can help out with situations that have never been seen before, not because it merely unlocks old preexisting knowledge. The fella who saved their dog with ChatGPT comes to mind. https://nypost.com/2023/03/27/chatgpt-saved-my-dogs-life-aft...

I think you have fallen into the trap of mistaking interpolation for generalization . Working with these models every day, it's clear that they can certainly interpolate between points in latent space and generate sensible answers to unseen questions, but it's pretty clear that they don't generalize . I've seen far to many examples of models failing to display any sense of generalization to believe otherwise. That's…

Given a multi-dimensional latent space with enough dimensions it's hard to imagine cases of generalization that aren't interpolation between points in latent space (given enough data).

The one possible exception is logical inference, and this problem seems tractable with tool use or programming.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#59

Post by codeulike »

codeulike wrote: post #34

I often do complicated reports in SQL or complicated transformations in SQL for system migrations. To really write a query and 'get it right' you usually need insider knowledge that you can't glean from the column names. I see in their training set they've got comments about columns too. e.g. "Country" TEXT NOT NULL, - country where the singer born But thats still not enough. You also need a bunch of information abou…

Also, with complicated queries or reports, there's quite a big element of "do the results look roughly as I expect?" - e.g does the total income for last month look about right? Then you might do a detailed reconciliation and find out your total is 1% off due to not accounting for refunds and then you go and adjust your query and try again.

Re: Fine-tuning GPT-3.5-turbo for natural language to SQL

#60

Post by antupis »

totalhack wrote: post #53

I think natural language to SQL is a worthwhile goal and will have its use cases, but perhaps fine-tuning a model on a semantic layer and having it talk to a semantic data modeling API will yield a more useful/practical result. You may even find the semantic layer makes a natural language approach moot in many cases. In Zillion* I added an experimental feature that uses OpenAI to form a report API call from natural l…

Yeah it is literally billion dollar question how to do that querying. Is there some good papers or blog posts that go through different approaches?
Post reply on HN