So if they're doing 50,000,000 tokens to do the fine-tune, @ $0.08/1000 tokens and 2 epochs, that's about $800 to do the fine-tune. Not prohibitive from a business standpoint.
Fine-tuning GPT-3.5-turbo for natural language to SQL
41–50 of 77 posts
Re: Fine-tuning GPT-3.5-turbo for natural language to SQL
#42Earlier quoted context omitted.
Did you read the article you posted? -ChatGPT initially gives the same diagnosis the vets did, Babesiosis -It notes that Babesiosis may have either been a misdiagnosis or there may be a secondary condition/infection causing the remaining symptoms after the Babesiosis treatment didn't resolve all of them -It suggests such a hypothetical secondary condition could be IMHA, which the article notes is an extremely common…
Are you suggesting that it was commonplace for machine learning models to be able to extrapolate medical case reports into an actual diagnosis? Even being able to read and understand a case report is a minor miracle in extrapolation, and it’s interesting how far the goal posts move.
I have another example of ChatGPT not generalizing but just being a really good statistical model. I needed a solution to a problem that you can't find on Google, and a variation of a problem that also can't be found on Google. I attempted to obtain around 15 lines of code from ChatGPT that would solve the problem, but it consistently failed to produce the correct solution. I spent a few hours trying different prompts, indicating its errors and receiving apologies, only for it to generate another incorrect solution while acknowledging its mistake.
Solving out of distribution problems correctly seems almost impossible for it.
Re: Fine-tuning GPT-3.5-turbo for natural language to SQL
#43The cost per question seems super high. I can't even think of an API where a single call would cost $1cent. You better have a good pricing model to follow up on this.
This is basically what put me off when trying to use GPT-3.5 for this with SQL LangChain. And you can’t go cheaper LLM from OpenAI because then accuracy plummeted. GPT-3.5 is the sweet spot.
Sure, some use cases might work but it’s not going to be a thing that Just Works™ for products even accuracy issues aside. There’s just so much data to feed into each and every prompt, schemas and all. Many of them too if you want to enable joins.
Re: Fine-tuning GPT-3.5-turbo for natural language to SQL
#44Earlier quoted context omitted.
Are you suggesting that it was commonplace for machine learning models to be able to extrapolate medical case reports into an actual diagnosis? Even being able to read and understand a case report is a minor miracle in extrapolation, and it’s interesting how far the goal posts move.
I have another example of ChatGPT not generalizing but just being a really good statistical model. I needed a solution to a problem that you can't find on Google, and a variation of a problem that also can't be found on Google. I attempted to obtain around 15 lines of code from ChatGPT that would solve the problem, but it consistently failed to produce the correct solution. I spent a few hours trying different prompt…
GPT 3.5 or 4? Surprisingly it makes a huge difference. I think a lot of peoples’ impressions are with 3.5, but many startups couldn’t have been built on it, whereas with 4 they can.
If it was 4 I’d be curious about the specific problem if you’d be willing to link to the chat.
Re: Fine-tuning GPT-3.5-turbo for natural language to SQL
#45Did you provide any db schema information in the prompts after the model was fine tuned?
Re: Fine-tuning GPT-3.5-turbo for natural language to SQL
#46We've been chasing this rabbit since the beginning. It currently seems to be uncatchable for the use cases that would be most valuable to us - writing complex queries we've never seen before. Our use case seems to confound any notion of training or fine-tuning, since the cases we need the most help with are also the ones we have the fewest examples of. Instead of going for generative, few-shot models, I am starting t…
> Binary classification into deterministic query building. I have no clue what this means.
I tried something similar to this so I assume it’s to classify a table as being part of the resulting query or not, and using the schemas of the resulting tables in the prompt for the question. I found it just as useful to give it the full schema and ask it for a list of relevant tables, certainly more cost effective.
Re: Fine-tuning GPT-3.5-turbo for natural language to SQL
#47We've been chasing this rabbit since the beginning. It currently seems to be uncatchable for the use cases that would be most valuable to us - writing complex queries we've never seen before. Our use case seems to confound any notion of training or fine-tuning, since the cases we need the most help with are also the ones we have the fewest examples of. Instead of going for generative, few-shot models, I am starting t…
Yeah louie.ai query generation internals feel more like classic program synthesis, where's it's more about staged compilation, iterative refinement, etc, and we use gpt3/gpt4 for each stage rather than a traditional solver. "Which tables would help this query, which parts should go to text search vs vector search, ...", and we iteratively rewrite and pass along until we reconstitute.
At the same time, we're also moving to make a lot more of that process AI controlled, just across LLM calls (e.g., LLM-dictated ones), so it's a funny maturity process.
Re: Fine-tuning GPT-3.5-turbo for natural language to SQL
#48It wasn't clear to me what evaluation method was being used, the chart in the blog says Execution Accuracy, but the numbers that seem to be used appear to correlate with "Exact Set Match" (comparing on SQL) instead of the "Execution With Values" (comparing on result set values). For example, DIN-SQL + GPT-4 achieves an 85.3% "Execution With Values" score. Is that what is being used here? See the following for more in…
Hello, thank you very much for your meticulous comment. The 85.3% accuracy reported in our paper (I'm one of the authors of the DIN-SQL paper) pertains to the test set. However, in the blog post, we are reporting the performance on the development set, which stands at 74.2%.
Re: Fine-tuning GPT-3.5-turbo for natural language to SQL
#49We've been chasing this rabbit since the beginning. It currently seems to be uncatchable for the use cases that would be most valuable to us - writing complex queries we've never seen before. Our use case seems to confound any notion of training or fine-tuning, since the cases we need the most help with are also the ones we have the fewest examples of. Instead of going for generative, few-shot models, I am starting t…
I think you've fallen into the trap of "AIs don't generalize, they memorize." But they do in fact generalize. The reason ChatGPT is so valuable is precisely because it can help out with situations that have never been seen before, not because it merely unlocks old preexisting knowledge. The fella who saved their dog with ChatGPT comes to mind. https://nypost.com/2023/03/27/chatgpt-saved-my-dogs-life-aft...
I think you have fallen into the trap of mistaking interpolation for generalization.
Working with these models every day, it's clear that they can certainly interpolate between points in latent space and generate sensible answers to unseen questions, but it's pretty clear that they don't generalize. I've seen far to many examples of models failing to display any sense of generalization to believe otherwise.
That's not to say that interpolation in a rich latent space of text isn't very useful. But it's not the same level of abstraction that comes from true generalization in this space.
Re: Fine-tuning GPT-3.5-turbo for natural language to SQL
#50I think natural language to SQL may not be as great as it sounds in many real-world applications. You often have permissions, tenants, rules about who-can-see-what. I love the idea of letting users use natural language to query and possibly even bulk-update data. But if the app has rules like those, this would just be a SQL injection vulnerability. You could possibly limit it - just allow query over certain views or…
You could implement row level security on the database to prevent unauthorized access.