Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Natural-SQL-7B, a strong text-to-SQL model

github.com

Post reply on HN
Print view View original

21–30 of 171 posts

Re: Show HN: Natural-SQL-7B, a strong text-to-SQL model

#21

Post by rgbrgb »

So it looks like it scores 76.5% on SQL-Eval [0], a bit behind GPT-4 at 83% and sqlcoder-15b at 78%.

What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time?

As a programmer who always has to look stuff up when I SQL, I could definitely see asking something like this for a first draft of a query but it seems like I'm slightly better off asking the bigger models in these one-off cases (and I can run a 15b easily on my 64GB m1). If I'm in a corporate setting I'm not going to leak my schema into OpenAI's training data and there are definitely times when I'd want to run queries offline. Small/local models are great when you want to do a ton of queries (save $$).

A mini data scientist that could be queried by non-technical folks would be awesome but I wonder if there's a way to determine whether the query is falling in the 25% "incorrect" case... maybe there's a RAID-like consensus algorithm where you have multiple interrogate each other's answers to get a higher overall success rate.

Mostly thinking out loud :) but maybe ya'll have more ideas. Congrats on the release, OP!

[0]: https://github.com/defog-ai/sql-eval

Re: Show HN: Natural-SQL-7B, a strong text-to-SQL model

#22

Post by thecalebf »

cuuupid wrote: post #19

This is a big improvement, but I'm not a believer that SQL is the most appropriate query lang here. Personally am more bullish on language models being trained with ORMs, as those normally capture much more information about the fields. e.g. Passing in some of my more complex table schemas related to flight data and asking about overflights, the model struggles to resolve out information related to aviation. However,…

Neat, would you ever use a local model for that if it could work with ORMs?

Re: Show HN: Natural-SQL-7B, a strong text-to-SQL model

#24

Post by thecalebf »

Semaphor wrote: post #20
thecalebf wrote: post #15

Earlier quoted context omitted.

No, those are benchmark, evaluation questions. The fine tune dataset was a custom, synthetically generated dataset of ~20k PostgreSQL Text to SQL pairs covering different SQL categories and question types. I mention a little more about it here https://x.com/calebfahlgren/status/1754247740291207198?s=20

So this is essentially postgres only? Or how will it handle e.g. MS SQL Schemas and output?

Currently Postgres yes, already working on a dataset with more DDLs like MySQL, DuckDB, MSSQL, etc for a second iteration.

Re: Show HN: Natural-SQL-7B, a strong text-to-SQL model

#25

Post by whimsicalism »

rgbrgb wrote: post #21

So it looks like it scores 76.5% on SQL-Eval [0], a bit behind GPT-4 at 83% and sqlcoder-15b at 78%. What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time? As a programmer who always has to look stuff up when I SQL, I could definitely see asking something like this for a first draft of a query but it seems like I'm slightly better off asking…

do you not trust the setting OAI provides to exclude your conversation from training data?

Re: Show HN: Natural-SQL-7B, a strong text-to-SQL model

#26

Post by tempusalaria »

whimsicalism wrote: post #25
rgbrgb wrote: post #21

So it looks like it scores 76.5% on SQL-Eval [0], a bit behind GPT-4 at 83% and sqlcoder-15b at 78%. What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time? As a programmer who always has to look stuff up when I SQL, I could definitely see asking something like this for a first draft of a query but it seems like I'm slightly better off asking…

do you not trust the setting OAI provides to exclude your conversation from training data?

1) OpenAI has consistently gone back on commitments it has made

2) Sam Altman has a shady track record publicly, and if you believe the things people say privately he has consistently done business very dishonestly throughout his career. He is the CEO, and virtually the entire executive team are people he brought in from his network. It’s his company.

3) To give one example of many, OpenAI recently changed the terms of ChatGPT so that web users conversations can now be trained on (and if you want to save any chats you must opt in). Presumably this also applies to all conversations you had under the old policy despite saying they would never train on those conversations.

I could go on at length…

Re: Show HN: Natural-SQL-7B, a strong text-to-SQL model

#27

Post by anonylizard »

whimsicalism wrote: post #25
rgbrgb wrote: post #21

So it looks like it scores 76.5% on SQL-Eval [0], a bit behind GPT-4 at 83% and sqlcoder-15b at 78%. What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time? As a programmer who always has to look stuff up when I SQL, I could definitely see asking something like this for a first draft of a query but it seems like I'm slightly better off asking…

do you not trust the setting OAI provides to exclude your conversation from training data?

Your average SQL query is worthless, there no special sauce in SQL queries, their value comes from the dataset they run on.

These small SQL LLMs are indeed worthless, since LLM performance on SQL queries is quite limited right now, every % of accuracy matters.

Re: Show HN: Natural-SQL-7B, a strong text-to-SQL model

#28

Post by rgbrgb »

whimsicalism wrote: post #25
rgbrgb wrote: post #21

So it looks like it scores 76.5% on SQL-Eval [0], a bit behind GPT-4 at 83% and sqlcoder-15b at 78%. What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time? As a programmer who always has to look stuff up when I SQL, I could definitely see asking something like this for a first draft of a query but it seems like I'm slightly better off asking…

do you not trust the setting OAI provides to exclude your conversation from training data?

about as much as I trust 23andme to keep my genetic data private

i give their intent the benefit of the doubt but I’ve been on the other side of too many data collection systems to trust that theirs is foolproof and more secure than my local machine.

Re: Show HN: Natural-SQL-7B, a strong text-to-SQL model

#29

Post by whimsicalism »

anonylizard wrote: post #27
whimsicalism wrote: post #25

Earlier quoted context omitted.

do you not trust the setting OAI provides to exclude your conversation from training data?

Your average SQL query is worthless, there no special sauce in SQL queries, their value comes from the dataset they run on. These small SQL LLMs are indeed worthless, since LLM performance on SQL queries is quite limited right now, every % of accuracy matters.

Did you mean to reply to a different comment?

Re: Show HN: Natural-SQL-7B, a strong text-to-SQL model

#30

Post by anonylizard »

tempusalaria wrote: post #26
whimsicalism wrote: post #25

Earlier quoted context omitted.

do you not trust the setting OAI provides to exclude your conversation from training data?

1) OpenAI has consistently gone back on commitments it has made 2) Sam Altman has a shady track record publicly, and if you believe the things people say privately he has consistently done business very dishonestly throughout his career. He is the CEO, and virtually the entire executive team are people he brought in from his network. It’s his company. 3) To give one example of many, OpenAI recently changed the terms…

If you don't trust OpenAI, you can choose to trust Microsoft. Azure OpenAI is 3-months behind OpenAI that costs 3x as much, but you get more 'security' and better performance.
Post reply on HN