Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Any text-to-SQL benchmark should address difficulties of real-world data stores

cacm.acm.org

Post reply on HN
Print view View original

21–24 of 24 posts

Re: Any text-to-SQL benchmark should address difficulties of real-world data stores

#21

Post by rgavuliak »

fivetenpen wrote: post #6

The biggest issue with business users (who don’t know SQL) writing SQL with an LLM is that there is no one to validate that query and now that business user will treat that LLM response as canon to share in meetings, presentations, and with clients. The LLM may have forgotten a filter, used the wrong definition of revenue, or misunderstood the user’s intent by writing a query that answers their question in the most l…

Fair point, but before business people relied on other people to pull it for them. Sure, it was checked by human, but a technical human often lacked the business context and there's been plenty of instances, where the business person knowing their revenue and sales numbers challenged the analysis of a technical person.

Re: Any text-to-SQL benchmark should address difficulties of real-world data stores

#22

Post by efromvt »

rgavuliak wrote: post #21
fivetenpen wrote: post #6

The biggest issue with business users (who don’t know SQL) writing SQL with an LLM is that there is no one to validate that query and now that business user will treat that LLM response as canon to share in meetings, presentations, and with clients. The LLM may have forgotten a filter, used the wrong definition of revenue, or misunderstood the user’s intent by writing a query that answers their question in the most l…

Fair point, but before business people relied on other people to pull it for them. Sure, it was checked by human, but a technical human often lacked the business context and there's been plenty of instances, where the business person knowing their revenue and sales numbers challenged the analysis of a technical person.

Yeah with self-serve analytics all the rage (for good reason) for a bit, the bar from some places I've worked wouldn't be "does the agent beat a good analyst" it's "does the agent beat the a business person with SQL access who might have an overworked analyst glance over it", which is a much more tractable break-even point - and as you said, the business person might be even better at sanity checking numbers at that point.

Re: Any text-to-SQL benchmark should address difficulties of real-world data stores

#23

Post by efromvt »

brotchie wrote: post #12

Text to SQL is the wrong framing of the problem. Ideally you ask the agent the thing you want to know / business question you want answered. An agentic loop then runs. The agent can look at the schema, look at any existing SQL scripts available that query similar tables, run a few limit 10 probe queries. Analyze the data, do some joins, check the data again. Show you the data, ask for feedback, etc. Pure zero / one s…

I think/hope that most benchmarks have moved to an agentic loop - I'd still call that 'text to sql', since you're going from the business question to one or more SQL queries that provide the answer.

With a loop you can get extremely high results on a clean DB with a clear question. (I'm usually seeing high 9s accuracy.). A messy/ambiguous DB schema then degrades that again (which is what companies actually have, where you get closer to 20-30% rates pre-context engineering), which a clean semantic/presentation layer can bring back up. Then you're just left with the problem of underspecified/poorly formulated question - which you can partially solve with the agent pushing back, but also can be solved by improving the human side of things.

Post reply on HN