CoT, OPA, CoALA … these techniques can deliver massive performance improvements. Are there any other methods for agent frameworks?
any way to follow these developments vs pure LLM research?
Show HN: We open sourced our entire text-to-SQL product
11–20 of 147 posts
Re: Show HN: We open sourced our entire text-to-SQL product
#12Not finding the license anywhere. Which one have you chosen?
Guess it's public domain since there is no license
Re: Show HN: We open sourced our entire text-to-SQL product
#13Interesting! I am assuming it can do complex joins. Are there any examples of text -> sql it produces? I looked on the website but only saw "coming soon"
Yes. There is a schema linking step which identifies relevant columns and tables including any foreign key relationships if they exist.
The agent also can be finetuned on sample NL SQL pairs or they can be used in few shot prompting.
Re: Show HN: We open sourced our entire text-to-SQL product
#14Have you considered enforcing a grammar on the LLM when it is generating SQL? This could ensure that it only generates syntactically valid SQL, including awareness of the valid set of field names and their types, and such. It would not be easy, by any means, but I believe it is theoretically possible.
That cannot be done when using OpenAI API calls as far as I know
Nobody in the original post or this entire discussion said anything about OpenAI until your comment.
I thought it was fairly obvious that we were talking about a local LLM agent... if DataHerald is a wrapper around only OpenAI, and no other options, then that seems unfortunate.
Re: Show HN: We open sourced our entire text-to-SQL product
#15Super cool to see this!! I've been prototyping with NL-to-SQL recently, one problem I've stumble into is how to prevent mistakes from impacting your database, be it a hallucination or even a malicious actor who was able to send a prompt to the LLM agent. I don't have much input about the questions you asked here, but feel free to contact me (info on my profile) if you'd like to talk about those other aspects!!
Re: Show HN: We open sourced our entire text-to-SQL product
#16Have you considered enforcing a grammar on the LLM when it is generating SQL? This could ensure that it only generates syntactically valid SQL, including awareness of the valid set of field names and their types, and such. It would not be easy, by any means, but I believe it is theoretically possible.
The agent currently executed the generated SQL (limited to 10 rows) and recovers from errors.
Re: Show HN: We open sourced our entire text-to-SQL product
#17Earlier quoted context omitted.
That cannot be done when using OpenAI API calls as far as I know
Nobody in the original post or this entire discussion said anything about OpenAI until your comment. I thought it was fairly obvious that we were talking about a local LLM agent... if DataHerald is a wrapper around only OpenAI, and no other options, then that seems unfortunate.
The agent is LLM agnostic and you can use it with OpenAI or self-hosted LLMs. For self hosted LLM we have benchmarked performance with Mixtral for tool selection and CodeLlama for code generation.
Re: Show HN: We open sourced our entire text-to-SQL product
#18I still wonder who the audience is for tools like this. The website posits you can answer data questions without going through an analyst, but the role of the analyst is not to be a SQL whisperer for PMs and Executives - it is to be an expert in the model and the data. A data warehouse of any real scale is going to have some amount of issues - anomalous data, different interpretations of the same numbers - how does the LLM deal with that consistently across a business?
Re: Show HN: We open sourced our entire text-to-SQL product
#19x
Re: Show HN: We open sourced our entire text-to-SQL product
#20I still wonder who the audience is for tools like this. The website posits you can answer data questions without going through an analyst, but the role of the analyst is not to be a SQL whisperer for PMs and Executives - it is to be an expert in the model and the data. A data warehouse of any real scale is going to have some amount of issues - anomalous data, different interpretations of the same numbers - how does t…
the target audience is developers who wish to embed text to SQL functionality into their own products. the target audience is less the 'internal use case' (i.e. a data analyst) and more about letting external users do things they couldn't do before. a good example is payroll software where this type of technology can allow users to pull reports.