The Problem: modern LLMs write syntactically correct SQL, but they struggle with real-world relational data. This is because real world data and schema is messy, natural language can often be ambiguous and LLMs are not trained on your specific dataset.
Solution: The core NL-to-SQL engine in Dataherald is an LLM based agent which uses Chain of Thought (CoT) reasoning and a number of different tools to generate high accuracy SQL from a given user prompt. The engine achieves this by:
- Collecting context at configuration from the database and sources such as data dictionaries and unstructured documents which are stored in a data store or a vector DB and injected if relevant
- Allowing users to upload sample NL SQL pairs (golden SQL) which can be used in few shot prompting or to fine-tune an NL-to-SQL LLM for that specific dataset
- Executing the SQL against the DB to get a few sample rows and recover from errors
- Using an evaluator to assign a confidence score to the generated SQL
The repo includes four services https://github.com/Dataherald/dataherald/tree/main/services:
1- Engine: The core service which includes the LLM agent, vector stores and DB connectors.
2- Admin Console: a NextJS front-end for configuring the engine and observability.
3- Enterprise Backend: Wraps the core engine, adding authentication, caching, and APIs for the frontend.
4- Slackbot: Integrate Dataherald directly into your Slack workflow for on-the-fly data exploration.
Would love to hear from the community on building natural language interfaces to relational data. Anyone live in production without a human in the loop? Thoughts on how to improve performance without spending weeks on model training?
Show HN: We open sourced our entire text-to-SQL product
github.com