Show HN: Dataherald AI – Natural Language to SQL Engine

1) A middle layer, such as SQL, offers a vital introspection point to understand and fix LLM mistakes. Without it, we will end up with a "black box" that could lead to errors in data queries and manipulations, making debugging and correction much more challenging.

2) Mathematical properties and data integrity: Databases are built around principles like 3NF and BCNF that are pivotal for efficiency and data integrity. It is highly doubtful that these essential principles could be as effectively replicated in a pure LLM-based system without losing critical aspects of data management and efficient query planning & execution.

3) You mentioned "translational losses" without clarity on what these might be. SQL is a specialized declarative language with clear structure and syntax, while human language often leaves room for ambiguity. Relying solely on an LLM could lead to interpretations that are likely but not necessarily correct. This ambiguity might actually increase "translational losses" rather than reduce them.

I understand there's a lot of excitement around generative AI at the moment, but DBMS are an extremely complex topic and this feels like another case of "when you have a hammer everything looks like a nail".

1. Agree on the introspection point, but it's worth noting that the future of LLMs might involve self-awareness capabilities, which could provide an introspective mechanism similar to SQL's transparency. the aim would be to build upon this middle layer, not blindly supplant it. 2. While 3NF, BCNF, and other normalization forms have served us well, they are essentially tools to manage imperfections in our storage and retrieval systems. LLMs can be trained to understand these nuances intrinsically. additionally, database theories have evolved, with advancements like distributed databases, graph-based models, and nosql. so, it's not entirely outside the realm of possibility that we can pivot and adapt to new paradigms. 3. The "translational losses" referred to the semantic disconnect between natural language queries and their SQL representations. while SQL is unambiguous, the leap from a user's intent to the SQL formulation often introduces errors or inefficiencies. LLMs can be trained to improve over time, constantly refining based on feedback loops.

not arguing that SQL or databases as we know them are obsolete today, just advocating for a more imaginative exploration about where the tech is headed.

The thing is, an SQL database is already a sort of AI - an old school algorithmic AI:

“The code generator, and especially the logic in where*.c and in select.c, is sometimes called the query planner. For any particular SQL statement, there might be hundreds, thousands, or millions of different algorithms to compute the answer. The query planner is an AI that strives to select the best algorithm from these millions of choices.”

https://www.sqlite.org/arch.html

Translating natural language into SQL is ironic, because SQL was intended to be a natural language that would be interpreted by an AI.

In the future maybe an AI could be trained to bypass high-level languages. Perhaps it could generate intermediate representation (IR) code - skipping the tokenising and parsing. Perhaps it could generate byte code directly. Or it could even skip the VM and generate machine code to work with the back-end API.

I did an NL-to-SQL startup, but now I think is a much better time to do this.