41–50 of 56 posts
> This abstraction shields users from the complexity of the underlying systems and allows us to add new data sources without changing the user experience. Cursed mission. These sorts of things do work amazingly well for toy problem domains. But, once you get into more complex business involving 4-way+ joins, things go sideways fast. I think it might be possible to have a human in the loop during the SQL authoring pha…
Earlier quoted context omitted.
You are doing a lot of the work a semantic layer would do for you. I wonder if you would have better luck having the LLM talk to a semantic layer instead of directly to the database.
Can you talk more about what an implementation of such a semantic layer would look like?
Earlier quoted context omitted.
It does require writing good instructions for the LLM to properly use the tables, and it works best if you carefully pick the tables that your agent is allowed to use beforehand. We have many users that use it for every day work with real data (definitely not toy problems).
Yes you are perfectly right. Our product pushes users to be selective on the tables they give access to a given agent for a given use-case :+1: The tricky part is correctly supporting multiple systems which each have their own specificity. All the way to Salesforce which is an entirely different beast in terms of query language. We're working on it right now and will likely follow-up with a blog post there :+1:
My general method for things like this is to: 1. Get a .dot file of the database. Many tools will export this. 2. Open the .dot in a tool I built for the purpose. 3. Select the tables I'm interested in, and export a subset of the .dot file representing just those tables and relationships. 4. Hand that subset .dot file to the LLM and say, "given this schema, write a query -- here's what I want: " That gets the job don…
Combining this with structured/constrained generation with grammars/pydantic can supercharge this btw
I use this every day: https://github.com/runekaagaard/mcp-alchemy
Earlier quoted context omitted.
Using a semantic layer is the cleanest way to have a human in the loop. A human can validate and create all important metrics (e.g. what does "monthly active users" really mean) then an LLM can use that metric definition whenever asked for MAU. With a semantic layer, you get the added benefit of writing queries in JSON instead of raw SQL. LLM's are much more consistent at writing a small JSON vs. hundreds of lines of…
Currently exploring cube for a "natural language to SQL" solution. My schema is - 90+ Tables, 2500+ Columns, well documented From your experience, does Cube look a fit? My use cases will definitely have JOINS.
with that many tables, you might want to use Views: https://cube.dev/docs/reference/data-model/view
Earlier quoted context omitted.
Using a semantic layer is the cleanest way to have a human in the loop. A human can validate and create all important metrics (e.g. what does "monthly active users" really mean) then an LLM can use that metric definition whenever asked for MAU. With a semantic layer, you get the added benefit of writing queries in JSON instead of raw SQL. LLM's are much more consistent at writing a small JSON vs. hundreds of lines of…
How well is that working for you? We use a pattern where we ETL things into tables that model the upstream source closely, then use SQL Views to tighten up the model and integrate across data sources where needed. Keeping this all inside one DB allows us to use tools that understand the schema for autocomplete, etc. I expect the developer experience would be significantly worse if we started writing views in YAML ins…
A semantic layer is the best way to enable self-serve analytics, but if you don't care about it, it's probably not worth the hassle.
We also use the semantic layer for other nice things like setting goals, column descriptions and other metadata.
Earlier quoted context omitted.
Currently exploring cube for a "natural language to SQL" solution. My schema is - 90+ Tables, 2500+ Columns, well documented From your experience, does Cube look a fit? My use cases will definitely have JOINS.
yes, that shouldn't be a problem. with that many tables, you might want to use Views: https://cube.dev/docs/reference/data-model/view
In my use case, it's going to be exposed to various kind of stakeholders and there will be versatility of user queries. I can't pre-create views/aggregations for all scenarios.
> This abstraction shields users from the complexity of the underlying systems and allows us to add new data sources without changing the user experience. Cursed mission. These sorts of things do work amazingly well for toy problem domains. But, once you get into more complex business involving 4-way+ joins, things go sideways fast. I think it might be possible to have a human in the loop during the SQL authoring pha…