141–147 of 147 posts
Earlier quoted context omitted.
Perhaps orthogonal problem - imagine you join a new company that has an enterprise product with hundreds of tables. Is there a way to connect Dataherald to my DB, and ask basic questions about the DB? E.g. "where are stored records related to X".
Yes when you connect Dataherald to a DB it scans it and you can do exploratory queries.
Earlier quoted context omitted.
With what level of accuracy? And what guarantee of correctness? Because a report that happens to get the joins wrong once every 1000 reports is going to lead to fun legal problems. You still need someone who understands why you should use which approach to get the data you need without getting completely wrong numbers back that _look_ perfectly fine but reflect fantasy, not reality.
Getting joins wrong once in 1000 queries would beat 99.9% of experienced data analysts. Our standards for AI are too high. If an autonomous car causes one wreck per ten million miles, people set the cars on fire. When someone finds an LLM that suggests eating a small rock every day, that anecdote is used to discredit all LLM results. This shit makes errors. But what is the alternative? Human analysts who get joins wr…
Decisions should be made against an alternative, not against some fictitious perfect solution.
Earlier quoted context omitted.
you wouldnt believe the amount of developers that don't know how to write sql
Because ORM libraries were invented 30 years ago. There is no requirement to learn SQL for most of the applications built today.
- awesome-Text2SQL: https://github.com/eosphoros-ai/Awesome-Text2SQL :
> Curated tutorials and resources for Large Language Models, Text2SQL, Text2DSL、Text2API、Text2Vis and more.
- Awesome-code-llm > Benchmarks > Text to SQL: https://github.com/codefuse-ai/Awesome-Code-LLM#text-to-sql
- underlines/awesome-ml//llm-tools.md > RAG > OpenAI > dataherald,: https://github.com/underlines/awesome-ml/blob/master/llm-too...
- underlines/awesome-ml//llm-tools.md > Benchmarking > Benchmark Suites, Leaderboards: https://github.com/underlines/awesome-ml/blob/master/llm-too... :
- sql-eval: https://github.com/defog-ai/sql-eval :
> This repository contains the code that Defog uses for the evaluation of generated SQL. It's based off the schema from the Spider, but with a new set of hand-selected questions and queries grouped by query category. For an in-depth look into our process of creating this evaluation approach, see this.
> Our testing procedure comprises the following steps. For each question/query pair: 1. We generate a SQL query (possibly from an LLM). 2. We run both the "gold" query and the generated query on their respective database to obtain 2 dataframes with the results. 3. We compare the 2 dataframes using an "exact" and a "subset" match. TODO add link to blogpost. 4. We log these alongside other metrics of interest (e.g. tokens used, latency) and aggregate the results for reporting
- dataherald/services/engine/dataherald/tests/sql_generator/test_generator.py: https://github.com/Dataherald/dataherald/blob/main/services/...
Earlier quoted context omitted.
the target audience is developers who wish to embed text to SQL functionality into their own products. the target audience is less the 'internal use case' (i.e. a data analyst) and more about letting external users do things they couldn't do before. a good example is payroll software where this type of technology can allow users to pull reports.
I agree that is a more reasonable use-case. The readme for this tool seems geared toward the business of answering business questions.
To me it seems a more risky use-case since you don't have control/observability over what an untrusted user is asking for?
Are you going to keep maintaining the package?