Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Dlt – Python library to automate the creation of datasets

colab.research.google.com

Post reply on HN
Print view View original

41–50 of 56 posts

Re: Show HN: Dlt – Python library to automate the creation of datasets

#41

Post by nerdponx »

All of this sounds very appealing, both as a standalone tool and as a complement to some thing like Dbt.

However the following seems like kind of an anti-feature, which I at least would want the option to disable:

> You do not need to worry about the structure of a database or parquet files

> dlt will create a nice, typed schema out of your data and will migrate it when the data changes. You can put some data contracts and Pydantic models on top to keep your data clean.

This is the opposite of what I want in 99% of projects. Most of the time, there is some kind of well-defined schema, even if it changes a little bit over time. If that schema is going to be depended upon by something like a data warehouse ELT pipeline, I want precise control over it. I do not want to hand that off to an opaque library.

Moreover, the work of actually writing out the schema is like 1% of the overall effort in consuming a new data source, and usually it turns out to be a constructive, useful exercise in pinning down assumptions, finding gaps in understanding, etc. So I see a little benefit in hiding it.

A schema essentially forms a business-critical contract between two major sections of the overall data pipeline, and that is absolutely not something I want to be changing dynamically without my explicit understanding and consent.

This reminds me of the temptation I have seen in some developers (several of them ostensibly "senior") to use MongoDB for a straightforward CRUD-like application. The argument that it's schema-less to me is a striking anti-feature, something I explicitly do not want!

The only time I really want this is in the rare and atypical case where I truly have no schema at all, or the schema is changing erratically and frequently in ways that I cannot reasonably anticipate and/or cannot dedicate developer resources to accommodating. That's a niche case that most people flatly do not have. Of course it's nice when a tool supports the niche use case that is very hard to deal with by conventional means (see also: OpenRefine), but it should absolutely not be the default and our tools should not encourage us to lie to ourselves that it's something we want or need.

If you just want to reduce manual grunt work effort, consider something like generating a schema from an OpenAPI specification / JSONSchema.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#42

Post by jarulraj »

Very cool project, @MatthausK!

What are your thoughts on reducing LLM cost?

We are also exploring LLM-based data wrangling using EvaDB and cost is an important concern [1, 2, 3].

[1] https://github.com/georgia-tech-db/evadb

[2] https://medium.com/evadb-blog/stargazers-reloaded-llm-powere...

[3] https://github.com/pchunduri6/stargazers-reloaded

Re: Show HN: Dlt – Python library to automate the creation of datasets

#43

Post by nerdponx »

adrianbr wrote: post #8
mnky9800n wrote: post #5

Tbh I found it confusing why suddenly you were using chatgpt in the middle of the example. It made it seem like this is a gpt based tool but it's not.

Thank you for the feedback! I can see now how it could be confusing. The reason we used chatgpt is because it's an easy starting point - why read through examples when you can get the one you want in seconds? Because dlt is a library, it's closer to how language works and gpt can just use it - from our experiments, we cannot say the same about frameworks.

That example looks completely opaque to me. Not only does it obfuscate what's actually happening by using some incomplete code from a chatbot, but it also is an actually relevant to the task at hand, which is to demonstrate your library, not to demonstrate some beginner level API data access. Skimming over it, I couldn't tell where your library actually got involved at all, it just looked like a couple of functions to access data, followed by links to your documentation. I suggest dumping the whole thing and starting with a more coherent demo that focuses on the features of the tool you actually built, not on features of irrelevant systems.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#44

Post by nerdponx »

adrianbr wrote: post #19
Terretta wrote: post #18

> dlt is a minimalistic Python library, requires no backend and works whenever Python works. You can finetune it to ... run with ... Dagster. Relating to dagster in particular, this is in your docs: dlt incorporates the concept of implicit extraction DAGs to handle the dependencies between data sources and their transformations automatically. A DAG represents a directed graph without cycles, where each node represent…

There are multiple ways to run together - we will show a few in a demo coming out soon. We also consider a tighter integration like with Airflow described here as a possible next step https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deplo... We will investigate the interest incrementally as to not build any plugins that don't end up used.

For an example of prior art, you should look into Astronomer's Cosmos Library to see how they integrate Dbt into Airflow.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#45

Post by adrianbr »

nerdponx wrote: post #44
adrianbr wrote: post #19

Earlier quoted context omitted.

There are multiple ways to run together - we will show a few in a demo coming out soon. We also consider a tighter integration like with Airflow described here as a possible next step https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deplo... We will investigate the interest incrementally as to not build any plugins that don't end up used.

For an example of prior art, you should look into Astronomer's Cosmos Library to see how they integrate Dbt into Airflow.

Thank you! That's the example we looked at for our dlt-airflow integration :) the dlt dag becomes an airflow dag.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#46

Post by rudolfix »

nerdponx wrote: post #41

All of this sounds very appealing, both as a standalone tool and as a complement to some thing like Dbt. However the following seems like kind of an anti-feature, which I at least would want the option to disable: > You do not need to worry about the structure of a database or parquet files > dlt will create a nice, typed schema out of your data and will migrate it when the data changes. You can put some data contrac…

we actually spent several weeks writing openAPI -> dlt pipeline converter. you can check what've got here: https://github.com/dlt-hub/dlt-init-openapi

we'll continue this project but I learnt from it that most of the openAPI specs are a mess with hundreds of endpoints, incomplete definitions, lack of relations between endpoints, unique constraints etc. so there's tons of heuristics needed anyway. but sometimes it works. and is quite amazning!

if your source has well defined schema, we support ie. arrow tables natively. we keep 100% of that schema: https://dlthub.com/docs/blog/dlt-arrow-loading if you want to define your own schemas you can do it in many different way: - via pydantic models: https://dlthub.com/docs/general-usage/resource#define-a-sche... - via json-schema like definitions: https://dlthub.com/docs/general-usage/resource#define-schema - in a schema file: https://dlthub.com/docs/walkthroughs/adjust-a-schema

if you want to enforce schema and data contracts: - you can use pydantic models to validate data (if you use pydantic model as a table definition, this is the default) - we have soon-to-be-merged schema contract PR: https://github.com/dlt-hub/dlt/pull/594

My observations are that it is more than 1% of people that are fine with auto-generated schemas. But that could be selection bias (they use our library because they like it).

Re: Show HN: Dlt – Python library to automate the creation of datasets

#47

Post by [deleted] »

nerdponx wrote: post #41

All of this sounds very appealing, both as a standalone tool and as a complement to some thing like Dbt. However the following seems like kind of an anti-feature, which I at least would want the option to disable: > You do not need to worry about the structure of a database or parquet files > dlt will create a nice, typed schema out of your data and will migrate it when the data changes. You can put some data contrac…

[deleted]

Re: Show HN: Dlt – Python library to automate the creation of datasets

#49

Post by juliusgeo »

MatthausK wrote: post #36
bendyBus wrote: post #17

This is nice. Pulling data from an API and putting it in a SQL database should be a simple everyday task but the tools for this are 99% massive overkill. Great to see a simple library for a simple job.

Pulling from and into production databases is one of the early favourites from our dlt user base. Some reasons explained here in this MongoDB example ( https://dlthub.com/docs/blog/MongoDB-dlt-Holistics )

This is a really cool project—congrats! A somewhat related project that I worked on at MongoDB is PyMongoArrow, it does some of the same transformations to take unstructured MongoDB data and convert it to tabular formats like Arrow data frames. I’m curious what the support for BSON types that do not map cleanly to JSON types looks like? One example I can think of off the top of my head is Decimal128

Re: Show HN: Dlt – Python library to automate the creation of datasets

#50

Post by adrianbr »

nerdponx wrote: post #41

All of this sounds very appealing, both as a standalone tool and as a complement to some thing like Dbt. However the following seems like kind of an anti-feature, which I at least would want the option to disable: > You do not need to worry about the structure of a database or parquet files > dlt will create a nice, typed schema out of your data and will migrate it when the data changes. You can put some data contrac…

ahh good old manual fine tuning and maintenance. We are adding data contracts for things like event ingeston where schema needs to be strict or cases where you know ahead of time what to expect.

Our experience comes from startups that usually do not have time to track down the knowledge and rather go out and find/make their own. Here you definitely want evolution with alerts before curation - so load to raw, and curate from there. Picking out data out of something without a schema is called "schema on read" and you can read about its shortcomings. So this is both robust and practical.

For the fine tuning, as I mentioned, data contracts are a PR review and some tweaks away. They will be highly configurable between strict, rule based evolution, or free evolution. Definitely use alerts for curation of evolution events!

Post reply on HN