Live data from Hacker News

Show HN: Dlt – Python library to automate the creation of datasets

colab.research.google.com

21–30 of 56 posts

Re: Show HN: Dlt – Python library to automate the creation of datasets

#21
post #7
post #4

A similar tool was discussed a few days ago. OpenRefine. https://news.ycombinator.com/item?id=37970800

I'd say they're complimentary. One could use dlt to load the data and then use OpenRefine for to clean/transform it. dlt already does when combined with dbt, for example.

For future reference, that's "complementary". By far one of the most common mistakes (at least, among relatively uncommon words).

Re: Show HN: Dlt – Python library to automate the creation of datasets

#22
In the same way that dbt has established a foundational format for relational data transformation, dlt seems poised to establish a more ubiquitous solution for batch data pipelines. Native upsert "relational-ization" with configurability is a huge challenge that data engineers have been fighting for decades. Obviously, this library will not replace Kafka setups or more intensive streaming jobs, but there are so many custom CRON scripts in the universe doing exactly this. Would be a big win to see some standardization to Python and an OSS tool like this.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#23
post #7

Earlier quoted context omitted.

I'd say they're complimentary. One could use dlt to load the data and then use OpenRefine for to clean/transform it. dlt already does when combined with dbt, for example.

For future reference, that's "complementary". By far one of the most common mistakes (at least, among relatively uncommon words).

Yes it's pretty common :). I did notice the funny looking spelling after I hit reply but there was no way to edit it.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#24

Sorry for not being interested in what your library does, I like looking at how different Python packages go about packaging / installation / testing and other infra stuff. Having a Makefile is kind of... unusual. A hand-written one is even less so. Now, I haven't used Poetry a lot, but... your Makefile is all about using Poetry. Which is kind of fun since it alone is supposed to provide most of that infra stuff. Als…

I think you have some interesting feedback and points, but I somehow think they get lost in some of the prose that arguably is against HN guidelines, notably:

>Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

>When disagreeing, please reply to the argument instead of calling names.

>Please don't fulminate. Please don't sneer, including at the rest of the community.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#25

Sorry for not being interested in what your library does, I like looking at how different Python packages go about packaging / installation / testing and other infra stuff. Having a Makefile is kind of... unusual. A hand-written one is even less so. Now, I haven't used Poetry a lot, but... your Makefile is all about using Poetry. Which is kind of fun since it alone is supposed to provide most of that infra stuff. Als…

I think you have some interesting feedback and points, but I somehow think they get lost in some of the prose that arguably is against HN guidelines, notably: >Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. >When disagreeing, please reply to the argument instead of calling names. >Please don't fulminate. Please don't sneer, including at the rest of the community.

Where's "disagreeing"? I'm not disagreeing with the authors of Dtl on anything...

Re: Show HN: Dlt – Python library to automate the creation of datasets

#26

Earlier quoted context omitted.

I think you have some interesting feedback and points, but I somehow think they get lost in some of the prose that arguably is against HN guidelines, notably: >Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. >When disagreeing, please reply to the argument instead of calling names. >Please don't fulminate. Please don't sneer, including at the rest of the community.

Where's "disagreeing"? I'm not disagreeing with the authors of Dtl on anything...

We took at least one immediate practical good piece of advice out of this which is that we should release a conda package and make sure that dlt works in it.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#28
post #23

Earlier quoted context omitted.

For future reference, that's "complementary". By far one of the most common mistakes (at least, among relatively uncommon words).

Yes it's pretty common :). I did notice the funny looking spelling after I hit reply but there was no way to edit it.

Interesting. Usually editing is enabled for some window of time, but perhaps not for new accounts.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#29
We are one of the early adopters and really like the “weight” of DLT. It’s heavy enough to be adding substantial value over homegrown scripts for extraction and loading. At the same time, it is light enough to be easy to add to a pre-existing data stack.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#30

As an FYI, Databricks' "Delta Live Tables" product keeps being contracted to "DLT" at my employer, so you have some potential naming confusion in your future.

Thank you for the heads up!

it is unfortunate, and with 3 letter acronyms this will happen.

An easy way to remember is that we are the one you can pip install and plays well in the ecosystem.

Databricks has interesting choice in marketing names, ngl - DLT named after the competing dbt - Renaming standards like raw/staging/prod to Bronze, Silver, Gold

Post reply on HN