Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Dlt – Python library to automate the creation of datasets

colab.research.google.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Show HN: Dlt – Python library to automate the creation of datasets

#31

Post by claytonjy »

MatthausK wrote: post #26
crabbone wrote: post #25

Earlier quoted context omitted.

Where's "disagreeing"? I'm not disagreeing with the authors of Dtl on anything...

We took at least one immediate practical good piece of advice out of this which is that we should release a conda package and make sure that dlt works in it.

I wouldn't make it a high priority. If there's one thing I know about conda users it's that "no conda package available" has never stopped them. In fact they prefer to pip install inside their conda environment, and the only conda packages they use are the ones that touch Nvidia drivers (e.g. pytorch).

Re: Show HN: Dlt – Python library to automate the creation of datasets

#32

Post by anentropic »

adrianbr wrote: post #30
NortySpock wrote: post #27

As an FYI, Databricks' "Delta Live Tables" product keeps being contracted to "DLT" at my employer, so you have some potential naming confusion in your future.

Thank you for the heads up! it is unfortunate, and with 3 letter acronyms this will happen. An easy way to remember is that we are the one you can pip install and plays well in the ecosystem. Databricks has interesting choice in marketing names, ngl - DLT named after the competing dbt - Renaming standards like raw/staging/prod to Bronze, Silver, Gold

I had a related problem when searching for "dlt on aws lambda"... Google thinks "dlt" is an abbreviation of "delete" and returns results accordingly

Re: Show HN: Dlt – Python library to automate the creation of datasets

#33

Post by MatthausK »

maximilian114 wrote: post #29

We are one of the early adopters and really like the “weight” of DLT. It’s heavy enough to be adding substantial value over homegrown scripts for extraction and loading. At the same time, it is light enough to be easy to add to a pre-existing data stack.

Thanks for your vote of confidence & support Max!

Re: Show HN: Dlt – Python library to automate the creation of datasets

#34

Post by MatthausK »

anentropic wrote: post #32
adrianbr wrote: post #30

Earlier quoted context omitted.

Thank you for the heads up! it is unfortunate, and with 3 letter acronyms this will happen. An easy way to remember is that we are the one you can pip install and plays well in the ecosystem. Databricks has interesting choice in marketing names, ngl - DLT named after the competing dbt - Renaming standards like raw/staging/prod to Bronze, Silver, Gold

I had a related problem when searching for "dlt on aws lambda"... Google thinks "dlt" is an abbreviation of "delete" and returns results accordingly

We hear a lot about the dlt & AWS Lambda. We have currently one user working on the use case (see our Slack https://dlthub-community.slack.com/archives/C04DQA7JJN6/p169...)

Re: Show HN: Dlt – Python library to automate the creation of datasets

#35

Post by adrianbr »

anentropic wrote: post #32
adrianbr wrote: post #30

Earlier quoted context omitted.

Thank you for the heads up! it is unfortunate, and with 3 letter acronyms this will happen. An easy way to remember is that we are the one you can pip install and plays well in the ecosystem. Databricks has interesting choice in marketing names, ngl - DLT named after the competing dbt - Renaming standards like raw/staging/prod to Bronze, Silver, Gold

I had a related problem when searching for "dlt on aws lambda"... Google thinks "dlt" is an abbreviation of "delete" and returns results accordingly

for now :) Thanks for pointing it out - and it looks like we should add an aws lambda guide too :)

If you want to deploy to lambda, try asking in the slack community, some folks there do it.

Or if you wanna try yourself, here is a similar guide that highlights some concerns from deploying on gcp cloud functions https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deplo...

Re: Show HN: Dlt – Python library to automate the creation of datasets

#36

Post by MatthausK »

bendyBus wrote: post #17

This is nice. Pulling data from an API and putting it in a SQL database should be a simple everyday task but the tools for this are 99% massive overkill. Great to see a simple library for a simple job.

Pulling from and into production databases is one of the early favourites from our dlt user base. Some reasons explained here in this MongoDB example (https://dlthub.com/docs/blog/MongoDB-dlt-Holistics)

Re: Show HN: Dlt – Python library to automate the creation of datasets

#37

Post by thornewolf »

brilliant. This is exactly the solution I always hoped for for the exact problem you describe. My specific use-case has always been reading news articles from a set of heterogeneous websites and consolidating them into one db.

why chose duckdb over sqlite?

Re: Show HN: Dlt – Python library to automate the creation of datasets

#39

Post by adrianbr »

thornewolf wrote: post #37

brilliant. This is exactly the solution I always hoped for for the exact problem you describe. My specific use-case has always been reading news articles from a set of heterogeneous websites and consolidating them into one db. why chose duckdb over sqlite?

Duckdb is analytical and gained popularity with the analytics crowd. it has multiple features that make it play well with use cases in that ecosystem such as aggregation speed, parquet support, etc
Post reply on HN