Live data from Hacker News

Show HN: Dlt – Python library to automate the creation of datasets

colab.research.google.com

51–56 of 56 posts

Re: Show HN: Dlt – Python library to automate the creation of datasets

#51

Earlier quoted context omitted.

We took at least one immediate practical good piece of advice out of this which is that we should release a conda package and make sure that dlt works in it.

I wouldn't make it a high priority. If there's one thing I know about conda users it's that "no conda package available" has never stopped them. In fact they prefer to pip install inside their conda environment, and the only conda packages they use are the ones that touch Nvidia drivers (e.g. pytorch).

> "no conda package available" has never stopped them.

Yes and no. They won't stop because they want to get things done, and the things usually don't involve honing the infrastructure. But installing packages with pip usually breaks conda installation, not even a particular virtual environment. (Usually pip nukes the setuptools that come with conda, and then once you want to install / upgrade anything in base environment, you discover that it's toast because conda itself depends on setuptools, but it's now broken and cannot be reinstalled).

So, in practice, if you give up and use pip to install stuff, it means that for the next project you will be reinstalling conda (and you will probably lose all your previous virtual environments). Kinda sucks.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#52

All of this sounds very appealing, both as a standalone tool and as a complement to some thing like Dbt. However the following seems like kind of an anti-feature, which I at least would want the option to disable: > You do not need to worry about the structure of a database or parquet files > dlt will create a nice, typed schema out of your data and will migrate it when the data changes. You can put some data contrac…

ahh good old manual fine tuning and maintenance. We are adding data contracts for things like event ingeston where schema needs to be strict or cases where you know ahead of time what to expect. Our experience comes from startups that usually do not have time to track down the knowledge and rather go out and find/make their own. Here you definitely want evolution with alerts before curation - so load to raw, and cura…

Fair enough, especially if explicit alerting is involved.

Have you considered a hybrid solution, something that generates a contract from a large corpus of data, which can then be deployed statically?

I consider "responding to change" as a somewhat different scenario from "heterogeneous but not changing". So statically generating a contract from an existing corpus supports the latter.

I could also envision some kind of graceful degradation, where you have a static contract, but you have dynamic adjustments instead of outright failures if the data does not conform to that contract.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#53
post #8

Earlier quoted context omitted.

Thank you for the feedback! I can see now how it could be confusing. The reason we used chatgpt is because it's an easy starting point - why read through examples when you can get the one you want in seconds? Because dlt is a library, it's closer to how language works and gpt can just use it - from our experiments, we cannot say the same about frameworks.

That example looks completely opaque to me. Not only does it obfuscate what's actually happening by using some incomplete code from a chatbot, but it also is an actually relevant to the task at hand, which is to demonstrate your library, not to demonstrate some beginner level API data access. Skimming over it, I couldn't tell where your library actually got involved at all, it just looked like a couple of functions t…

Yes I agree. Based on what they show compared to what they say, I'm not really sure what this library actually does.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#54
post #9

Do you plan on integrating with metadata sources as well such as Amundsen or Datahub? Or is the plan that DLT will become the metadata source?

DataHub is AMAZING!!!! I can't believe it slipped under my radar! Thanks for sharing/prompting me.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#55
I've been fiddling around with meltano and finicky singer taps for 2 weeks, and today I found dlt. What a breath of fresh air -- thank you for the work! I think this is the future over the meltano and singer ecosystem -- the code quality, simplicity and ease of integration into my project, and documentation really makes this project shine.

Re: Show HN: Dlt – Python library to automate the creation of datasets

#56

Earlier quoted context omitted.

ahh good old manual fine tuning and maintenance. We are adding data contracts for things like event ingeston where schema needs to be strict or cases where you know ahead of time what to expect. Our experience comes from startups that usually do not have time to track down the knowledge and rather go out and find/make their own. Here you definitely want evolution with alerts before curation - so load to raw, and cura…

Fair enough, especially if explicit alerting is involved. Have you considered a hybrid solution, something that generates a contract from a large corpus of data, which can then be deployed statically? I consider "responding to change" as a somewhat different scenario from "heterogeneous but not changing". So statically generating a contract from an existing corpus supports the latter. I could also envision some kind…

I worked with dlt guys on exactly that. Using OpenAI functions to generate a schema for the data based on the raw data structure. You can check that work here: https://github.com/topoteretes/PromethAI-Memory It's in the level 1 folder
Post reply on HN