Show HN: Dlt – Python library to automate the creation of datasets

Having a Makefile is kind of... unusual. A hand-written one is even less so.

Now, I haven't used Poetry a lot, but... your Makefile is all about using Poetry. Which is kind of fun since it alone is supposed to provide most of that infra stuff. Also, allegedly in a cross-platform way, but you kind of made it very Linux-exclusive by including some Bash, and, well, Make itself being a foreigner in the MS land.

Another funny part that I can see there is how Python (not very smart) community was fighting against removing integrations from setuptools to kill stuff like "setup.py test" or "setup.py install", but Poetry does exactly that. I mean, it's not on you, and it's not bad. I actually believe that that's the better way to do it. But you'll find a lot of Python (not very smart) apologists foaming in their mouth when telling you how you are supposed to use different utilities on your project and how pyproject.toml is supposed to be the connecting link between all of them. Which, turns out, you also have.

And, hey-ho! you are also using tox, while running tests from Poetry.

Another funny part is that you specify all your dependencies while including patch in the version. You are definitely not the only one, and it's a common thing in (not very smart) Python community, but it still cracks me up every time I see this. Like, the whole point of semantic versioning was that you are supposed to depend on the version of public API. So, in principle, any package needs to depend only on the major version. Minor version if you've done something stupid (like depending on features that haven't been officially released). And patch -- well, that should never happen. The cherry on top if the caret (^) in your dependency specifications. That's another thing that should've never happened. The >= was supposed to work like that (but that's not on you, that the not very smart community behind SemVer's fault).

Now, just to taunt you a little bit more: you do realize that the most sizable chunk of Anaconda Python users are in the research community, which is... sort of your target audience? Don't you think it's ironic that you don't make conda packages for your project? (And wait until you discover that virtually none of the infra code you wrote for your project will work well in that environment).

I'm not trying to disparage you. I see a lot of projects aimed at the science research world. Not being a scientist myself, ironically, I get credit in scientific papers for working on projects' infra :) I see a lot of struggle to keep up with the programming world from the scientific community, and I also believe I see some terrible choices (eg. Python) that are now put this whole group of people in front of a very difficult choice: either to continue with Python because there's some knowledge garnered, even though it's not enough by any measure, some still want to clutch the pearls... while a smaller part of this community wants to cut the rope before it's too late.

From what I see, in the research world, dealing with Python infra is an unmitigated disaster with lots of negative consequences. But the way forward is not very clear.

Relating to dagster in particular, this is in your docs:

dlt incorporates the concept of implicit extraction DAGs to handle the dependencies between data sources and their transformations automatically. A DAG represents a directed graph without cycles, where each node represents a data source or transformation step.

When using dlt, the tool automatically generates an extraction DAG based on the dependencies identified between the data sources and their transformations. This extraction DAG determines the optimal order for extracting the resources to ensure data consistency and integrity.

How do you think about tying and running this and dagster together?