Along the lines of Airflow, Prefect, Dagster, Argo, etc. What produced the least WTF per minute?
Ask HN: What is the simplest data orchestration tool you've worked with?
1–10 of 35 posts
Re: Ask HN: What is the simplest data orchestration tool you've worked with?
#2Straightforward programs in languages like Java, Python, etc.
The tools you describe all have the endpoint "you can't get there from here" and the only difference is if it takes you 5 seconds, 5 minutes, 5 days, 5 weeks or 5 months to learn that.
Re: Ask HN: What is the simplest data orchestration tool you've worked with?
#3My experience entails:
* Luigi -- extensive usage (4y+)
* Makefiles -- (15y+)
* GitHub Actions -- (4y+)
* Airflow -- little usage (* Dagster -- very little, just trying it out
* Prefect -- just followed tutorial
Although it lacks a lot of the monitoring and advanced web ui other platforms have (maybe because of it), Luigi is the simplest to reason about IMHO.
For a new project that will require complex orchestrations, I'd probably go with Dagster or Prefect nowadays. Dagster seems more complex and more powerful with its data lineage functionality, but I have very little experience with either tool.
If it's a simple project, a mix of Makefiles + GH Actions can work well.
Re: Ask HN: What is the simplest data orchestration tool you've worked with?
#4Either Perfect or Dagster. FWIW, the Dagster team is actively reducing the learning curve with each release.
Re: Ask HN: What is the simplest data orchestration tool you've worked with?
#5Pure python scripts, maybe using the #%%-convention (https://code.visualstudio.com/docs/python/jupyter-support-py...) so you get the best of both notebooks and scripts, in a right-sized instance/container/machine.
And if you need to run jobs in parallel, then orchestrate using make, like so: https://www.sumsar.net/blog/makefile-recipe-python-data-pipe...
Re: Ask HN: What is the simplest data orchestration tool you've worked with?
#6I’ve been using airflow for quite some time. Due to the maturity of where we are at, and while I’ve tested other solutions, I don’t really see changing things.
Re: Ask HN: What is the simplest data orchestration tool you've worked with?
#7My experience entails: * Luigi -- extensive usage (4y+) * Makefiles -- (15y+) * GitHub Actions -- (4y+) * Airflow -- little usage ( * Dagster -- very little, just trying it out * Prefect -- just followed tutorial Although it lacks a lot of the monitoring and advanced web ui other platforms have (maybe because of it), Luigi is the simplest to reason about IMHO. For a new project that will require complex orchestration…
Is there anything even more lightweight, where you don't have to write your code any differently? For instance, say I have 10 jobs that don't depend on each other, all of them pretty small.
Dagster and even Luigi feel like overkill but I'd still like to plug those into a unified interface where I can view previous runs, mainly logs and exit codes. Being able to do some light job configuration or add retries would be nice but not required. For the moment I just use a logging handler that writes to a database table and that's fine
Re: Ask HN: What is the simplest data orchestration tool you've worked with?
#8My experience entails: * Luigi -- extensive usage (4y+) * Makefiles -- (15y+) * GitHub Actions -- (4y+) * Airflow -- little usage ( * Dagster -- very little, just trying it out * Prefect -- just followed tutorial Although it lacks a lot of the monitoring and advanced web ui other platforms have (maybe because of it), Luigi is the simplest to reason about IMHO. For a new project that will require complex orchestration…
Is there anything even more lightweight, where you don't have to write your code any differently? For instance, say I have 10 jobs that don't depend on each other, all of them pretty small. Dagster and even Luigi feel like overkill but I'd still like to plug those into a unified interface where I can view previous runs, mainly logs and exit codes. Being able to do some light job configuration or add retries would be…
I think that Airflow 2 implemented a decorator mode which you can just use on functions.
Honestly, just use airflow, it has its issues but it sucks in well known and predictable ways.
Re: Ask HN: What is the simplest data orchestration tool you've worked with?
#9We’ve migrated to Flyte. Mostly using the Java/Scala API which can be a bit verbose. The official Python API is actually easy on the eyes.
Re: Ask HN: What is the simplest data orchestration tool you've worked with?
#10Makefile with make2graph to visualize DAGs.