Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is the simplest data orchestration tool you've worked with?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 35 posts

Re: Ask HN: What is the simplest data orchestration tool you've worked with?

#23

Post by aradox66 »

cicdw wrote: post #16
vector_spaces wrote: post #7

Earlier quoted context omitted.

Is there anything even more lightweight, where you don't have to write your code any differently? For instance, say I have 10 jobs that don't depend on each other, all of them pretty small. Dagster and even Luigi feel like overkill but I'd still like to plug those into a unified interface where I can view previous runs, mainly logs and exit codes. Being able to do some light job configuration or add retries would be…

One of the goals of Prefect's SDK is to be minimally invasive from a code-standpoint (in the simplest case you only need two lines to convert a script to a `flow`). Our deployment model also makes infrastructure job config a first-class citizen so you might have a good time trying it out. (disclosure: work at Prefect)

Love prefect! but for workflows involving concurrency, Prefect code needs to get somewhat invasive.

Prefect relies on prefect.task()-wrapped methods as the lowest granularity of concurrency in a program, and requires you to use the (somewhat immature) prefect task APIs to implement that concurrency.

more on this complaint here: https://austinweisgrau.github.io/migrating-to-prefect-part-3...

Re: Ask HN: What is the simplest data orchestration tool you've worked with?

#24

Post by akgfab »

spaCy’s weasel package allows you to put a bunch of commands meant to be run in sequence in one project.yml file, pull assets, etc—- I find it to be the right level of abstraction and I’m pretty sure it’s not trying to become a cloud hosted do everything tool: https://github.com/explosion/weasel

Re: Ask HN: What is the simplest data orchestration tool you've worked with?

#25

Post by cicdw »

aradox66 wrote: post #23
cicdw wrote: post #16

Earlier quoted context omitted.

One of the goals of Prefect's SDK is to be minimally invasive from a code-standpoint (in the simplest case you only need two lines to convert a script to a `flow`). Our deployment model also makes infrastructure job config a first-class citizen so you might have a good time trying it out. (disclosure: work at Prefect)

Love prefect! but for workflows involving concurrency, Prefect code needs to get somewhat invasive. Prefect relies on prefect.task()-wrapped methods as the lowest granularity of concurrency in a program, and requires you to use the (somewhat immature) prefect task APIs to implement that concurrency. more on this complaint here: https://austinweisgrau.github.io/migrating-to-prefect-part-3...

This is an excellent write up thank you for sharing! Yea, our concurrency API needs an upgrade - coincidentally this is going to be a theme of the next sprint or two so I hope I can report some improvements back soon.

Re: Ask HN: What is the simplest data orchestration tool you've worked with?

#27

Post by jusonchan81 »

Unmeshed - it’s not open source. It’s a new version of Netflix Conductor. Scales really well and has a GitHub actions style agent that can be used to run commands orchestrated by the platform. It’s probably the cheapest commercial tool you can get.

Re: Ask HN: What is the simplest data orchestration tool you've worked with?

#29

Post by mettamage »

niwtsol wrote: post #11
rasmusab wrote: post #5

Pure python scripts, maybe using the #%%-convention ( https://code.visualstudio.com/docs/python/jupyter-support-py... ) so you get the best of both notebooks and scripts, in a right-sized instance/container/machine. And if you need to run jobs in parallel, then orchestrate using make, like so: https://www.sumsar.net/blog/makefile-recipe-python-data-pipe...

Yeah, I love this — pure Python with cron or periodic tasks (e.g., Django) works great. Celery task for parallelization, and if you pipe logs/alerts into a Slack channel, you can actually get really far without needing a "proper" orchestration layer. I recently took over an Airflow system from a former colleague, and in our case, it’s just overly complex for what’s really a pretty simple data flow.

I don’t know much about airflow

But isn’t it just also python with cron?

Re: Ask HN: What is the simplest data orchestration tool you've worked with?

#30

Post by margor »

saturn8601 wrote: post #13

I used to work for an automation company that produced a product called ActiveBatch. It was such an amazing tool for just drag and drop automation. Its focus was on full fledged workflow automation and not just data orchestration. What I loved was its simplicity + its out of the box features. To set it up just took a simple MS SQL DB + An Installer. Bam you are up and running an absolute rock solid scheduler(i've see…

As I worked adjacent to people who ran thousands of jobs in ActiveBatch - that software indeed was very simple to use and its GUI might have been awesome - but it's been double edged sword where if you have hundreds of people working on it - it becomes maintenance nightmare and promoting changes between environments was non-existent, causing multiple incidents.

Mind you, it might have been just culture at that place, but I don't think this is as good of an example as you make it be. Sure, it was easy to get started and made the life easier at the beginning, but running it at scale was not in any way easy.

Post reply on HN