Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is the simplest data orchestration tool you've worked with?

news.ycombinator.com

Post reply on HN
Print view View original

31–35 of 35 posts

Re: Ask HN: What is the simplest data orchestration tool you've worked with?

#31

Post by pramodbiligiri »

margor wrote: post #30
saturn8601 wrote: post #13

I used to work for an automation company that produced a product called ActiveBatch. It was such an amazing tool for just drag and drop automation. Its focus was on full fledged workflow automation and not just data orchestration. What I loved was its simplicity + its out of the box features. To set it up just took a simple MS SQL DB + An Installer. Bam you are up and running an absolute rock solid scheduler(i've see…

As I worked adjacent to people who ran thousands of jobs in ActiveBatch - that software indeed was very simple to use and its GUI might have been awesome - but it's been double edged sword where if you have hundreds of people working on it - it becomes maintenance nightmare and promoting changes between environments was non-existent, causing multiple incidents. Mind you, it might have been just culture at that place,…

The parent comment made me curious enough to go look it up. Is it this same ActiveBatch that you both are referring to? https://www.advsyscon.com/

Re: Ask HN: What is the simplest data orchestration tool you've worked with?

#32

Post by datadrivenangel »

mettamage wrote: post #29
niwtsol wrote: post #11

Earlier quoted context omitted.

Yeah, I love this — pure Python with cron or periodic tasks (e.g., Django) works great. Celery task for parallelization, and if you pipe logs/alerts into a Slack channel, you can actually get really far without needing a "proper" orchestration layer. I recently took over an Airflow system from a former colleague, and in our case, it’s just overly complex for what’s really a pretty simple data flow.

I don’t know much about airflow But isn’t it just also python with cron?

Airflow is python with cron, and the option for very sophisticated and useful orchestration tools, like retries, dependencies, etc. All the stuff you'll end up rolling yourself as your simple scheduled tasks grow.

Re: Ask HN: What is the simplest data orchestration tool you've worked with?

#33

Post by margor »

pramodbiligiri wrote: post #31
margor wrote: post #30

Earlier quoted context omitted.

As I worked adjacent to people who ran thousands of jobs in ActiveBatch - that software indeed was very simple to use and its GUI might have been awesome - but it's been double edged sword where if you have hundreds of people working on it - it becomes maintenance nightmare and promoting changes between environments was non-existent, causing multiple incidents. Mind you, it might have been just culture at that place,…

The parent comment made me curious enough to go look it up. Is it this same ActiveBatch that you both are referring to? https://www.advsyscon.com/

ActiceBatch by Redwood, correct. Its widely used by some hedge funds.

Re: Ask HN: What is the simplest data orchestration tool you've worked with?

#34

Post by saturn8601 »

margor wrote: post #30
saturn8601 wrote: post #13

I used to work for an automation company that produced a product called ActiveBatch. It was such an amazing tool for just drag and drop automation. Its focus was on full fledged workflow automation and not just data orchestration. What I loved was its simplicity + its out of the box features. To set it up just took a simple MS SQL DB + An Installer. Bam you are up and running an absolute rock solid scheduler(i've see…

As I worked adjacent to people who ran thousands of jobs in ActiveBatch - that software indeed was very simple to use and its GUI might have been awesome - but it's been double edged sword where if you have hundreds of people working on it - it becomes maintenance nightmare and promoting changes between environments was non-existent, causing multiple incidents. Mind you, it might have been just culture at that place,…

How long back are you talking about? Do you remember what version? When I was working there, there were improvements made to "Change Management" ie. promoting changes from Test -> Production. After I left I heard the improvements continued. When I was there, it was a ~50 person company that was very focused so this was a pain point they were well aware of.

Re: Ask HN: What is the simplest data orchestration tool you've worked with?

#35

Post by saturn8601 »

pramodbiligiri wrote: post #31
margor wrote: post #30

Earlier quoted context omitted.

As I worked adjacent to people who ran thousands of jobs in ActiveBatch - that software indeed was very simple to use and its GUI might have been awesome - but it's been double edged sword where if you have hundreds of people working on it - it becomes maintenance nightmare and promoting changes between environments was non-existent, causing multiple incidents. Mind you, it might have been just culture at that place,…

The parent comment made me curious enough to go look it up. Is it this same ActiveBatch that you both are referring to? https://www.advsyscon.com/

Yes its best to google for some real pics of the GUI. They just have "drawings" on their site.

When I was there it was not owned by Redwood, after 30+ years in business the original owners sold the company to them a few years back to retire.

Post reply on HN