Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dolphin Scheduler

github.com

Post reply on HN
Print view View original

11–20 of 27 posts

Re: Dolphin Scheduler

#11

Post by elcritch »

rb808 wrote: post #2

This looks great, I've always wanted something like this. I've always had autosys or controlm at work and they both suck. I'd just prefer if it had been around longer. Any other open source alternatives out there? I only know of Airflow, k8s Cronjobs.

I was just reading up on Broadway, written in Elixir, (https://hexdocs.pm/broadway/Broadway.html) that provides the fundamentals of batching/job control. It’s by the creator of Elixir and is based on 7 years of libraries in the area so the fundamentals are pretty well honed.

Re: Dolphin Scheduler

#12

Post by tasogare »

y4mi wrote: post #9
rb808 wrote: post #4

Also interesting that its one of the first open source applications from China I've seen.

I've actually stopped paying attention to https://github.com/vitalets/github-trending-repos because there are so many Chinese repositories each week. It's just rare to get them on HN, because it's a nightmare to go through their docs and they're usually not even attempting to write their code in English. Basically unusable for all intents and purposes, even if it were quality software.

> Basically unusable for all intents and purposes, even if it were quality software.

Only if you don’t have anyone who can’t read Chinese on your team. Also most repositories are not documented at well or at all anyway so language hardly matters.

Re: Dolphin Scheduler

#13

Post by massive »

rb808 wrote: post #2

This looks great, I've always wanted something like this. I've always had autosys or controlm at work and they both suck. I'd just prefer if it had been around longer. Any other open source alternatives out there? I only know of Airflow, k8s Cronjobs.

There you go https://github.com/meirwah/awesome-workflow-engines

Re: Dolphin Scheduler

#14

Post by TeMPOraL »

tbrock wrote: post #5
rb808 wrote: post #2

This looks great, I've always wanted something like this. I've always had autosys or controlm at work and they both suck. I'd just prefer if it had been around longer. Any other open source alternatives out there? I only know of Airflow, k8s Cronjobs.

The only other one I can think of off the top of my head is dagster: https://github.com/dagster-io/dagster It’s made by Nick Schrock of graphql fame, among others. I’m sure there are 100s of these projects though.

From the beginning of Dagster's readme:

> Dagster is a system for building modern data applications.

> Combining an elegant programming model and beautiful tools, Dagster allows infrastructure engineers, data engineers, and data scientists to seamlessly collaborate to process and produce the trusted, reliable data needed in today's world.

Two paragraphs, communicating zero bits of information. I wish Github repositories, of all places, didn't contain such noninformative copy.

Re: Dolphin Scheduler

#15

Post by hartzell »

rb808 wrote: post #2

This looks great, I've always wanted something like this. I've always had autosys or controlm at work and they both suck. I'd just prefer if it had been around longer. Any other open source alternatives out there? I only know of Airflow, k8s Cronjobs.

> Any other open source alternatives out there?

Here's a big handful of them: https://github.com/pditommaso/awesome-pipeline

Re: Dolphin Scheduler

#16

Post by y4mi »

tasogare wrote: post #12
y4mi wrote: post #9

Earlier quoted context omitted.

I've actually stopped paying attention to https://github.com/vitalets/github-trending-repos because there are so many Chinese repositories each week. It's just rare to get them on HN, because it's a nightmare to go through their docs and they're usually not even attempting to write their code in English. Basically unusable for all intents and purposes, even if it were quality software.

> Basically unusable for all intents and purposes, even if it were quality software. Only if you don’t have anyone who can’t read Chinese on your team. Also most repositories are not documented at well or at all anyway so language hardly matters.

yes, most repositories arent documented well either, thats definitely true and was part of my point, really.

how are you going to figure out why you're encountering a bug if not even the code itself is written in english?

its fine for learning repositories or simple toy projects, but if you actually want your code to be used... please use the world language. (and no, english isnt my native language either)

Re: Dolphin Scheduler

#17

Post by hinkley »

y4mi wrote: post #16
tasogare wrote: post #12

Earlier quoted context omitted.

> Basically unusable for all intents and purposes, even if it were quality software. Only if you don’t have anyone who can’t read Chinese on your team. Also most repositories are not documented at well or at all anyway so language hardly matters.

yes, most repositories arent documented well either, thats definitely true and was part of my point, really. how are you going to figure out why you're encountering a bug if not even the code itself is written in english? its fine for learning repositories or simple toy projects, but if you actually want your code to be used... please use the world language. (and no, english isnt my native language either)

I wonder if we’re hitting a point where a better decompiler would be useful. Transliterate the code into your first language, English or not.

Re: Dolphin Scheduler

#18

Post by jpitz »

monstrado wrote: post #10
rb808 wrote: post #2

This looks great, I've always wanted something like this. I've always had autosys or controlm at work and they both suck. I'd just prefer if it had been around longer. Any other open source alternatives out there? I only know of Airflow, k8s Cronjobs.

I've had alot of success using Apache NiFi as a distributed scheduler / general purpose workflow tool.

I'd love to see what kind of complexity you are managing there, and how.

Re: Dolphin Scheduler

#20

Post by JonathonW »

hinkley wrote: post #17
y4mi wrote: post #16

Earlier quoted context omitted.

yes, most repositories arent documented well either, thats definitely true and was part of my point, really. how are you going to figure out why you're encountering a bug if not even the code itself is written in english? its fine for learning repositories or simple toy projects, but if you actually want your code to be used... please use the world language. (and no, english isnt my native language either)

I wonder if we’re hitting a point where a better decompiler would be useful. Transliterate the code into your first language, English or not.

A decompiler can't pull contextually appropriate variable and function/method names out of nowhere (not to mention comments), which is the big roadblock when reading foreign-language code.

That is, you're just as likely to be able to follow foreign-language code as you are decompiled code. Either way, you've basically thrown out all the documentation and swapped out all the names for gibberish.

Post reply on HN