Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Koheesio: Nike's Python-based framework to build advanced data-pipelines

github.com

Post reply on HN
Print view View original

21–30 of 75 posts

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#21

Post by esafak »

boffinAudio wrote: post #19
esafak wrote: post #3

> Koheesio is not in competition with other libraries. Yes, it is, because nobody wants to run multiple orchestrators, and the "What sets Koheesio apart from other libraries?" section does little to help users decide why they should pick yours. Workflow orchestration is a mature category, as evidenced by the length of this list: https://github.com/meirwah/awesome-workflow-engines I would expect someone who's seriousl…

>nobody wants to run multiple orchestrators Straw man argument. >I would expect someone who's seriously writing a new orchestrator in 2024 to cite the alternatives, their shortcomings, and how you intend to address them. Bonus points if you make a neat little table. Did you even try to read the docs before you launched this critical diatribe? From the docs ( https://engineering.nike.com/koheesio/latest/tutorials/onbo…

Every modern workflow orchestrator does those things you quote, and more. You make it sound like they're innovations when they're table stakes. Why wouldn't you just use Flyte or Kubeflow?

Also, the fact that they say that the alternative is "raw Spark" tells me either that they're confused, or not very good at explaining. Spark is used to execute tasks in a pipeline, not to orchestrate it.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#22

Post by serial_dev »

esafak wrote: post #3

> Koheesio is not in competition with other libraries. Yes, it is, because nobody wants to run multiple orchestrators, and the "What sets Koheesio apart from other libraries?" section does little to help users decide why they should pick yours. Workflow orchestration is a mature category, as evidenced by the length of this list: https://github.com/meirwah/awesome-workflow-engines I would expect someone who's seriousl…

> Workflow orchestration is a mature category, as evidenced by the length of this list (...)

Or could it be an evidence that the existing tools all have their flaws and any reasonably sized organization will hit these flaws pretty early, so many orgs come to the conclusion that the best approach is to just roll their own that fits their case relatively well?

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#23

Post by yunohn »

muspimerol wrote: post #15
mb5 wrote: post #14

Earlier quoted context omitted.

Why do you think the poster is connected with the repo? I couldn’t see any link.

I don't. Regardless who posted it, discussion and constructive criticism is warranted in the comments.

/s

Like it or hate it, it’s how social media works. Everyone posts other people’s things, and then more people get together to rip it to shreds.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#24

Post by thenaturalist »

serial_dev wrote: post #22
esafak wrote: post #3

> Koheesio is not in competition with other libraries. Yes, it is, because nobody wants to run multiple orchestrators, and the "What sets Koheesio apart from other libraries?" section does little to help users decide why they should pick yours. Workflow orchestration is a mature category, as evidenced by the length of this list: https://github.com/meirwah/awesome-workflow-engines I would expect someone who's seriousl…

> Workflow orchestration is a mature category, as evidenced by the length of this list (...) Or could it be an evidence that the existing tools all have their flaws and any reasonably sized organization will hit these flaws pretty early, so many orgs come to the conclusion that the best approach is to just roll their own that fits their case relatively well?

Are you at all familiar with the data integration / ETL framework space?

If so, I think you would recognize how unreasonable the presumption of your comment is. It certainly strikes me as such.

Most if not all the big OSS frameworks and commercial offerings originated in one big corp and subsequently moved into the Apache direction (Airflow) or spun out into their own companies.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#25

Post by thenaturalist »

esafak wrote: post #21
boffinAudio wrote: post #19

Earlier quoted context omitted.

>nobody wants to run multiple orchestrators Straw man argument. >I would expect someone who's seriously writing a new orchestrator in 2024 to cite the alternatives, their shortcomings, and how you intend to address them. Bonus points if you make a neat little table. Did you even try to read the docs before you launched this critical diatribe? From the docs ( https://engineering.nike.com/koheesio/latest/tutorials/onbo…

Every modern workflow orchestrator does those things you quote, and more. You make it sound like they're innovations when they're table stakes. Why wouldn't you just use Flyte or Kubeflow? Also, the fact that they say that the alternative is "raw Spark" tells me either that they're confused, or not very good at explaining. Spark is used to execute tasks in a pipeline, not to orchestrate it.

While I generally tend to agree with your basic criticism, I think you need to keep in mind our perspectives might be biased due to limited data.

Flyte went OSS what, 4 years ago? I'm not super familiar with it, but a) could have been that it was too unpolished at the time or b) requiring K8s to be a non-starter for some teams/ orgs. Same for Kubeflow.

We also don't know for how long Koheesio existed within Nike.

In short, there's a lot we don't know and there's a good chance the internal reasoning for investing in this made sense under certain circumstances in the past.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#26

Post by adrianbr »

That's really cool, did you already saw the dlt library? That one's done for very easy to use EL in python. It's similarly modular and built by senior data engineers for the data team, and the sources are generators which you could probably use too.

How is koheesio different to dlt? Where could they complement each other?

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#27

Post by gonzo41 »

esafak wrote: post #3

> Koheesio is not in competition with other libraries. Yes, it is, because nobody wants to run multiple orchestrators, and the "What sets Koheesio apart from other libraries?" section does little to help users decide why they should pick yours. Workflow orchestration is a mature category, as evidenced by the length of this list: https://github.com/meirwah/awesome-workflow-engines I would expect someone who's seriousl…

Every task orchestration tool kinda has a crappy security model. Sure they're a ton of them but when you start putting it all over the place it's just a hectic to get right. That is a feel the space someone could make gains in a big way.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#28

Post by koliber »

zaptheimpaler wrote: post #7
esafak wrote: post #3

> Koheesio is not in competition with other libraries. Yes, it is, because nobody wants to run multiple orchestrators, and the "What sets Koheesio apart from other libraries?" section does little to help users decide why they should pick yours. Workflow orchestration is a mature category, as evidenced by the length of this list: https://github.com/meirwah/awesome-workflow-engines I would expect someone who's seriousl…

This is the kind of attitude that makes people, companies, researchers hesitant to publish code online. Its free code for everyone to see, they don't owe you anything. Its not necessarily a "product" for your consumption, its just a repo.

It's a fair set of questions, posed fairly directly, without any sugarcoating. If you read it as a ruthless critique, it stings. If you read it as constructive feedback, it can bring the author a lot of value.

If the author can clearly show the value proposition of their library, it will get more adoption, and the community and the author will get value.

If the author realizes they coded something that is already a well-solved problem, or a poorly-constructed alternative, the author and Nike could gain by throwing this away and going with a better alternative.

Personally, I wasted too many hours of my life creating a solution that did not solve any problems. Had I done some thinking and/or market research ahead of time, it would have saved me ton's of time to work on more worthwhile endeavors.

Good feedback is worth its weight in gold, even if it hurts a bit to hear it.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#29

Post by ubercore »

serial_dev wrote: post #20

I used to work a little with ETLs, Spark, Storm, etc and I honestly don't understand the value proposition of this library. I'm no data engineer expert by any means (it was like 2 years working on data eng stuff about 30% of the time 5+ years ago), but I expected that at least I'd get what this is useful for.

This, at a glance, seems pretty simplistic. A neat project, but not something I would have expected on HN front page.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#30

Post by teekert »

hipadev23 wrote: post #4

Had Nike as a client for a period of time, interacted with quite a few people across their data org. There is absolutely no software you want authored by them.

I worked at a large Healthtech company before. I get the sentiment. But, in the forest of software based on poor decisions, there were definitely a couple of valuable gems made by very knowledgeable people. Generally those were the people with an "opensource mindset" (as opposed to the "my ultra-crappy code that is just some FOSS glued together is super valuable IP and you need to read 100s of QMS docs before you can lay your eyes on it"-people). Don't measure the whole org by the same yardstick.
Post reply on HN