Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Koheesio: Nike's Python-based framework to build advanced data-pipelines

github.com

Post reply on HN
Print view View original

41–50 of 75 posts

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#41

Post by whimsicalism »

serial_dev wrote: post #22
esafak wrote: post #3

> Koheesio is not in competition with other libraries. Yes, it is, because nobody wants to run multiple orchestrators, and the "What sets Koheesio apart from other libraries?" section does little to help users decide why they should pick yours. Workflow orchestration is a mature category, as evidenced by the length of this list: https://github.com/meirwah/awesome-workflow-engines I would expect someone who's seriousl…

> Workflow orchestration is a mature category, as evidenced by the length of this list (...) Or could it be an evidence that the existing tools all have their flaws and any reasonably sized organization will hit these flaws pretty early, so many orgs come to the conclusion that the best approach is to just roll their own that fits their case relatively well?

no offense, but you clearly don’t know much about orchestrators

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#42

Post by whimsicalism »

esafak wrote: post #3

> Koheesio is not in competition with other libraries. Yes, it is, because nobody wants to run multiple orchestrators, and the "What sets Koheesio apart from other libraries?" section does little to help users decide why they should pick yours. Workflow orchestration is a mature category, as evidenced by the length of this list: https://github.com/meirwah/awesome-workflow-engines I would expect someone who's seriousl…

yeah they need to explain why this is better than flyte

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#43

Post by waffletower »

Many data engineering problems are impeded by strong typing, particularly type transduction applications (translating between a database type system and a transport such as Avro, for example). While in many cases that is somebody else's problem -- it is solved in a library -- when it isn't the strengths and facility of a dynamic language can save you considerable code complexity and maintenance. Type control is often central to reporting as well, and it is, again, more awkward in a strong typing context. I would tend to argue that insistence upon type frameworks such as pydantic in a data engineering framework is naive and imposed by academic rather than industry experience. There is a reason that python is chosen for data processing applications, and it certainly isn't typing.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#44

Post by steveBK123 »

newfocogi wrote: post #39

I'd like to understand what data engineering inside Nike is actually like. I'm curious because I have relevant experience on my LinkedIn profile, and I get reached out to almost weekly from third party recruiters trying to fill really low paying contract data engineering and ML jobs with Nike. These roles seem to be targeting people with professional experience in the US but pay roughly a 3rd of what I would consider…

My experience with these kinds of tools (and I've built some myself) tells me that you're better off hiring people who know what they are doing or having enough experienced people PLUS culture to train up juniors.

The idea that you'll just build a tool that makes hiring 10x as many inexperienced devs work is dubious. Just one more new DSL bro. Certainly we have cracked the code that no one else has.

The problem with these types of orgs/tools is that by its nature your DSL constrains the juniors/inexperienced devs to what is currently possible. There is not a lot of learning unless you rotate them through the tools team periodically, which no one does. It's also awful for the devs who are building in experience in something they can't use anywhere else.

I was in one shop where the "tools team" guys epiphany was he would meta-recruit by poaching the best data engineers out of the ETL team, lol. Very explicit "good team / bad team" vibes.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#45

Post by datavirtue »

steveBK123 wrote: post #33

If I had to guess, a tool like this might be useful in a shop with a lot of inexperienced devs. It's a thin wrapper to make sure everyone walks the same well worn path the same way. You have 2-3 devs work on the tooling, and a much larger team doing rote ETL. I worked at a shop that did this and the trade-off is TTM, as your 2 person tools team is constantly needing to unblock ETL team with new features as they encou…

I built a data processing framework at GE that let junior devs write whatever code they needed to transform a particular input. It provided an interface that they had to satisfy (for data lineage metrics) but otherwise scaled their code without them having to understand the distributed architecture or anything about the platform. Exceptions flowed up to the platform and became part of the data lineage metrics.

I walked into 20 years of adhoc code that had zero data lineage, recoverability, or scalability that was breaking daily. There were contractors with over a decade of tenure whose job it was to troubleshoot and fix their own brittle processes (and make new ones) daily.

I got laid off (747Max plus pandemic) as I was rolling it out and they went back to the old way.

Subsequently, a new startup (Pantomath) emerged with former GE engineers (and other former colleagues of mine) from my former department to address that problem domain.

Based on my experience trying to socialize this type of solution, sales are going to be a bitch.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#46

Post by esafak »

thenaturalist wrote: post #25
esafak wrote: post #21

Earlier quoted context omitted.

Every modern workflow orchestrator does those things you quote, and more. You make it sound like they're innovations when they're table stakes. Why wouldn't you just use Flyte or Kubeflow? Also, the fact that they say that the alternative is "raw Spark" tells me either that they're confused, or not very good at explaining. Spark is used to execute tasks in a pipeline, not to orchestrate it.

While I generally tend to agree with your basic criticism, I think you need to keep in mind our perspectives might be biased due to limited data. Flyte went OSS what, 4 years ago? I'm not super familiar with it, but a) could have been that it was too unpolished at the time or b) requiring K8s to be a non-starter for some teams/ orgs. Same for Kubeflow. We also don't know for how long Koheesio existed within Nike. In…

This project is two weeks old!

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#47

Post by waffletower »

waffletower wrote: post #43

Many data engineering problems are impeded by strong typing, particularly type transduction applications (translating between a database type system and a transport such as Avro, for example). While in many cases that is somebody else's problem -- it is solved in a library -- when it isn't the strengths and facility of a dynamic language can save you considerable code complexity and maintenance. Type control is often…

Giving this some more thought: I do know that Nike has a revolving door for developers. It seems that a framework like Koheesio allows Nike to essentially hire for scala from a pool of candidates that only have python experience. Once hired, as they use pyspark and koheesio daily, they don't even know they are scala developers. Much easier to hire/fire python developers these days.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#48

Post by steveBK123 »

datavirtue wrote: post #45
steveBK123 wrote: post #33

If I had to guess, a tool like this might be useful in a shop with a lot of inexperienced devs. It's a thin wrapper to make sure everyone walks the same well worn path the same way. You have 2-3 devs work on the tooling, and a much larger team doing rote ETL. I worked at a shop that did this and the trade-off is TTM, as your 2 person tools team is constantly needing to unblock ETL team with new features as they encou…

I built a data processing framework at GE that let junior devs write whatever code they needed to transform a particular input. It provided an interface that they had to satisfy (for data lineage metrics) but otherwise scaled their code without them having to understand the distributed architecture or anything about the platform. Exceptions flowed up to the platform and became part of the data lineage metrics. I walk…

A framework with composable building blocks, allowing devs to unblock themselves by adding the functionality they need is a good solution.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#49

Post by hipadev23 »

newfocogi wrote: post #39

I'd like to understand what data engineering inside Nike is actually like. I'm curious because I have relevant experience on my LinkedIn profile, and I get reached out to almost weekly from third party recruiters trying to fill really low paying contract data engineering and ML jobs with Nike. These roles seem to be targeting people with professional experience in the US but pay roughly a 3rd of what I would consider…

Nike's data engineering is very bad. It's hundreds of temporary contractors, mostly offshore, all with 6-18 month tenures, and everyone reinvents their own square wheel. Thousand upon thousands of abandoned confluence pages of documentation. The most convoluted SQL and data architecture you'll ever find. Getting answers to simple questions like "How many shoes did we sell in-store vs ecommerce last week?" is a nearly impossible task.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#50

Post by alex_lav »

newfocogi wrote: post #39

I'd like to understand what data engineering inside Nike is actually like. I'm curious because I have relevant experience on my LinkedIn profile, and I get reached out to almost weekly from third party recruiters trying to fill really low paying contract data engineering and ML jobs with Nike. These roles seem to be targeting people with professional experience in the US but pay roughly a 3rd of what I would consider…

Speaking only my own experiences, my contract was ~2x what competitors were paying in the US. This was similar amongst the contractors I worked with, depending on seniority.
Post reply on HN