Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Koheesio: Nike's Python-based framework to build advanced data-pipelines

github.com

Post reply on HN
Print view View original

51–60 of 75 posts

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#51

Post by IshKebab »

waffletower wrote: post #43

Many data engineering problems are impeded by strong typing, particularly type transduction applications (translating between a database type system and a transport such as Avro, for example). While in many cases that is somebody else's problem -- it is solved in a library -- when it isn't the strengths and facility of a dynamic language can save you considerable code complexity and maintenance. Type control is often…

Python is chosen for data processing because it's one of the most popular languages in the world and it has a passable REPL (which is more than most languages) so you can use it for experimentation.

From the readme they describe it as "robust" and having a high level of type safety, so I'm guessing they're just leaning towards the "learn about bugs up before they hit production" end of the spectrum than you.

Then again I don't do any of this data engineering stuff so maybe it doesn't matter too much if it doesn't work reliably?

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#52

Post by esafak »

alessmar wrote: post #35

A few weeks ago, I chose to write my data pipelines using Apache Beam. It seems that Koheesio shares some features with this project, but I believe Apache Beam is superior due to its ability to run on various runners, support multiple programming languages, and integrate with numerous data sources and destinations.

If I wanted an abstraction layer for writing pipelines today I'd look into ZenML: https://www.zenml.io/vs/zenml-vs-orchestrators

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#53

Post by rockostrich »

esafak wrote: post #3

> Koheesio is not in competition with other libraries. Yes, it is, because nobody wants to run multiple orchestrators, and the "What sets Koheesio apart from other libraries?" section does little to help users decide why they should pick yours. Workflow orchestration is a mature category, as evidenced by the length of this list: https://github.com/meirwah/awesome-workflow-engines I would expect someone who's seriousl…

> Workflow orchestration is a mature category

As far as I can tell from the docs, this is not an orchestrator at all. It is a Python-wrapper for certain runtimes like PySpark. I don't see anything in the docs that mentions DAGs, dependency definitions, scheduling, or deployment.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#55

Post by [deleted] »

zaptheimpaler wrote: post #7
esafak wrote: post #3

> Koheesio is not in competition with other libraries. Yes, it is, because nobody wants to run multiple orchestrators, and the "What sets Koheesio apart from other libraries?" section does little to help users decide why they should pick yours. Workflow orchestration is a mature category, as evidenced by the length of this list: https://github.com/meirwah/awesome-workflow-engines I would expect someone who's seriousl…

This is the kind of attitude that makes people, companies, researchers hesitant to publish code online. Its free code for everyone to see, they don't owe you anything. Its not necessarily a "product" for your consumption, its just a repo.

[deleted]

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#56

Post by notyourwork »

hipadev23 wrote: post #49
newfocogi wrote: post #39

I'd like to understand what data engineering inside Nike is actually like. I'm curious because I have relevant experience on my LinkedIn profile, and I get reached out to almost weekly from third party recruiters trying to fill really low paying contract data engineering and ML jobs with Nike. These roles seem to be targeting people with professional experience in the US but pay roughly a 3rd of what I would consider…

Nike's data engineering is very bad. It's hundreds of temporary contractors, mostly offshore, all with 6-18 month tenures, and everyone reinvents their own square wheel. Thousand upon thousands of abandoned confluence pages of documentation. The most convoluted SQL and data architecture you'll ever find. Getting answers to simple questions like "How many shoes did we sell in-store vs ecommerce last week?" is a nearly…

> Getting answers to simple questions like "How many shoes did we sell in-store vs ecommerce last week?" is a nearly impossible task.

I find this type of thing scary as an outsider looking in. How a company so large has such immature engineering continues to astonish me.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#58

Post by shepardrtc »

notyourwork wrote: post #56
hipadev23 wrote: post #49

Earlier quoted context omitted.

Nike's data engineering is very bad. It's hundreds of temporary contractors, mostly offshore, all with 6-18 month tenures, and everyone reinvents their own square wheel. Thousand upon thousands of abandoned confluence pages of documentation. The most convoluted SQL and data architecture you'll ever find. Getting answers to simple questions like "How many shoes did we sell in-store vs ecommerce last week?" is a nearly…

> Getting answers to simple questions like "How many shoes did we sell in-store vs ecommerce last week?" is a nearly impossible task. I find this type of thing scary as an outsider looking in. How a company so large has such immature engineering continues to astonish me.

> I find this type of thing scary as an outsider looking in. How a company so large has such immature engineering continues to astonish me.

It's management that doesn't want to risk their positions by doing the very difficult business of either starting over or properly simplifying their stack. It's not easy, it's not quick, but if they can't even answer that basic question then they need to do the work.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#59

Post by rootusrootus »

hipadev23 wrote: post #49
newfocogi wrote: post #39

I'd like to understand what data engineering inside Nike is actually like. I'm curious because I have relevant experience on my LinkedIn profile, and I get reached out to almost weekly from third party recruiters trying to fill really low paying contract data engineering and ML jobs with Nike. These roles seem to be targeting people with professional experience in the US but pay roughly a 3rd of what I would consider…

Nike's data engineering is very bad. It's hundreds of temporary contractors, mostly offshore, all with 6-18 month tenures, and everyone reinvents their own square wheel. Thousand upon thousands of abandoned confluence pages of documentation. The most convoluted SQL and data architecture you'll ever find. Getting answers to simple questions like "How many shoes did we sell in-store vs ecommerce last week?" is a nearly…

> How many shoes did we sell in-store vs ecommerce last week?

That is perhaps not a great example. My brother is a business analyst at Nike (has been for 15 years or more). I just asked him how hard it would be to answer that question and he said it would be pretty easy. Granted, this is the kind of data he works with routinely, so it may be more difficult for other teams that do not.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#60

Post by crowcroft »

shepardrtc wrote: post #58
notyourwork wrote: post #56

Earlier quoted context omitted.

> Getting answers to simple questions like "How many shoes did we sell in-store vs ecommerce last week?" is a nearly impossible task. I find this type of thing scary as an outsider looking in. How a company so large has such immature engineering continues to astonish me.

> I find this type of thing scary as an outsider looking in. How a company so large has such immature engineering continues to astonish me. It's management that doesn't want to risk their positions by doing the very difficult business of either starting over or properly simplifying their stack. It's not easy, it's not quick, but if they can't even answer that basic question then they need to do the work.

To defend management a little bit, these massive companies have existed through many eras of technology with many different managers. They work with many external companies in many different ways. They have an exceptionally complex, but functioning tech stack, that allows all of these many dependencies to function together. Lastly, they are successful as they are!

It's not usually an issue of immaturity, it's just really hard. To make things worse, often people don't really want to do the work because literally any other data engineering job would probably be more enjoyable.

Simplifying the tech stack would probably require simplifying their business operations, which probably means less revenue.

Starting over is often literally not possible because there are so many interconnected systems that aren't all necessarily owned by the company trying to make the decision...

Post reply on HN