Koheesio: Nike's Python-based framework to build advanced data-pipelines

Post by MisterTea » Tue, Jun 04, 2024, 9:12 PM UTC

rootusrootus wrote: post #59

hipadev23 wrote: post #49

Earlier quoted context omitted.

Nike's data engineering is very bad. It's hundreds of temporary contractors, mostly offshore, all with 6-18 month tenures, and everyone reinvents their own square wheel. Thousand upon thousands of abandoned confluence pages of documentation. The most convoluted SQL and data architecture you'll ever find. Getting answers to simple questions like "How many shoes did we sell in-store vs ecommerce last week?" is a nearly…

> How many shoes did we sell in-store vs ecommerce last week? That is perhaps not a great example. My brother is a business analyst at Nike (has been for 15 years or more). I just asked him how hard it would be to answer that question and he said it would be pretty easy. Granted, this is the kind of data he works with routinely, so it may be more difficult for other teams that do not.

Post by hipadev23 » Wed, Jun 05, 2024, 5:09 AM UTC

crowcroft wrote: post #60

shepardrtc wrote: post #58

Earlier quoted context omitted.

> I find this type of thing scary as an outsider looking in. How a company so large has such immature engineering continues to astonish me. It's management that doesn't want to risk their positions by doing the very difficult business of either starting over or properly simplifying their stack. It's not easy, it's not quick, but if they can't even answer that basic question then they need to do the work.

To defend management a little bit, these massive companies have existed through many eras of technology with many different managers. They work with many external companies in many different ways. They have an exceptionally complex, but functioning tech stack, that allows all of these many dependencies to function together. Lastly, they are successful as they are! It's not usually an issue of immaturity, it's just re…

Post by thenaturalist » Wed, Jun 05, 2024, 6:39 AM UTC

esafak wrote: post #46

thenaturalist wrote: post #25

Earlier quoted context omitted.

While I generally tend to agree with your basic criticism, I think you need to keep in mind our perspectives might be biased due to limited data. Flyte went OSS what, 4 years ago? I'm not super familiar with it, but a) could have been that it was too unpolished at the time or b) requiring K8s to be a non-starter for some teams/ orgs. Same for Kubeflow. We also don't know for how long Koheesio existed within Nike. In…

This project is two weeks old!

Post by AtlasBarfed » Wed, Jun 05, 2024, 12:43 PM UTC

waffletower wrote: post #43

Many data engineering problems are impeded by strong typing, particularly type transduction applications (translating between a database type system and a transport such as Avro, for example). While in many cases that is somebody else's problem -- it is solved in a library -- when it isn't the strengths and facility of a dynamic language can save you considerable code complexity and maintenance. Type control is often…

Post by crowcroft » Wed, Jun 05, 2024, 1:43 PM UTC

rexpop wrote: post #63

crowcroft wrote: post #60

Earlier quoted context omitted.

To defend management a little bit, these massive companies have existed through many eras of technology with many different managers. They work with many external companies in many different ways. They have an exceptionally complex, but functioning tech stack, that allows all of these many dependencies to function together. Lastly, they are successful as they are! It's not usually an issue of immaturity, it's just re…

> Simplifying the tech stack would probably require simplifying their business operations, which probably means less revenue. I don't understand this perspective. Simplifying the tech stack might mean taking multiple services in multiple languages, and deprecating some in favor of migrating that functionality to the most maintainable codebase. This shouldn't mean "simplifying their business operations", or affecting…