Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Koheesio: Nike's Python-based framework to build advanced data-pipelines

github.com

Post reply on HN
Print view View original

31–40 of 75 posts

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#31

Post by tiew9Vii »

serial_dev wrote: post #20

I used to work a little with ETLs, Spark, Storm, etc and I honestly don't understand the value proposition of this library. I'm no data engineer expert by any means (it was like 2 years working on data eng stuff about 30% of the time 5+ years ago), but I expected that at least I'd get what this is useful for.

From their docs: > Koheesio is a Python library that simplifies the development of data engineering pipelines. It provides a structured way

I think this pretty much sums it up, "a structured way".

It's looks to be a thin wrapper around spark to provide a consistent way to structure ETL jobs. They've implemented a mini dsl defining jobs as a datastructure on top of Spark.

I've seen several companies build stuff similar to this internally, defining jobs as a data structure. It all amounts to each company having their own internal conventions, their own view of what is easier for their devs and creating a framework for it. Nike have just decided to make theirs public.

You can do all this simply with simple spark scripts. Personally I'd use simple spark scripts. Big companies with lots of people love making these tools as companies love conventions, their conventions, their style guides, deal with staff churn/on-boarding frequently so believe these kind of things make that easier.

Probably makes sense in Nike as a way of organizing their ETL jobs but that looks to be all it brings. A way to structure/define your simple spark jobs the way Nike devs believe it should be done.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#32

Post by anentropic »

serial_dev wrote: post #20

I used to work a little with ETLs, Spark, Storm, etc and I honestly don't understand the value proposition of this library. I'm no data engineer expert by any means (it was like 2 years working on data eng stuff about 30% of the time 5+ years ago), but I expected that at least I'd get what this is useful for.

it looks like a layer of sugar over PySpark

seems to be Spark-only AFAICT?

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#33

Post by steveBK123 »

If I had to guess, a tool like this might be useful in a shop with a lot of inexperienced devs. It's a thin wrapper to make sure everyone walks the same well worn path the same way. You have 2-3 devs work on the tooling, and a much larger team doing rote ETL.

I worked at a shop that did this and the trade-off is TTM, as your 2 person tools team is constantly needing to unblock ETL team with new features as they encounter new requirements in the wild.

If your ETL team is 20+ people and the tools team doesn't have a head start, tools team will quickly fall behind an insurmountable backlog as your ETL team spins its wheels. But you might save some money if you choose the right KPI..

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#34

Post by haddr »

steveBK123 wrote: post #33

If I had to guess, a tool like this might be useful in a shop with a lot of inexperienced devs. It's a thin wrapper to make sure everyone walks the same well worn path the same way. You have 2-3 devs work on the tooling, and a much larger team doing rote ETL. I worked at a shop that did this and the trade-off is TTM, as your 2 person tools team is constantly needing to unblock ETL team with new features as they encou…

I think this is the case: when you run your pipelines at scale you want to standardize and simplify some repeatable aspects to lower the cost of managing them. You may also want to be orthogonal to orchestrator engines (or triggering engines) and avoid getting too opinionated and inflexible in the future. So this framework is exploring some sweet spot between raw spark pipelines and low code etl engines.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#35

Post by alessmar »

A few weeks ago, I chose to write my data pipelines using Apache Beam. It seems that Koheesio shares some features with this project, but I believe Apache Beam is superior due to its ability to run on various runners, support multiple programming languages, and integrate with numerous data sources and destinations.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#36

Post by steveBK123 »

haddr wrote: post #34
steveBK123 wrote: post #33

If I had to guess, a tool like this might be useful in a shop with a lot of inexperienced devs. It's a thin wrapper to make sure everyone walks the same well worn path the same way. You have 2-3 devs work on the tooling, and a much larger team doing rote ETL. I worked at a shop that did this and the trade-off is TTM, as your 2 person tools team is constantly needing to unblock ETL team with new features as they encou…

I think this is the case: when you run your pipelines at scale you want to standardize and simplify some repeatable aspects to lower the cost of managing them. You may also want to be orthogonal to orchestrator engines (or triggering engines) and avoid getting too opinionated and inflexible in the future. So this framework is exploring some sweet spot between raw spark pipelines and low code etl engines.

yeah though a lot of these fall for a variant of the "universal standard" conceit joked about in xkcd. All these low-code solutions suck, so we'll build our own in-house that surely won't have the same pitfalls..

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#37

Post by whoiscroberts »

ubercore wrote: post #29
serial_dev wrote: post #20

I used to work a little with ETLs, Spark, Storm, etc and I honestly don't understand the value proposition of this library. I'm no data engineer expert by any means (it was like 2 years working on data eng stuff about 30% of the time 5+ years ago), but I expected that at least I'd get what this is useful for.

This, at a glance, seems pretty simplistic. A neat project, but not something I would have expected on HN front page.

After decades of working on overly abstracted clever applications the only place I see elegance is in simplicity. I’d like to see more libraries like this on the front page.

Re: Koheesio: Nike's Python-based framework to build advanced data-pipelines

#39

Post by newfocogi »

I'd like to understand what data engineering inside Nike is actually like. I'm curious because I have relevant experience on my LinkedIn profile, and I get reached out to almost weekly from third party recruiters trying to fill really low paying contract data engineering and ML jobs with Nike. These roles seem to be targeting people with professional experience in the US but pay roughly a 3rd of what I would consider the going rate. There's another top level comment here that this tool might make sense "in a shop with a lot of inexperienced devs", which would confirm my anecdata. Maybe the roles are actually scams, who knows :shrug:
Post reply on HN