Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

31–40 of 120 posts

Re: Engineers Shouldn’t Write ETL

#31

Post by liamconnell »

amarshall wrote: post #9

> There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. Instead, give people end-to-end ownership of the work they produce (autonomy). I think this is more the point than “engineers shouldn’t write ETL”: the engineering-related department consuming the ETL’s output should likely be the ones writing/maintaining it. Or,…

> the engineering-related department consuming the ETL’s output should likely be the ones writing/maintaining it.

This is exactly the author's point. The Data Scientists are consuming the ETL's output, so they should learn how to write and maintain ETL since it isnt very hard or time consuming with modern tools.

Re: Engineers Shouldn’t Write ETL

#32

Post by DoreenMichele »

internet555 wrote: post #16
whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything…

You see that a lot with 2e people -- bright people with some weaknesses or a disability. That could account for how negatively you have experienced this. Many 2e people have never really been taught good ways to handle the combination of big strengths and big weaknesses.

I serve as a sounding board a lot for my oldest son and that works well, but it's not uncommon for such people to just be trying to meet their own need to process information and/or feed their ego, oblivious to how it impacts other people and not really welcoming of the feedback they really need for this to be constructive. A good sounding board doesn't just listen, they ask pertinent questions and make insightful comments that help move the thought process along.

Sometimes when I meet people like that, I'm able to direct the conversation to a more constructive back and forth of that sort. But some people just know they have this need to talk, they have a lot of baggage that makes them openly hostile to meaningful feedback and they crave validation. Anything other than praising their half-baked ideas is met with toxic reactions. In such cases, the best you may be able to do is basically make a few polite noises and then disengage as quickly as possible.

Re: Engineers Shouldn’t Write ETL

#33

Post by Zaheer »

msencenb wrote: post #11

Does anyone have experience with ETL as a service like StitchData (not related to stitchfix)? The startup I'm employed at needs some data analysis, but it is not big data, simply a way to unify analytics into a queryable database. I'm not looking forward to writing any ETL code, and was hoping someone here had a tool to help.

I work on AWS Glue (https://aws.amazon.com/glue/). Depending on how complicated the pipeline you'd like to setup, you may not need to write any code.

Re: Engineers Shouldn’t Write ETL

#34

Post by bigger_cheese »

protomyth wrote: post #24
bigger_cheese wrote: post #18

Earlier quoted context omitted.

At least in my Org Reports are pretty much an after thought left to the data engineers (like me) to "take this metric I've developed" and display it on the morning report. Writing/updating a report is easiest part of my job it's the data that goes into building it that is hard translating the "simple metric I've developed" and getting it to run in a robust automated and sane fashion is the difficult part. The complex…

Ok, I think I'm getting the specifics of this situation. So, we are talking about internal reports, not something that could actually get in the external customer's hands.

Yes this is internal stuff. I work at a large industrial manufacturing plant.

Reports that go externally are done by certified people. (Laboratory technicians for product specifications and finance analysts for stock market stuff).

Re: Engineers Shouldn’t Write ETL

#35

Post by haney »

meritt wrote: post #10
iblaine wrote: post #5

> March 16, 2016 It's 2018. A lot has changed since 2016. The line between sw engineering & data engineering is much thinner.

I'm curious what innovative tools have emerged in the past two years that changed the dynamic?

Not sure what the author was referring to but, Airflow has gotten better / gained wider adoption during that time and my team started using DBT which saved a bunch time and was new during that period.

Re: Engineers Shouldn’t Write ETL

#36

Post by finnley »

There's a huge difference between writing ETL to apply business logic vs. grab the data from a common API like Google Analytics. There's no tool in the world that can write all the logic you need to transform data the way your organization uses it unless you have an extremely simple, common use case.

What this article is really saying is that replicating your data from source apps shouldn't be manually coded. The harder part still needs someone to write code so business users don't need to.

Re: Engineers Shouldn’t Write ETL

#38

Post by humbleMouse »

amarshall wrote: post #9

> There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. Instead, give people end-to-end ownership of the work they produce (autonomy). I think this is more the point than “engineers shouldn’t write ETL”: the engineering-related department consuming the ETL’s output should likely be the ones writing/maintaining it. Or,…

This quote is ridiculous. Some people enjoy plumbing high speed reliable/transparent data pipes. I could care less what goes thru the pipes I make.

Re: Engineers Shouldn’t Write ETL

#39

Post by evrydayhustling »

This is a great read, and this is a critical sentence:

> We are not optimizing the organization for efficiency, we are optimizing for autonomy.

Efficiency is for production pipelines where the product is thoroughly defined and production costs eat deeply into profit margin. Most software organizations have massive margins - but only if they get to the right product. Organizing people for ownership and autonomy engages their creativity, but also ensures that the org can move forward even when one side or the other falls behind.

Re: Engineers Shouldn’t Write ETL

#40

Post by jakecodes »

msencenb wrote: post #11

Does anyone have experience with ETL as a service like StitchData (not related to stitchfix)? The startup I'm employed at needs some data analysis, but it is not big data, simply a way to unify analytics into a queryable database. I'm not looking forward to writing any ETL code, and was hoping someone here had a tool to help.

Check out Meltano, a GitLab startup. It’s new and we’re iterating but it sounds like it could solve your exact situation. Feel free to leave issues where we could improve. https://gitlab.com/meltano/meltano
Post reply on HN