Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

1–10 of 120 posts

Re: Engineers Shouldn’t Write ETL

#2

Post by corporateguy5 »

This article matches my experience exactly. Some companies will hire a “data scientist” on pedigree. They will be low on skills and high on charisma. The engineers are burdened with implementing the ideas as well as shoulder the failure of the algorithms. “You spent the last few months implementing algorithms and none worked?”. Very little blame will go to the data scientist. In tons of cases data scientists are more like product managers.

Re: Engineers Shouldn’t Write ETL

#4

Post by protomyth »

Report Developers, on the other hand, are folks who have made a career around designing reports in a specific tool (e.g. Microstrategy, et al). They are specialists.

Is this the common perception, because it really doesn't line up with my experience?

Re: Engineers Shouldn’t Write ETL

#6

Post by kevinpet »

protomyth wrote: post #4

Report Developers, on the other hand, are folks who have made a career around designing reports in a specific tool (e.g. Microstrategy, et al). They are specialists. Is this the common perception, because it really doesn't line up with my experience?

Seems like the author is talking about the pre-big data version of business intelligence with star schemas and attempts at drag and drop tools, which has been supplanted somewhere around 2010-2015 by open source big data tools. I wasn't at a big enough company to have a proper BI department pre the data science renaming, so I can't really opine on whether it's true.

Re: Engineers Shouldn’t Write ETL

#7

Post by EToS »

Lots of people want their key discipline to be the centre of the universe, you see it across designers, content creators, engineers, testers etc.. The key to any team in my experience is to have a healthy mixture of specialists (narrow scope, high resolution) and polyglots (wide scope, lower resolution), and to promote collaboration as much as possible..

Re: Engineers Shouldn’t Write ETL

#8

Post by whack »

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset.

I sometimes gave my own suggestions on how to improve upon their ideas, but for the most part, I was happy to focus on implementing their ideas, in the most clean, elegant, robust and testable manner possible. I was happy to do the "plumbing" work of improving upon our tech stack and architecture, in order to make the entire system better functioning and easier to maintain.

According to the author, I'm supposed to resent the fact that I'm a "doer/plumber", and not a "thinker". In reality, it was the opposite. Do I really want to spend my entire day reading the Bloomberg manual and figuring out which tables/columns will give us the data we want, and the nuances of what this dataset does and does not cover? Sorry, I have zero interest in doing that.

I enjoy programming. I enjoy system design. I enjoy building stuff. I have zero interest in becoming an expert on how to interpret the Bloomberg symbology file. Besides, if I ever left the financial industry and joined a tech company, that knowledge will become completely useless.

Did I or anyone consider myself to be a "menial" plumber? I don't think so. I was getting paid hundreds of thousands of dollars, because the "thinkers" recognized the value that I brought to the table. They appreciated that I could quickly and robustly implement the ideas that they had, and keep the system running smoothly without hiccups. They recognized anyone can do a "good enough" job, but it's much much harder to find someone who can do a great job. And for my part, I was perfectly happy to be that guy.

If you're someone who wants to expand your breadth and take on more "thinker" responsibilities, more power to you. But just don't forget that there are people like me out there too. There's no shame in being an excellent "doer".

Re: Engineers Shouldn’t Write ETL

#9

Post by amarshall »

> There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. Instead, give people end-to-end ownership of the work they produce (autonomy).

I think this is more the point than “engineers shouldn’t write ETL”: the engineering-related department consuming the ETL’s output should likely be the ones writing/maintaining it. Or, perhaps more generally: don’t delegate entirely to another team if the team that cares about the result is capable of meeting their own needs.

Post reply on HN