Engineers Shouldn’t Write ETL • Hacker News Board

Re: Engineers Shouldn’t Write ETL

#51

Post by kafkaesq » Fri, Mar 18, 2016, 5:24 PM UTC

kaspm wrote: post #40

kafkaesq wrote: post #12

There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. I'm not sure I get why writing ETL code for data you'll never consume is any more soul-sucking than, say, refactoring JS code for a website you couldn't begin to care about (and which will never be properly re-designed anyway); or even doing "thinker"-level work bu…

And I also fundamentally disagree with the notion that moving a large amount of realtime data reliably and with accuracy, monitored and consistent with relatively little failure is not an interesting engineering challenge in itself. I find that for all the talk about data-driven organizations, most don't use a tenth of what is available but that when the tenth is needed, it's hugely satisfying to be able to provide i…

I'm with you. I hear people reflexively dissing ETL (and other aspects of front-line data engineering) all the time, but I've come to suspect they don't really know these systems are actually about.

Re: Engineers Shouldn’t Write ETL

#52

Post by v64 » Fri, Mar 18, 2016, 5:28 PM UTC

lafay wrote: post #30

Is ETL really even necessary anymore? Why not just run fast ad hoc queries over the raw data with something like Google BigQuery?

Getting the raw data into BigQuery or another tool is an ETL problem in itself. You have unstructured log files, you have external APIs such as SalesForce, you have various relational databases, etc etc. Someone has to come in and transform this data into a unified format that can be inserted into BigQuery.

Re: Engineers Shouldn’t Write ETL

#53

Post by btilly » Fri, Mar 18, 2016, 5:28 PM UTC

The thinker/doer problem goes way back. In most organizations the person who thinks of something gets the lion's share of the credit, and the person who implements it does the lion's share of the work. And if it turns out to be a bad idea, the thinker can always blame a bad implementation, thereby passing the lion's share of the blame to the doer.

I've seen careers made and broken based on whether people got to play thinker or doer.

This makes rewards for thinking very lopsided. However the problem is that actual credit for success REALLY belongs with the people who did the work.

This problem shows up at every scale in every organization. For example there are hundred people who want to be the business side of a startup for every person who wants to build the tech. Why? The business person gets to be the thinker, the developer does the work. And then the business person expects to become the CEO and get the bulk of the payout!

Re: Engineers Shouldn’t Write ETL

#54

Post by hodwik » Fri, Mar 18, 2016, 5:34 PM UTC

TeMPOraL wrote: post #43

throwawy31816 wrote: post #2

This author seriously needs to expand all of his TLIs (three-letter initialisms) the first time he uses them, as any writer worth his or her salt would do. There are those who may be interested in what he has to say, but can't follow because of assuming abbreviations.

100% agreed. I was clueless about ETL too. While the acronym itself may be old, people forget that this field is hip and sexy for maybe like last 3 years or so.

Nope.

https://www.google.com/trends/explore#q=extract%20transform%...

Re: Engineers Shouldn’t Write ETL

#55

Post by dspillett » Fri, Mar 18, 2016, 5:36 PM UTC

kaspm wrote: post #40

kafkaesq wrote: post #12

There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. I'm not sure I get why writing ETL code for data you'll never consume is any more soul-sucking than, say, refactoring JS code for a website you couldn't begin to care about (and which will never be properly re-designed anyway); or even doing "thinker"-level work bu…

And I also fundamentally disagree with the notion that moving a large amount of realtime data reliably and with accuracy, monitored and consistent with relatively little failure is not an interesting engineering challenge in itself. I find that for all the talk about data-driven organizations, most don't use a tenth of what is available but that when the tenth is needed, it's hugely satisfying to be able to provide i…

> And I also fundamentally disagree with the notion that [ETL work] is not an interesting engineering challenge in itself.

A lot of people think that certain DBA/ETL/BI/similar work is boring and simlpy don't want to do it and so don't learn to do it well. Which is fine by me: it means those of us who can do it well can get paid good money when someone needs it.

The only problem with this theory in practise is that many also think such work is easy and free of complications; so they baulk at paying for people genuinely can do it well, get people less experienced who say they can do it well but do it badly, and judge the rest of us by that standard and assume database people are thick and can't do easy jobs properly...

Re: Engineers Shouldn’t Write ETL

#56

Post by achompas » Fri, Mar 18, 2016, 5:37 PM UTC

gtrubetskoy wrote: post #47

I agree with the beginning of the article, which describes the present state pretty well, the part about "better engineers than statisticians and better statisticians than engineers", etc. But then I disagree with the rest. The distinction between "Data Scientists" and "Engineers" is bogus, and the point about whether your data is "Big" is a red herring. In reality, there should not be any distinctions between "scien…

> If you're in this field, and consider yourself an "engineer" but your math sucks, go read up on all you can about mathematics and statistics, just like you did back when you were learning about programming, operating systems and networking.

This assumes availability of time. Obviously, given enough time, people could develop both top-tier engineering and DS skillsets!

Of course, if lots of free time were common we'd all be full-stack developers who also field sales calls and work on product strategy etc. etc.

Re: Engineers Shouldn’t Write ETL

#57

Post by drited » Fri, Mar 18, 2016, 5:38 PM UTC

saosebastiao wrote: post #38

RivieraKid wrote: post #35

Earlier quoted context omitted.

What technologies / architecture does Amazon use for business intelligence? I've just done a business intelligence course so I'm interested how do more "technology-centered" companies approach BI, whether it's the same thing I learned in the course (put everything into an integrated relational database if I simplify a lot).

Lots and lots of SQL (a very good thing IMO). This can come in the form of Oracle, Redshift, or any of the commonly available RDS databases (probably not SQL Server though). This is augmented with a lot of Big Data stuff, which used to be pretty diverse (Pig, Hive, raw Hadoop, etc.) but is moving very quickly towards a Spark-centric platform. There is occasionally some commercial software like Tableau/Microstrategy.…

Would you know why sql server generally isn't used?

Re: Engineers Shouldn’t Write ETL

#58

Post by pneumatics » Fri, Mar 18, 2016, 5:39 PM UTC

ignoramous wrote: post #39

darksaints wrote: post #18

Earlier quoted context omitted.

This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…

Fellow amazonian here. We switched from a massively distributed datastore (not to be named) to rodb for storage and found 10x improvement, not to mention eliminating cost and other head-aches; kind of expected since rodb is an embedded db...

What is rodb?

Re: Engineers Shouldn’t Write ETL

#59

Post by emcq » Fri, Mar 18, 2016, 5:41 PM UTC

mikestew wrote: post #5

... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…

My favorite is when you jump on a project and do a simple estimation of compute throughput for the highly complex distributed system, its something like hundreds of kilobytes per second. You could literally copy all the files to one computer and process faster than the web of broken parts. It becomes cancerous; to work around system slowness ever complex caching mechanisms and adhoc work is constructed.

I think part of the problem is many engineers can debug an application, but surprisingly few learn performance optimization and finding bottlenecks. This leads to an ignorance is bliss mindset where engineers assume they are doing things in reasonably performant ways and so the next step must be to scale, without even a simple estimate for throughput. It turns into bad software architecture and code debt that will cause high maintenance costs.

Re: Engineers Shouldn’t Write ETL

#60

Post by dxbydt » Fri, Mar 18, 2016, 5:44 PM UTC

gtrubetskoy wrote: post #47

I agree with the beginning of the article, which describes the present state pretty well, the part about "better engineers than statisticians and better statisticians than engineers", etc. But then I disagree with the rest. The distinction between "Data Scientists" and "Engineers" is bogus, and the point about whether your data is "Big" is a red herring. In reality, there should not be any distinctions between "scien…

This is a supremely ridiculous set of suggestions that has no merit whatsoever. Companies aren't libraries. They aren't paying you to sit and read books. There is an assigned dayjob, a set of tasks you have on your jira that you have to resolve by your deadlines, and that occupies the 8 hour workday ID you are doing any justice to it. So any reading you do is on the side, on your own time.

Furthermore, people have these roles precisely because of their talents and their choices. As a Data Scientist, most of what I do is read ML literature, build ML models and write technical reports in Tex on what worked and what didn't. The skills to do this were acquired over many painful years of graduate work in math, statistics, ML. To suggest somebody can just read their way through that material is quite laudable, but you are underestimating the difficulty by orders of magnitude. Essentially, you are suggesting that all of the graduate study and mentoring and homeworks and assignments and all that went into the learning process be condensed into a book which one can just plow through and become a DS. Well, good luck with that. By the same token, expecting me to have the same level of efficiency and passion as a data engineer when faced with a Hadoop/Oozie/Presto/Pig/kafka or what have you is silly. I don't care for these technologies and how to work them. I know it takes a really long time to get good at them - that's why the engineers get paid a lot of money and also get yelled at when the ETL job fails. Because it's a set of seriously valuable skills that were no doubt acquired over lots of time and practice. It's not like I can buy a book on these things, just read through them and suddenly I am a DE! I neither have the interest nor the time to do that.

>>the distinction between data scientists and data engineers is bogus

Not at all! Both DS and DE professionals do distinctly different work and conflating everything under 1 umbrella buys you nothing.