Engineers Shouldn’t Write ETL

Post by TheLogothete » Fri, Mar 18, 2016, 6:26 PM UTC

nmkridler wrote: post #64

v64 wrote: post #49

> The fundamental flaw that prevents the Thinker and Doer model from living up to its recruiting hype is the assumption that there exists an army of soulless non-mediocre Doer engineers who eagerly implement the ideas and vision of data scientists. There's a large, active community of engineers who specialize in data, whose job is to technologically enable data scientists the means to perform their analyses. I know t…

I completely disagree, data scientists who can not create the data they need are at a significant disadvantage to those who can. Our job is more than being able to analyze and interpret data. If you have someone in your organization that spends no time thinking about how they get the data, you need to fire them or reduce their salary.

Post by jessaustin » Fri, Mar 18, 2016, 6:33 PM UTC

cowardlydragon wrote: post #69

Everybody who writes code does ETL in some form. That's the fundamental action of computation. Read, compute, write. This is a stupid article.

Post by v64 » Fri, Mar 18, 2016, 6:37 PM UTC

nmkridler wrote: post #64

v64 wrote: post #49

> The fundamental flaw that prevents the Thinker and Doer model from living up to its recruiting hype is the assumption that there exists an army of soulless non-mediocre Doer engineers who eagerly implement the ideas and vision of data scientists. There's a large, active community of engineers who specialize in data, whose job is to technologically enable data scientists the means to perform their analyses. I know t…

I completely disagree, data scientists who can not create the data they need are at a significant disadvantage to those who can. Our job is more than being able to analyze and interpret data. If you have someone in your organization that spends no time thinking about how they get the data, you need to fire them or reduce their salary.

Post by khc » Fri, Mar 18, 2016, 6:47 PM UTC

ams6110 wrote: post #65

web007 wrote: post #34

Earlier quoted context omitted.

Yeah - just Extract it from your MySQL / Mongo / Postgres / logfiles / whatever system it's in right now, Transform it into a CSV or whatever the input needs to be and Load it into BigQuery. Once it's there, you can do whatever you need!

On a smaller scale, the "q" utility has been a boon for me in the handling of ad-hoc delimited data files. http://harelba.github.io/q/ Really one of the best things I've discovered in the past 5 years. Saves so much work compared to doing stuff with sed, awk, and the like.

Post by ksenzee » Fri, Mar 18, 2016, 7:20 PM UTC

mikestew wrote: post #7

throwawy31816 wrote: post #2

This author seriously needs to expand all of his TLIs (three-letter initialisms) the first time he uses them, as any writer worth his or her salt would do. There are those who may be interested in what he has to say, but can't follow because of assuming abbreviations.

Though I agree with you on expanding TLIs, if you have to have "ETL" defined for you, you probably won't get the "joke". And though this will come out more cynical than I intend, if you don't know the acronyms, then you probably won't be buying what Stitch Fix is selling. Filtering their funnel, maybe?