Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

61–70 of 178 posts

Re: Engineers Shouldn’t Write ETL

#61

Post by gtrubetskoy »

achompas wrote: post #56
gtrubetskoy wrote: post #47

I agree with the beginning of the article, which describes the present state pretty well, the part about "better engineers than statisticians and better statisticians than engineers", etc. But then I disagree with the rest. The distinction between "Data Scientists" and "Engineers" is bogus, and the point about whether your data is "Big" is a red herring. In reality, there should not be any distinctions between "scien…

> If you're in this field, and consider yourself an "engineer" but your math sucks, go read up on all you can about mathematics and statistics, just like you did back when you were learning about programming, operating systems and networking. This assumes availability of time. Obviously, given enough time, people could develop both top-tier engineering and DS skillsets! Of course, if lots of free time were common we'…

I don't know... Are you saying that someone like Donald Knuth had a lot of free time? Data problems are arguably the hardest problems out there and they require deep understanding of mathematics as well as computers, that's just the way it is. It does take time and effort and even may be a bit of talent as well, not everyone is cut out for it.

Re: Engineers Shouldn’t Write ETL

#62

Post by ams6110 »

mikestew wrote: post #7
throwawy31816 wrote: post #2

This author seriously needs to expand all of his TLIs (three-letter initialisms) the first time he uses them, as any writer worth his or her salt would do. There are those who may be interested in what he has to say, but can't follow because of assuming abbreviations.

Though I agree with you on expanding TLIs, if you have to have "ETL" defined for you, you probably won't get the "joke". And though this will come out more cynical than I intend, if you don't know the acronyms, then you probably won't be buying what Stitch Fix is selling. Filtering their funnel, maybe?

While possibly true, it's simply a courtesy to the reader to parenthetically define any acronym the first time it's used in a published piece of writing (of course this would not apply to internal emails, casual comments such as discussion forums here, etc.)

Re: Engineers Shouldn’t Write ETL

#63

Post by gtrubetskoy »

dxbydt wrote: post #60
gtrubetskoy wrote: post #47

I agree with the beginning of the article, which describes the present state pretty well, the part about "better engineers than statisticians and better statisticians than engineers", etc. But then I disagree with the rest. The distinction between "Data Scientists" and "Engineers" is bogus, and the point about whether your data is "Big" is a red herring. In reality, there should not be any distinctions between "scien…

This is a supremely ridiculous set of suggestions that has no merit whatsoever. Companies aren't libraries. They aren't paying you to sit and read books. There is an assigned dayjob, a set of tasks you have on your jira that you have to resolve by your deadlines, and that occupies the 8 hour workday ID you are doing any justice to it. So any reading you do is on the side, on your own time. Furthermore, people have th…

>> I don't care for these technologies and how to work them.

And I'd argue that this is precisely the problem. I personally care about all of it.

Re: Engineers Shouldn’t Write ETL

#64

Post by nmkridler »

v64 wrote: post #49

> The fundamental flaw that prevents the Thinker and Doer model from living up to its recruiting hype is the assumption that there exists an army of soulless non-mediocre Doer engineers who eagerly implement the ideas and vision of data scientists. There's a large, active community of engineers who specialize in data, whose job is to technologically enable data scientists the means to perform their analyses. I know t…

I completely disagree, data scientists who can not create the data they need are at a significant disadvantage to those who can. Our job is more than being able to analyze and interpret data. If you have someone in your organization that spends no time thinking about how they get the data, you need to fire them or reduce their salary.

Re: Engineers Shouldn’t Write ETL

#65

Post by ams6110 »

web007 wrote: post #34
lafay wrote: post #30

Is ETL really even necessary anymore? Why not just run fast ad hoc queries over the raw data with something like Google BigQuery?

Yeah - just Extract it from your MySQL / Mongo / Postgres / logfiles / whatever system it's in right now, Transform it into a CSV or whatever the input needs to be and Load it into BigQuery. Once it's there, you can do whatever you need!

On a smaller scale, the "q" utility has been a boon for me in the handling of ad-hoc delimited data files.

http://harelba.github.io/q/

Really one of the best things I've discovered in the past 5 years. Saves so much work compared to doing stuff with sed, awk, and the like.

Re: Engineers Shouldn’t Write ETL

#66

Post by [deleted] »

drited wrote: post #57
saosebastiao wrote: post #38

Earlier quoted context omitted.

Lots and lots of SQL (a very good thing IMO). This can come in the form of Oracle, Redshift, or any of the commonly available RDS databases (probably not SQL Server though). This is augmented with a lot of Big Data stuff, which used to be pretty diverse (Pig, Hive, raw Hadoop, etc.) but is moving very quickly towards a Spark-centric platform. There is occasionally some commercial software like Tableau/Microstrategy.…

Would you know why sql server generally isn't used?

[deleted]

Re: Engineers Shouldn’t Write ETL

#67

Post by sshykes »

drited wrote: post #57
saosebastiao wrote: post #38

Earlier quoted context omitted.

Lots and lots of SQL (a very good thing IMO). This can come in the form of Oracle, Redshift, or any of the commonly available RDS databases (probably not SQL Server though). This is augmented with a lot of Big Data stuff, which used to be pretty diverse (Pig, Hive, raw Hadoop, etc.) but is moving very quickly towards a Spark-centric platform. There is occasionally some commercial software like Tableau/Microstrategy.…

Would you know why sql server generally isn't used?

Probably because:

* SQL Server is proprietary

* SQL Server licenses are expensive

* SQL Server runs only on Windows (or least used to?)

Re: Engineers Shouldn’t Write ETL

#68

Post by sshykes »

pneumatics wrote: post #58
ignoramous wrote: post #39

Earlier quoted context omitted.

Fellow amazonian here. We switched from a massively distributed datastore (not to be named) to rodb for storage and found 10x improvement, not to mention eliminating cost and other head-aches; kind of expected since rodb is an embedded db...

What is rodb?

Good question. Is it this [0]?

[0] http://researcher.watson.ibm.com/researcher/view_group.php?i...

Re: Engineers Shouldn’t Write ETL

#70

Post by logicallee »

mikestew wrote: post #5

... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…

> data sets sooooo huge, that they would fit on my laptop

well yes this is an embarrassing fact for data scientists. A midrange stock macbook can easily handle a database of everyone on earth. In RAM. While you play a game without any dropped frames.

Post reply on HN