Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

71–80 of 178 posts

Re: Engineers Shouldn’t Write ETL

#71

Post by lomnakkus »

darksaints wrote: post #18
mikestew wrote: post #5

... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…

This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…

The really fun aspect of this for me, personally, is that I've been doing computers for ages (25 yrs or so?) now and 3TiB still intuitively feels like a massive amount of data even though I think I have something like 6TiB free space on my home server disks... which, in total, didn't even cost as much as a months' grocery shopping. Sometimes it really takes effort to rid yourself of these old intuitions that don't really work any more.

(I'm getting better at it!)

EDIT: And my desktop machine has -- let's see -- about 500000 times the RAM my first computer had. Truly astounding if you think about it.

Re: Engineers Shouldn’t Write ETL

#72

Post by AndrewUnmuted »

> Most companies structure their data science departments into 3 groups:

> Data scientists ... aka “the thinkers”

> Data engineers ... aka "the doers"

> Infrastructure engineers ... aka "the plumbers"

The author is clearly not an infrastructure engineer.

Re: Engineers Shouldn’t Write ETL

#73

Post by systems »

i would bet that the author must be thinking that those who disagree with him are ... mediocre engineers or developers

personally, i think, there is nothing wrong with being average .. people with average skills built great things

mediocre is just a mean way to say average

Re: Engineers Shouldn’t Write ETL

#74

Post by dllthomas »

AndrewUnmuted wrote: post #72

> Most companies structure their data science departments into 3 groups: > Data scientists ... aka “the thinkers” > Data engineers ... aka "the doers" > Infrastructure engineers ... aka "the plumbers" The author is clearly not an infrastructure engineer.

Why? I have certainly described work I've done as "plumbing".

Re: Engineers Shouldn’t Write ETL

#75

Post by [deleted] »

AndrewUnmuted wrote: post #72

> Most companies structure their data science departments into 3 groups: > Data scientists ... aka “the thinkers” > Data engineers ... aka "the doers" > Infrastructure engineers ... aka "the plumbers" The author is clearly not an infrastructure engineer.

[deleted]

Re: Engineers Shouldn’t Write ETL

#76

Post by gedrap »

mikestew wrote: post #5

... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…

This is very very true, very often when it comes to more than a gigabyte of data or more then a hundred of queries a second. Quite a few inexperienced devs suddenly think it's big data they've been reading about and FINALLY they can play with it!

With all the hype around 'big data' and all that crap, many people seem to forget how far you can go with plain simple SQL when it's properly configured, and not talking about some complicated optimizations, just solid fundamentals. And no problem if you can't do it, things like Amazon RDS will help you.

Re: Engineers Shouldn’t Write ETL

#77

Post by arikfr »

saosebastiao wrote: post #38
RivieraKid wrote: post #35

Earlier quoted context omitted.

What technologies / architecture does Amazon use for business intelligence? I've just done a business intelligence course so I'm interested how do more "technology-centered" companies approach BI, whether it's the same thing I learned in the course (put everything into an integrated relational database if I simplify a lot).

Lots and lots of SQL (a very good thing IMO). This can come in the form of Oracle, Redshift, or any of the commonly available RDS databases (probably not SQL Server though). This is augmented with a lot of Big Data stuff, which used to be pretty diverse (Pig, Hive, raw Hadoop, etc.) but is moving very quickly towards a Spark-centric platform. There is occasionally some commercial software like Tableau/Microstrategy.…

Are you using Re:dash[1]? Seems like a good fit if you're doing lots of SQL and working with many different databases.

[1] http://redash.io/

Re: Engineers Shouldn’t Write ETL

#78

Post by andrewflnr »

btilly wrote: post #53

The thinker/doer problem goes way back. In most organizations the person who thinks of something gets the lion's share of the credit, and the person who implements it does the lion's share of the work. And if it turns out to be a bad idea, the thinker can always blame a bad implementation, thereby passing the lion's share of the blame to the doer. I've seen careers made and broken based on whether people got to play…

But you can't say that the doer always deserves the credit either. Sometimes the idea is the hard part. Similar for blame. It doesn't work to make generalizations. You have to make a judgment call every time, and usually the answer will be a complicated mixture.

Re: Engineers Shouldn’t Write ETL

#79

Post by leblancfg »

dxbydt wrote: post #60
gtrubetskoy wrote: post #47

I agree with the beginning of the article, which describes the present state pretty well, the part about "better engineers than statisticians and better statisticians than engineers", etc. But then I disagree with the rest. The distinction between "Data Scientists" and "Engineers" is bogus, and the point about whether your data is "Big" is a red herring. In reality, there should not be any distinctions between "scien…

This is a supremely ridiculous set of suggestions that has no merit whatsoever. Companies aren't libraries. They aren't paying you to sit and read books. There is an assigned dayjob, a set of tasks you have on your jira that you have to resolve by your deadlines, and that occupies the 8 hour workday ID you are doing any justice to it. So any reading you do is on the side, on your own time. Furthermore, people have th…

>> Companies aren't libraries. They aren't paying you to sit and read books.

That stroke you've brushed is too wide. Smart employers will have some of the money they're paying an employee going towards learning... and if they're really smart, they can even measure their ROI. Leads to less turnover, and better long-term vision for their projects.

I get your point, but give someone passionate enough 6 months in a new work environment, and with a decent mentor, and you might find they become surprisingly adept at it. The hard part is hiring for the capability to learn (fast).

Re: Engineers Shouldn’t Write ETL

#80

Post by chubot »

darksaints wrote: post #18
mikestew wrote: post #5

... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…

This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…

That's awesome... I'm working with some 15-20 GB sqlite databases. Though 2 TB sounds kind of big?

Was it 2 TB before compression? Because sqlite does blow up the data size over the raw data usually (depending on the original format obviously). It can be kind of wasteful, and I ended up storing some fields as compressed JSON for this reason (that actually beats the sqlite format).

Also, the sqlite insert speed can be much slower than the disk's sequential write speed (even if you make sure you're not committing/flushing on every row, and if you have no indices to update, etc.)

So I think inserting and laying on the data could be nontrivial. But the queries should be fast as long as they are indexed. In theory, sqlite queries should be slower for a lot of use cases because it is row-oriented, but in practice distributed systems usually add 10x overhead themselves anyway...

Post reply on HN