Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

161–170 of 178 posts

Re: Engineers Shouldn’t Write ETL

#161

Post by optimusclimb »

amake wrote: post #138
yeukhon wrote: post #134

Earlier quoted context omitted.

There was a HN story in which the author had the same "big data claim" (it was his intention to show there is very little big data in the real world, for most people out there) but instead he only needed to process couple GBs. He just used standard Unix commands and the performance was proven to be incredibly awesome. Here is an example for SQLite: https://news.ycombinator.com/item?id=9359568

I think you're talking about this: http://aadrake.com/command-line-tools-can-be-235x-faster-tha... I quite enjoyed it as well.

That guy's problem set allowed him to filter out massive percentages of his data from the get go.

Re: Engineers Shouldn’t Write ETL

#162

Post by Terr_ »

bazqux2 wrote: post #120
dizzystar wrote: post #37

Nobody enjoys writing and maintaining data pipelines or ETL. It’s the industry’s ultimate hot potato. It really shouldn’t come as a surprise then that ETL engineering roles are the archetypal breeding ground of mediocrity. There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. This is like... your opinion. Some people…

I love ETL. I've worked with real Big Data (5PB+) and working on the data pipeline was my favorite part. The feeling you get when you rewrite a job to run 1000x faster so the company can make way more money.

That implies the work is in a revenue-center or that you're at the rare company that isn't myopically focused on sales.

Re: Engineers Shouldn’t Write ETL

#163

Post by [deleted] »

rspeer wrote: post #155
amzthrow23424 wrote: post #153

Earlier quoted context omitted.

Read only database. It is a hand optimized/compressed database engine that is used for big "once a day" data sets.

I'm Googling and finding a few things called "RODB" that don't quite match your descriptions. What in particular is it? Or are you talking about just rolling your own format to dump your data into? I've done that, but I'd still appreciate if I could use something that someone else had put the thought into. (Something like "cdb" by Daniel J. Bernstein, but that's an old 32-bit library that's limited to 4 GB files.)

[deleted]

Re: Engineers Shouldn’t Write ETL

#164

Post by StudyAnimal »

ETL is just a small part of it, and engineers should probably have more of a role than they do. My last project was a data warehouse one, where the thing was obviously slapped together by PowerCenter users. They thought it was all about the ETL, They forgot the other 95%, how to engineer large scale, complex, maintainable software solutions.

Sure, let the powecenter users "write the ETL", but then they need to get the heck out of the way and let the big boys actually build the warehouse.

Re: Engineers Shouldn’t Write ETL

#165

Post by x0x0 »

darksaints wrote: post #18
mikestew wrote: post #5

... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…

This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…

I worked at a place that shall remain anonymous. An engineer later terminated for cause spent almost 2 months setting up a hadoop cluster with all the stuff that goes along with it to analyze some log data. When a senior engineer manager finally got incredibly tired of waiting for the results, said senior engineer manager wrote the desired analysis in an afternoon in ruby, and it ran in a couple hours mostly limited by the spindle speed on his laptop drive.

The productivity factor of ruby or python in a single memory space vs hadoop is at least 10x.

Re: Engineers Shouldn’t Write ETL

#166

Post by TheLogothete »

Terr_ wrote: post #162
bazqux2 wrote: post #120

Earlier quoted context omitted.

I love ETL. I've worked with real Big Data (5PB+) and working on the data pipeline was my favorite part. The feeling you get when you rewrite a job to run 1000x faster so the company can make way more money.

That implies the work is in a revenue-center or that you're at the rare company that isn't myopically focused on sales.

No, it doesn't. If you make an M/R job work 1000 times more efficiently you save money for the company. It doesn't matter if you are a profit center or a cost center.

Re: Engineers Shouldn’t Write ETL

#168

Post by kakoni »

saosebastiao wrote: post #38
RivieraKid wrote: post #35

Earlier quoted context omitted.

What technologies / architecture does Amazon use for business intelligence? I've just done a business intelligence course so I'm interested how do more "technology-centered" companies approach BI, whether it's the same thing I learned in the course (put everything into an integrated relational database if I simplify a lot).

Lots and lots of SQL (a very good thing IMO). This can come in the form of Oracle, Redshift, or any of the commonly available RDS databases (probably not SQL Server though). This is augmented with a lot of Big Data stuff, which used to be pretty diverse (Pig, Hive, raw Hadoop, etc.) but is moving very quickly towards a Spark-centric platform. There is occasionally some commercial software like Tableau/Microstrategy.…

Do you have somesort of framework for those ETL jobs? How do you handle ETL dependencies/workflow?(=thinking spotify's luigi[1] here)

[1] https://github.com/spotify/luigi

Re: Engineers Shouldn’t Write ETL

#170

Post by mattexx »

The best-case outcome of many efforts of data scientists is an artifact meant for a machine consumer, not a human one.

I posit this outcome is absolutely necessary for any data science project to be worthwhile in any organization.

In the case where the project produces a report and goes no further, you still need the data and code for reproducibility, one of the main principles of the scientific method [1].

In the case where the project gets handed off to engineers to re-implement, reproducibility is even more critical, since the engineers best effort to reproduce the code will almost certainly not be successful the first time, and you will need to validate many versions of the production model. Doing this by hand even once it wasteful, doing so many times is tragically so.

In the case where the data scientists can produce a service worthy of production use, kudos!! But understand the caveat that in truly big data or big compute flows, this outcome remains highly unlikely.

[1] https://en.wikipedia.org/wiki/Reproducibility

Post reply on HN