Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

131–140 of 178 posts

Re: Engineers Shouldn’t Write ETL

#131

Post by aidenn0 »

lomnakkus wrote: post #112
7952 wrote: post #97

Earlier quoted context omitted.

I can remember going through this with a 10MB file about 15 year ago. It felt like a lot after growing up with floppy disks. But even a modest CPU could iterate over it quickly, I just didn't realise. I just assumed I would need to process it in a database!

Exactly! On a somewhat related note: The original Another World[1] would probably fit into the caches that your CPU has as a matter of course these days. [1] https://www.youtube.com/watch?v=Zgkf6wooDmw

My first hard drive was only 2MB larger than the caches on my CPU.

Re: Engineers Shouldn’t Write ETL

#132

Post by MechanicalTwerk »

dizzystar wrote: post #37

Nobody enjoys writing and maintaining data pipelines or ETL. It’s the industry’s ultimate hot potato. It really shouldn’t come as a surprise then that ETL engineering roles are the archetypal breeding ground of mediocrity. There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. This is like... your opinion. Some people…

Could not agree more. With all this talk of data and ideas, you would think he could include some data to back up his ideas. He can't, because this is all just opinion.

Re: Engineers Shouldn’t Write ETL

#133

Post by hadley »

v64 wrote: post #85
nmkridler wrote: post #64

Earlier quoted context omitted.

I completely disagree, data scientists who can not create the data they need are at a significant disadvantage to those who can. Our job is more than being able to analyze and interpret data. If you have someone in your organization that spends no time thinking about how they get the data, you need to fire them or reduce their salary.

The data scientists I work with are statistics PhDs. The extent of their programming knowledge is R and SQL. What are they supposed to do if the data they need to analyze is only available through a SOAP API you log into with OAuth, and they need to log in once a day to retrieve the latest day of data? Unless you're a software engineer, you probably don't have the skillset necessary to easily get that data. The data…

SOAP + Oauth is a weird combination but you could definitely work with it in R.

Re: Engineers Shouldn’t Write ETL

#134

Post by yeukhon »

Buttons840 wrote: post #82
darksaints wrote: post #18

Earlier quoted context omitted.

This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…

Me and a coworker were laughing at the parent comment, and I told him: "I guarantee that somewhere, sometime, an engineer has been like 'hay guys, I loaded our big data into SQLite on my laptop and it ended up being faster than our fancy cluster'". We then joked that the engineer would be fired a few weeks later for not being a "cultural fit". A few minutes later you commented with your story. I hope you didn't get f…

There was a HN story in which the author had the same "big data claim" (it was his intention to show there is very little big data in the real world, for most people out there) but instead he only needed to process couple GBs. He just used standard Unix commands and the performance was proven to be incredibly awesome.

Here is an example for SQLite: https://news.ycombinator.com/item?id=9359568

Re: Engineers Shouldn’t Write ETL

#135

Post by romming »

mikestew wrote: post #5

... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…

I think there's a more common reason why companies end up with "awful-to-work-with messes": ETL is deceptively simple.

Moving data from A to B and applying some transformations on the way through seems like a straightforward engineering task. However, creating a system that is fault-tolerant, handles data source changes, surfaces errors in a meaningful way, requires little maintenance, etc. is hard. Getting to a level of abstraction where data scientists can build on top of it in a way that doesn't require development skills is harder.

I don't think most data engineers are mediocre or find their job boring. The expectation from management is that ETL doesn't require significant effort is unrealistic, and leads to a technology gap between developers and scientists that tends to be filled with ad-hoc scripting and poor processes.

Disclosure: I'm the founder of Etleap[1], where we're creating tools to make ETL better for data teams.

[1] http://etleap.com/

Re: Engineers Shouldn’t Write ETL

#136

Post by v64 »

hadley wrote: post #133
v64 wrote: post #85

Earlier quoted context omitted.

The data scientists I work with are statistics PhDs. The extent of their programming knowledge is R and SQL. What are they supposed to do if the data they need to analyze is only available through a SOAP API you log into with OAuth, and they need to log in once a day to retrieve the latest day of data? Unless you're a software engineer, you probably don't have the skillset necessary to easily get that data. The data…

SOAP + Oauth is a weird combination but you could definitely work with it in R.

I just randomly picked two of the most painful protocols I could think of :) It doesn't surprise me though, I feel like I can't go a workday without hearing the phrase "Oh, actually, I can do that in R"

Re: Engineers Shouldn’t Write ETL

#137

Post by dunkelheit »

In other words division of labor does not work quite so well for a data science department as for a pin factory. The proposed solution (letting data scientists code more) is not radical enough in my opinion. Why not muddle the roles even further? Let everybody feel the pains that people in the other roles experience. Foster empathy and personal connections. Let developers talk to the users and vice versa.

I worked at a company where distinction between the roles was emphasized by physical separation, presumably so that they won't interfere with each others day-to-day duties. The downside is that each group starts caring about their particular thing only, feeling that they are the ones who really keep the place running and other groups are bozos doing their job incredibly poorly.

Re: Engineers Shouldn’t Write ETL

#138

Post by amake »

yeukhon wrote: post #134
Buttons840 wrote: post #82

Earlier quoted context omitted.

Me and a coworker were laughing at the parent comment, and I told him: "I guarantee that somewhere, sometime, an engineer has been like 'hay guys, I loaded our big data into SQLite on my laptop and it ended up being faster than our fancy cluster'". We then joked that the engineer would be fired a few weeks later for not being a "cultural fit". A few minutes later you commented with your story. I hope you didn't get f…

There was a HN story in which the author had the same "big data claim" (it was his intention to show there is very little big data in the real world, for most people out there) but instead he only needed to process couple GBs. He just used standard Unix commands and the performance was proven to be incredibly awesome. Here is an example for SQLite: https://news.ycombinator.com/item?id=9359568

I think you're talking about this: http://aadrake.com/command-line-tools-can-be-235x-faster-tha...

I quite enjoyed it as well.

Re: Engineers Shouldn’t Write ETL

#139

Post by ap22213 »

darksaints wrote: post #18
mikestew wrote: post #5

... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…

This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…

How long does it take you to load 2 TB into SQLite? How long do queries take? I believe you, but I'm in disbelief that it could be close to as efficient as throwing into ram. I mean, an EMR cluster takes like 5 minutes to spin up.

Where do I learn how to do this? I've tried loading a TiB (one table one index) into SQLite on disk before, and it took forever. Granted this was a couple years ago, but I must be doing something fundamentally wrong.

I want to try this out. I've got 6TiB here of uncompressed CSV, 32 GiB ram. Is this something I could start tonight and complete a few queries before bed?

Actually, out of curiosity, I looked it up on the sqlite site. If I'm reading the docs correctly, with atomic sync turned off, I should expect 50,000 inserts per second. So, with my data set of 50B rows, I should expect to have it all loaded in ... Just 13 days. What am I missing?

Re: Engineers Shouldn’t Write ETL

#140

Post by yeukhon »

amake wrote: post #138
yeukhon wrote: post #134

Earlier quoted context omitted.

There was a HN story in which the author had the same "big data claim" (it was his intention to show there is very little big data in the real world, for most people out there) but instead he only needed to process couple GBs. He just used standard Unix commands and the performance was proven to be incredibly awesome. Here is an example for SQLite: https://news.ycombinator.com/item?id=9359568

I think you're talking about this: http://aadrake.com/command-line-tools-can-be-235x-faster-tha... I quite enjoyed it as well.

Wow thanks! I couldn't find it in search :-) glad you found it.
Post reply on HN