Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

151–160 of 178 posts

Re: Engineers Shouldn’t Write ETL

#151

Post by andrewflnr »

cookiecaper wrote: post #95
andrewflnr wrote: post #78

Earlier quoted context omitted.

But you can't say that the doer always deserves the credit either. Sometimes the idea is the hard part. Similar for blame. It doesn't work to make generalizations. You have to make a judgment call every time, and usually the answer will be a complicated mixture.

"Ideas" alone are almost never worth anything. You have to do the work to back it up. Everyone I know has about a dozen ideas (you hear them all the time as someone who makes ideas real). What matters is the technical skill to make the idea go from a fantasy to a reality semi-reminiscent of the idealized fantastic version, whether that skill is in business, accounting, programming, marketing, or whatever.

Yes, I've heard the "ideas are worthless" meme, I just don't entirely believe it. Ideas are easier to have, so a bigger fraction of them are worthless, but that's not actually a good measure of the "relative worth" of ideas and implementations, precisely because the mechanics of producing them is different. Good ideas, e.g. actual market opportunities, problem-solving breakthroughs, are rare and valuable. Insight is valuable.

You can execute as hard as you want, but if you're headed in the wrong direction, the only good thing that's going to happen is that you're going to learn when ideas really are important.

Re: Engineers Shouldn’t Write ETL

#152

Post by andrewflnr »

btilly wrote: post #91
andrewflnr wrote: post #78

Earlier quoted context omitted.

But you can't say that the doer always deserves the credit either. Sometimes the idea is the hard part. Similar for blame. It doesn't work to make generalizations. You have to make a judgment call every time, and usually the answer will be a complicated mixture.

And I didn't say that the doer always deserves the credit. As an example, who deserves more credit for the success of Apple, Steve Wozniak or Steve Jobs? Wozniak created the Apple I and most of the Apple II. But clearly Jobs' ideas built the current company. However these cases are the exception, not the rule. As a rule ideas are cheap, implementations are hard. And success has more to do with iterating on the implem…

  > However the problem is that actual credit for
  > success REALLY belongs with the
  > people who did the work.

  > And I didn't say that the doer always
  > deserves the credit.
You kind of did.

Re: Engineers Shouldn’t Write ETL

#153

Post by amzthrow23424 »

scottlocklin wrote: post #122
ignoramous wrote: post #39

Earlier quoted context omitted.

Fellow amazonian here. We switched from a massively distributed datastore (not to be named) to rodb for storage and found 10x improvement, not to mention eliminating cost and other head-aches; kind of expected since rodb is an embedded db...

I'd like to know what rodb is. Columnar DB? Columnar DBs often beat "big data" -I've beaten decent sized spark clusters in one thread of J.

Read only database. It is a hand optimized/compressed database engine that is used for big "once a day" data sets.

Re: Engineers Shouldn’t Write ETL

#154

Post by newjersey »

avon wrote: post #117
sshykes wrote: post #67

Earlier quoted context omitted.

Probably because: * SQL Server is proprietary * SQL Server licenses are expensive * SQL Server runs only on Windows (or least used to?)

Amazon actually does allow SQL Server. The poster saying it probably didn't was influenced by the fact that Azure, Microsoft's own cloud solution is an AWS competitor. To your own points: SQL Server is as proprietary as Oracle SQL Server is cheaper than Oracle SQL Server is being ported to Linux in 2017 :)

Amazon.com seems to strongly encourage the companies it acquires to use aws. Apparently, woot.com is fixing to rewrite the website in Java or something just so they do not have to use Windows. Apparently, it makes sense to them.

Re: Engineers Shouldn’t Write ETL

#155

Post by rspeer »

amzthrow23424 wrote: post #153
scottlocklin wrote: post #122

Earlier quoted context omitted.

I'd like to know what rodb is. Columnar DB? Columnar DBs often beat "big data" -I've beaten decent sized spark clusters in one thread of J.

Read only database. It is a hand optimized/compressed database engine that is used for big "once a day" data sets.

I'm Googling and finding a few things called "RODB" that don't quite match your descriptions. What in particular is it?

Or are you talking about just rolling your own format to dump your data into? I've done that, but I'd still appreciate if I could use something that someone else had put the thought into. (Something like "cdb" by Daniel J. Bernstein, but that's an old 32-bit library that's limited to 4 GB files.)

Re: Engineers Shouldn’t Write ETL

#156

Post by cookiecaper »

rukuu001 wrote: post #149
cookiecaper wrote: post #95

Earlier quoted context omitted.

"Ideas" alone are almost never worth anything. You have to do the work to back it up. Everyone I know has about a dozen ideas (you hear them all the time as someone who makes ideas real). What matters is the technical skill to make the idea go from a fantasy to a reality semi-reminiscent of the idealized fantastic version, whether that skill is in business, accounting, programming, marketing, or whatever.

Sure, an unrealised idea is next to worthless. The real pain is making a decision and expending resources on your challenging/risky idea. There's very little appetite for the responsibility and risk that come with big ideas (in a BigCo). Got the ability to think up new ideas, sell them within an organisation, and get them executed (hello 'doer') in a way that provides value to that organisation? You're gold, and wort…

>Got the ability to think up new ideas, sell them within an organisation, and get them executed (hello 'doer') in a way that provides value to that organisation?

No one can "have" this ability because it's transient. Unless you control the entire corporation (in which case you don't need to influence anyone else anyway), there is always someone who can come in and break your previously-perfect ability to "sell" your ideas inside the org. You're claiming that artful politicians (or, more blatantly, "good bullshit artists") are more valuable than skilled engineers. I don't believe that.

Re: Engineers Shouldn’t Write ETL

#157

Post by rspeer »

jjoonathan wrote: post #113
banku_brougham wrote: post #103

Earlier quoted context omitted.

thanks for the SQLite tip, I've been meaning to add it to my tool set. question: is SQLite incrementally helpful when I'm already comfortable with a local pgsql db to handle the use case you suggested? would SQLite be redundant for me in this case? question: between postgres and unix tools (sed, awk) is there reason to use SQLite?

Reasons to prefer sed/awk: you're in bash Reasons to prefer sqlite: it's easier to embed in an app, you want sort, join, split-apply-combine, scale, transactions, compression, etc. Reasons to prefer pgsql: sqlite's perf tools suck compared to pgsql (last time I got stuck anyway) and I'm sure there are lots of sql-isms that sqlite doesn't handle if that's your jam. EDIT: forgot everything-is-a-string in sqlite, just w…

Another reason to use Unix tools like sed and awk (and grep and sort and...) is that, if you just need to iterate over all your data and don't need a random-access index, they are really really fast.

Re: Engineers Shouldn’t Write ETL

#158

Post by cookiecaper »

hnal943 wrote: post #99
cookiecaper wrote: post #95

Earlier quoted context omitted.

"Ideas" alone are almost never worth anything. You have to do the work to back it up. Everyone I know has about a dozen ideas (you hear them all the time as someone who makes ideas real). What matters is the technical skill to make the idea go from a fantasy to a reality semi-reminiscent of the idealized fantastic version, whether that skill is in business, accounting, programming, marketing, or whatever.

Eh, this is true of start up business ideas, but not necessarily true in an enterprise that has hired a bunch of doers. Vision is rare and important.

I actually don't really agree. From all the medium-large sized business I've been on the inside of, as both a consultant and employee, it seems that a lot of people have a lot of pretty OK ideas about how to fix things within the org and where to take the products. In general, excluding a few noobs and whack-jobs, these ideas seem to be about as viable and plausible as the ideas that actually do get pushed down.

The valuable thing in a big company isn't the idea; it's the way the idea is executed. You'd be amazed what kind of nonsensical ideas will work if your team gets it just right.

Re: Engineers Shouldn’t Write ETL

#159

Post by rspeer »

vikiomega9 wrote: post #145
mikestew wrote: post #5

... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…

It depends a lot on what you're doing, but 2 TB in general seems a lot. If you have to perform an out-of-disk sort, you probably need a distributed setup. The funny thing is that most setups that use EMR defeat the data locality principle and I think that's the speedup people experience when they run it on a single laptop for example. Reading the original Google paper helped me a lot in understanding this.

If you have to perform an out-of-disk sort, buy another hard disk and now it's not out-of-disk anymore. This will suffice for nearly any data set you would ever need to sort.

A 4 TB drive costs about $120, and you'll spend way more than that on software development and extra computers if you do distributed computing when you don't need to.

Re: Engineers Shouldn’t Write ETL

#160

Post by saosebastiao »

ap22213 wrote: post #139
darksaints wrote: post #18

Earlier quoted context omitted.

This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…

How long does it take you to load 2 TB into SQLite? How long do queries take? I believe you, but I'm in disbelief that it could be close to as efficient as throwing into ram. I mean, an EMR cluster takes like 5 minutes to spin up. Where do I learn how to do this? I've tried loading a TiB (one table one index) into SQLite on disk before, and it took forever. Granted this was a couple years ago, but I must be doing som…

There was a little bit of unfair comparison. I didn't have to load the data over a busy network connection, and I didn't have to decrypt the data once it was loaded (I had the benefit of a firewall and the raw data on a USB3 hard drive). I think there was a conversion from CSV to parquet on the cluster as well. And the engineer who set up the cluster was multitasking so I'm sure there was some latency issues just from that. But my analysis still only took a few hours (5? Maybe 6?).

There are a handful of things that make a difference. First of all, don't use inserts, use the .import command. This alone is enough to saturate all the available write bandwidth on a 7200rpm drive. It is not transactional, so you don't have to worry about that...it bypasses the query engine entirely, really is more like a shell command that marshals data directly into the table's on disk representation. You can also disable journaling and increase page sizes for a tiny boost.

Once imported into SQLite you get the benefit of binary representation which (for my use case) really cut down on the dataset size for the read queries. I only had a single join and it was against a dimensional table that fit in memory, so indexes were small and took insignificant time to build. One single table scan with some aggregation, and that was it.

Post reply on HN