Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving product recommendations from Hadoop to Redshift saves us time and money

engineering.monetate.com

Post reply on HN
Print view View original

11–20 of 66 posts

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#11

Post by personZ »

ergest wrote: post #3

Is it me or are people switching to non-relational data warehouse architectures simply because it's en vogue? How many companies do you know that have enough data where a non-relational DW would actually make sense? I wonder, have we really pushed relational databases to their breaking point?

I wonder, have we really pushed relational databases to their breaking point?

The primary limitation of relational databases were traditionally that without expert level optimizations (which, realistically, data-focused organizations should have. But they very seldom do, especially in the start-up space), many queries would generate large numbers of effectively random IO. When you're rolling with magnetic drives, each drive offers maybe 60-150 IOPS, so this quickly becomes an enormous scaling problem. A large storage array offered maybe 2000 IOPS. Scaling becomes entirely about scaling IOPS, as CPU is seldom a limitation in databases.

Add that many firms were starting on EC2 which not only gave you minimal memory, it offered absolutely miserable IOPS performance.

Digg famously, and disastrously, solved this problem by essentially "denormalizing" every bit of data, enormously exploding the raw data they stored, but allowing for individual queries to be entirely localized, often served in a single, large IO: Instead of looking up all of your friends and finding the things they dug, the system would push every bit of data proactively to containers for every possible user. This is the model promoted by many advocates of alternative storage (e.g the advantage of MongoDb is always the "pull a single giant data bag versus pulling it together from various places").

If Kevin Rose dug something, it would update the "things my friends liked" containers for 40,000 or so of his friends, rather than having those 40,000 users check on-demand to see what each of their friends liked.

But they did that right when flash storage was coming into the mainstream. A technology that offers, on simple, inexpensive cards, 100s of thousands to millions of IOPS. Add that RAM has exploded, such that servers with 256GB of memory are very affordable (that was enough to put the entire universe of Digg's data in memory, where of course random IO is in the tens to hundreds of millions).

So now we're at a situation where having non-duplicated, highly relational database is often the highest performance, outside of all of its other advantages, because it fits in memory, and fits on economical flash storage. It has completely flipped the equation.

http://www.commitstrip.com/en/2014/06/03/the-problem-is-not-...

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#12

Post by meritt »

jgrahamc wrote: post #7
meritt wrote: post #6

Earlier quoted context omitted.

I've looked at and avoided doing anything serious with hdfs/mr for 6 years now. I'm glad some people are starting to realize that re-processing your entire dataset every single time you want to do something isn't very efficient. I'm still waiting for lightbulb moment where the usefulness of it really makes sense to me. Can anyone point me to a book or blog that discusses good uses of hadoop/map-reduce?

I'm waiting for the day people realize that materialized views in databases are awesome and decide to incorporate them into a framework.

At least if you're using Oracle they are, as it supports auto-refreshing. Postgres has only had them since 9.3 (and have to be manually refreshed). Meanwhile MySQL is still struggling with regular views.

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#13

Post by jpat »

lsb wrote: post #9

It's also unclear how many rows they're trying to do this on, and at what frequencies; that's the crux of what turns this from a small-to-medium-data problem, which you can easily solve on a large box with 10 lines of code, to a big data problem, which requires completely different tooling

In my testing of this query, I ran it against a time range that included over 40 million purchase lines, and our configuration of Redshift returned the result in ~6 minutes. That was much quicker than our legacy EMR implementation.

Currently, we update our product recommendations nightly. However, the speed up we see here from this reimplementation may allow us to update product recommendations more frequently.

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#14

Post by onion2k »

ergest wrote: post #3

Is it me or are people switching to non-relational data warehouse architectures simply because it's en vogue? How many companies do you know that have enough data where a non-relational DW would actually make sense? I wonder, have we really pushed relational databases to their breaking point?

It's often cheaper to use the "wrong" architecture than optimising the right architecture. I know I could have a CouchDB datastore searching a few GB with an afternoon of work. I imagine I could get MySQL fast enough with a few days of optimisation. In terms of time, which is by far the biggest cost in most development, CouchDB is the better option.

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#15

Post by mattj »

I've gone through a similar transition (hive to redshift) in a very large scale data environment. Raw Hadoop / cascading is still very useful for more complicated workflows, but redshift is so vastly superior to hive it's not even funny. I thought I would miss adding my own UDFs, but this hasn't been an issue at all. I'm under the impression presto is a similar improvement, but I haven't spent any time with it.

One huge advantage of redshift over hive: you can connect with plain old Postgres libraries, so you can build redshift results into your admin interfaces, one off scripts, and anywhere else you're fine trading a few seconds of latency for extra data.

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#16

Post by crb »

zatkin wrote: post #10

Wasn't Hadoop the first of it's kind in Big Data?

Hadoop was the first (major) open-source implementation of Google's MapReduce framework. http://research.google.com/archive/mapreduce.html

In terms of data warehousing and near-real-time query over Big Data, Google's framework for that is called "Dremel", http://research.google.com/pubs/pub36632.html

Google offer Dremel as a service known as BigQuery.

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#17

Post by growse »

mattj wrote: post #15

I've gone through a similar transition (hive to redshift) in a very large scale data environment. Raw Hadoop / cascading is still very useful for more complicated workflows, but redshift is so vastly superior to hive it's not even funny. I thought I would miss adding my own UDFs, but this hasn't been an issue at all. I'm under the impression presto is a similar improvement, but I haven't spent any time with it. One h…

I'm not surprised, given that my experiences with Hive are that it's extremely quirky and hardly ever the fastest way to do anything. Given the fact that people seem to be falling over themselves to reinvent better solutions to the same sorts of problems in the Hadoop space (see: Impala, Shark), I don't think I'm alone on that.

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#18

Post by personZ »

onion2k wrote: post #14
ergest wrote: post #3

Is it me or are people switching to non-relational data warehouse architectures simply because it's en vogue? How many companies do you know that have enough data where a non-relational DW would actually make sense? I wonder, have we really pushed relational databases to their breaking point?

It's often cheaper to use the "wrong" architecture than optimising the right architecture. I know I could have a CouchDB datastore searching a few GB with an afternoon of work. I imagine I could get MySQL fast enough with a few days of optimisation. In terms of time, which is by far the biggest cost in most development, CouchDB is the better option.

In terms of time, which is by far the biggest cost in most development, CouchDB is the better option.

For a single, one-off utility, sure. For anything that you ever planned for production, that would be crazy.

Just to be clear, the mentality that onion proposes (at least from my interpretation, though I apologize if I'm misunderstanding), usually justified under a gross misinterpretation of the "premature optimization" warning, is exactly how disaster implementations that end up failing or requiring enormous amounts of engineering time to try to triage and bandage into something usable.

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#19

Post by endersshadow »

mattj wrote: post #15

I've gone through a similar transition (hive to redshift) in a very large scale data environment. Raw Hadoop / cascading is still very useful for more complicated workflows, but redshift is so vastly superior to hive it's not even funny. I thought I would miss adding my own UDFs, but this hasn't been an issue at all. I'm under the impression presto is a similar improvement, but I haven't spent any time with it. One h…

Just as a quick note: You can use Postgres libraries because Redshift is a slightly modified version Postgres 8.1 under the covers. In fact, almost all massively-parallel-processing (MPP) databases are Postgres under the covers (including Microsoft's PDW). It really speaks to how impressive Postgres is at scaling. Even old releases, like 8.1!

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#20

Post by tomphoolery »

personZ wrote: post #11
ergest wrote: post #3

Is it me or are people switching to non-relational data warehouse architectures simply because it's en vogue? How many companies do you know that have enough data where a non-relational DW would actually make sense? I wonder, have we really pushed relational databases to their breaking point?

I wonder, have we really pushed relational databases to their breaking point? The primary limitation of relational databases were traditionally that without expert level optimizations (which, realistically, data-focused organizations should have. But they very seldom do, especially in the start-up space), many queries would generate large numbers of effectively random IO. When you're rolling with magnetic drives, eac…

The Digg thing is interesting...they pretty much took the complete opposite approach of Reddit, who basically store everything in two big SQL tables.
Post reply on HN