Moving product recommendations from Hadoop to Redshift saves us time and money
engineering.monetate.com
Moving product recommendations from Hadoop to Redshift saves us time and money
1–10 of 66 posts
Re: Moving product recommendations from Hadoop to Redshift saves us time and money
#2I chuckled when I read "We have a legacy data warehouse based in Hive and Elastic MapReduce, with backing data stores in S3.". I guess things have come full circle. It wasn't long ago that a relational database solution would have been "legacy".
Re: Moving product recommendations from Hadoop to Redshift saves us time and money
#3Is it me or are people switching to non-relational data warehouse architectures simply because it's en vogue? How many companies do you know that have enough data where a non-relational DW would actually make sense? I wonder, have we really pushed relational databases to their breaking point?
Re: Moving product recommendations from Hadoop to Redshift saves us time and money
#4Is it me or are people switching to non-relational data warehouse architectures simply because it's en vogue? How many companies do you know that have enough data where a non-relational DW would actually make sense? I wonder, have we really pushed relational databases to their breaking point?
If you optimize for latency relational databases won't cut it.
Re: Moving product recommendations from Hadoop to Redshift saves us time and money
#5Is it me or are people switching to non-relational data warehouse architectures simply because it's en vogue? How many companies do you know that have enough data where a non-relational DW would actually make sense? I wonder, have we really pushed relational databases to their breaking point?
If you optimize for latency relational databases won't cut it.
This might be the most inaccurate statement on the internet.
Re: Moving product recommendations from Hadoop to Redshift saves us time and money
#6Is it me or are people switching to non-relational data warehouse architectures simply because it's en vogue? How many companies do you know that have enough data where a non-relational DW would actually make sense? I wonder, have we really pushed relational databases to their breaking point?
I've looked at and avoided doing anything serious with hdfs/mr for 6 years now. I'm glad some people are starting to realize that re-processing your entire dataset every single time you want to do something isn't very efficient. I'm still waiting for lightbulb moment where the usefulness of it really makes sense to me.
Can anyone point me to a book or blog that discusses good uses of hadoop/map-reduce?
Re: Moving product recommendations from Hadoop to Redshift saves us time and money
#7Is it me or are people switching to non-relational data warehouse architectures simply because it's en vogue? How many companies do you know that have enough data where a non-relational DW would actually make sense? I wonder, have we really pushed relational databases to their breaking point?
I've looked at and avoided doing anything serious with hdfs/mr for 6 years now. I'm glad some people are starting to realize that re-processing your entire dataset every single time you want to do something isn't very efficient. I'm still waiting for lightbulb moment where the usefulness of it really makes sense to me. Can anyone point me to a book or blog that discusses good uses of hadoop/map-reduce?
I'm waiting for the day people realize that materialized views in databases are awesome and decide to incorporate them into a framework.
Re: Moving product recommendations from Hadoop to Redshift saves us time and money
#8Is it me or are people switching to non-relational data warehouse architectures simply because it's en vogue? How many companies do you know that have enough data where a non-relational DW would actually make sense? I wonder, have we really pushed relational databases to their breaking point?
If you optimize for latency relational databases won't cut it.
I'm the author of the article. At Monetate, we've chosen our data warehouses to maximize throughput, rather than minimize latency. That's where something like Redshift really shines, it's great a large bulk ingests and running large queries relatively quickly, but awful at running lots of small queries quickly.
On our busiest day last year, we ingested over a quarter billion page views across all of our clients' websites. I'm sure someone has made MySQL scale to that volume, but for us Redshift has been working great for a relatively low price point.
Re: Moving product recommendations from Hadoop to Redshift saves us time and money
#9It's also unclear how many rows they're trying to do this on, and at what frequencies; that's the crux of what turns this from a small-to-medium-data problem, which you can easily solve on a large box with 10 lines of code, to a big data problem, which requires completely different tooling
Re: Moving product recommendations from Hadoop to Redshift saves us time and money
#10Wasn't Hadoop the first of it's kind in Big Data?