Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving product recommendations from Hadoop to Redshift saves us time and money

engineering.monetate.com

Post reply on HN
Print view View original

31–40 of 66 posts

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#31

Post by monstrado »

These type of articles baffle me, you're comparing a high-performance analytical database to a batch-orientated SQL engine.

The whole point behind these query engines on Hadoop (Hive, Presto, Impala, etc) is to separate the database from the query engine. With these engines you can project schemas over raw data in its original form, without having to load it into a table. With Redshift, or other similar analytical databases, you're forced to define a schema, and then load the data in row by row...bulk inserts are very slow in comparison to Hadoop technologies.

Regardless, Hive in general should nver be used for interactive analytics, that's not what it's intended for. Where Hive shines is when you can dump 250TB of raw text data into a folder and then run a SQL query to extract useful information out of it. The extracted data could then be loaded into a RDBMS like RedShift for real-time reporting.

With all that being said, if you want to run SQL queries on data in Hadoop at the speeds of Redshift, you should have used Impala with Parquet, which is known to be even faster than Redshift in many cases, and is based on the same technology Google uses (Dremel and F1). The benefits of keeping your data in Hadoop are enormous, not every problem can be solved using SQL. The same data you're querying with Impala could actually be used to do machine learning using Spark or Mahout. Maybe you want to start indexing one of your tables into Solr to provide search capabilities on a subset of your columns to your users...or maybe you want to use Giraph or Sparks' GraphX to do parallel graph computation. The data never moves, there's still only ONE copy of that data in Hadoop, and you can bring any kind of workload to it.

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#32

Post by sixdimensional »

endersshadow wrote: post #19
mattj wrote: post #15

I've gone through a similar transition (hive to redshift) in a very large scale data environment. Raw Hadoop / cascading is still very useful for more complicated workflows, but redshift is so vastly superior to hive it's not even funny. I thought I would miss adding my own UDFs, but this hasn't been an issue at all. I'm under the impression presto is a similar improvement, but I haven't spent any time with it. One h…

Just as a quick note: You can use Postgres libraries because Redshift is a slightly modified version Postgres 8.1 under the covers. In fact, almost all massively-parallel-processing (MPP) databases are Postgres under the covers (including Microsoft's PDW). It really speaks to how impressive Postgres is at scaling. Even old releases, like 8.1!

I work with the folks who built PDW. It's all SQL Server now. That said, I'm often amazed at how many commercial db products are based on Postgres and other open source dbs. Postgres has a nice page on their site showing all the products derived from it - for example, Netezza or Pivotal's Greenplum. Relational dbs and SQL (especially SQL) are far from dead.

Link: https://wiki.postgresql.org/wiki/PostgreSQL_derived_database...

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#34

Post by bitL »

jpat wrote: post #8
bitL wrote: post #4

Earlier quoted context omitted.

If you optimize for latency relational databases won't cut it.

I'm the author of the article. At Monetate, we've chosen our data warehouses to maximize throughput, rather than minimize latency. That's where something like Redshift really shines, it's great a large bulk ingests and running large queries relatively quickly, but awful at running lots of small queries quickly. On our busiest day last year, we ingested over a quarter billion page views across all of our clients' webs…

Thank you for sharing your experience! It's always inspiring to read well-written articles as is yours!

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#35

Post by bbrunner »

monstrado wrote: post #31

These type of articles baffle me, you're comparing a high-performance analytical database to a batch-orientated SQL engine. The whole point behind these query engines on Hadoop (Hive, Presto, Impala, etc) is to separate the database from the query engine. With these engines you can project schemas over raw data in its original form, without having to load it into a table. With Redshift, or other similar analytical da…

Redshift is an especially limited SQL engine considering it doesn't support UDFs. It is wicked fast, but what you get in speed you lose in flexibility. Current (well, February, but fairly current) benchmarks[0] place Impala and Shark (SQL on top of Spark) within grasp of Redshift while pulling data from disk and, for certain workloads, on par or faster than Redshift. This is without using a columnar file format.

Impala is impressive technology, but it does require you to run dedicated Impala daemons as it doesn't use map reduce under the hood. Shark is especially interesting, however, because it is fast AND build on top of spark, so you can run raw Spark jobs, SQL queries, graph processing and ML all on the same cluster. Shark currently uses Hive to generate it's query plans, but the Spark project is working on implementing it's own SQL engine called Catalyst[1] that promises to be a significant improvement.

[0] https://amplab.cs.berkeley.edu/benchmark/

[1] https://spark-summit.org/talk/armbrust-catalyst-a-query-opti...

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#36

Post by ulisesrmzroche »

personZ wrote: post #18
onion2k wrote: post #14

Earlier quoted context omitted.

It's often cheaper to use the "wrong" architecture than optimising the right architecture. I know I could have a CouchDB datastore searching a few GB with an afternoon of work. I imagine I could get MySQL fast enough with a few days of optimisation. In terms of time, which is by far the biggest cost in most development, CouchDB is the better option.

In terms of time, which is by far the biggest cost in most development, CouchDB is the better option. For a single, one-off utility, sure. For anything that you ever planned for production, that would be crazy . Just to be clear, the mentality that onion proposes (at least from my interpretation, though I apologize if I'm misunderstanding), usually justified under a gross misinterpretation of the "premature optimizat…

That's just like your opinion though. You ever used CouchDB in production before?

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#37

Post by monstrado »

bbrunner wrote: post #35
monstrado wrote: post #31

These type of articles baffle me, you're comparing a high-performance analytical database to a batch-orientated SQL engine. The whole point behind these query engines on Hadoop (Hive, Presto, Impala, etc) is to separate the database from the query engine. With these engines you can project schemas over raw data in its original form, without having to load it into a table. With Redshift, or other similar analytical da…

Redshift is an especially limited SQL engine considering it doesn't support UDFs. It is wicked fast, but what you get in speed you lose in flexibility. Current (well, February, but fairly current) benchmarks[0] place Impala and Shark (SQL on top of Spark) within grasp of Redshift while pulling data from disk and, for certain workloads, on par or faster than Redshift. This is without using a columnar file format. Impa…

Although I have a lot of respect for the amplab, they did not do their due diligence with that benchmark. Mainly for a few reasons, they didn't test using columnar storage in Hadoop (ORC / Parquet), which is what Redshift is using underneath (a proprietary columnar store). Also, the most complicated query they ran was a two table join, and from what I can tell, there wasn't any concurrent workload testing.

(disclaimer: I'm a Cloudera employee):

I recommend checking out the following blog, not because my employer wrote it, but because the guys behind the benchmark did an incredible job making the benchmark competitive. They also show metrics that a lot of the other people are not showing, for example concurrent workload capabilities, CPU efficiency, etc.

Impala, Hive (on Tez), Shark, Presto

http://blog.cloudera.com/blog/2014/05/new-sql-choices-in-the...

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#38

Post by verelo »

monstrado wrote: post #31

These type of articles baffle me, you're comparing a high-performance analytical database to a batch-orientated SQL engine. The whole point behind these query engines on Hadoop (Hive, Presto, Impala, etc) is to separate the database from the query engine. With these engines you can project schemas over raw data in its original form, without having to load it into a table. With Redshift, or other similar analytical da…

So we recently did this same switch. I can say that in our experience data pulls that were taking (on a good day) 6 hours now take under 10 seconds. There are some jobs we had that literally took days to run that now complete in minutes. Redshift is amazingly simple to use, crazy fast and so very cost effective.

Generally we're dealing with datasets that are around 1-3TB, and pretty well organized. Its just amazing how forgiving Redshift is when it comes to unusually written SQL and how useful it is to us as a business. Extracting serious insights was once a job that only a few people could do, now its something that anyone with a SQL background can manage.

Re: Moving product recommendations from Hadoop to Redshift saves us time and money

#39

Post by bjt »

endersshadow wrote: post #19
mattj wrote: post #15

I've gone through a similar transition (hive to redshift) in a very large scale data environment. Raw Hadoop / cascading is still very useful for more complicated workflows, but redshift is so vastly superior to hive it's not even funny. I thought I would miss adding my own UDFs, but this hasn't been an issue at all. I'm under the impression presto is a similar improvement, but I haven't spent any time with it. One h…

Just as a quick note: You can use Postgres libraries because Redshift is a slightly modified version Postgres 8.1 under the covers. In fact, almost all massively-parallel-processing (MPP) databases are Postgres under the covers (including Microsoft's PDW). It really speaks to how impressive Postgres is at scaling. Even old releases, like 8.1!

Anyone who likes Postgres and is looking for a good analytics DB should check out the cstore_fdw Postgres extension. [1] It allows Postgres to create and query files in the Optimized Row Columnar (ORC) format [2] from Hive.

It was created and recently open-sourced by Citus Data (YC S11), who've made it a key component of their MPP Postgres offering.

I don't work there. I'm just a fan.

[1] https://github.com/citusdata/cstore_fdw [2] http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.0.2/ds...

Post reply on HN