Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data-Processing Frameworks Benchmark: Redshift, Hive, Shark, Impala

amplab.cs.berkeley.edu

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Data-Processing Frameworks Benchmark: Redshift, Hive, Shark, Impala

#11

Post by hnriot »

AmiiJewels wrote: post #3

I applaud the effort but is this really "big data" - the largest data sets they seem to test are ~150GB, that would fit comfortably on my Mac Book Pro a number of times over. Many of these systems being tested are designed to scale efficiently when the data starts peaking > 5TB and therefore I am dubious about the median response time results - things that work well for small datasets (where small is defined as < 1TB…

Big Data is less about the size, and more about characterizing how you want to deal with the data. Straightforward structures queries are the purview of RDBMS, whereas complex analytics is for "Big Data" frameworks like those evaluated.

Re: Data-Processing Frameworks Benchmark: Redshift, Hive, Shark, Impala

#12

Post by vosper »

espeed wrote: post #10

Spark is a big deal ( http://spark.incubator.apache.org/ ). It's a next-gen open source cluster-computing system built on top of the Berekely Data Analytics Stack (BDAS - https://amplab.cs.berkeley.edu/software/ ), which includes Mesos, Spark, SparkStreaming, Shark, and GraphX (to name a few). Mesos is the foundation of the stack, and Spark started out as a research project because they needed something to run on Mes…

I've been following Spark for a while, and you excellent comment has motivated me to take another look - thanks you for the reading materials.

Have you used Spark in production?

Re: Data-Processing Frameworks Benchmark: Redshift, Hive, Shark, Impala

#13

Post by bcoates »

ceyhunkazel wrote: post #2

SAP HANA would be a better option than Redshift. You can get cloud version of HANA. It support R, JavaScript, ArgGIS and more SQL data types.

Redshift is $0.43/TB-hour, it looks like HANA on AWS is around $59/TB-hour. You get a lot for the money (HANA software, in-memory, tons more CPU) but your workload had better really need it at that price difference!

Re: Data-Processing Frameworks Benchmark: Redshift, Hive, Shark, Impala

#14

Post by keithgabryelski »

CurtMonash wrote: post #5

The author had a terrible brain cramp in the sentence "Redshift uses columnar compression which allows it to bypass a field which is not used in the query." That totally confuses columnar compression with columnar I/O, an error I've been railing against for several years, e.g. in http://www.dbms2.com/2011/02/06/columnar-compression-databas... (I.e., ever since Oracle tried to popularize the confusion.) But this is a…

I also don't see a schema that is tuned for redshift.

There is no description of a sortkey or a distribution key (let alone which compression encoding was used)

And at least for the first query you could use UNLOAD instead of select (depending on how you are managing the data coming out of redshift it might be a reasonable solution and doesn't force all your data through the leader node constrained by whatever client driver you are using to read the results).

Instead of trying to select out of redshift this much data -- select the data into another table or (again) use unload.

Re: Data-Processing Frameworks Benchmark: Redshift, Hive, Shark, Impala

#15

Post by izendejas »

espeed wrote: post #10

Spark is a big deal ( http://spark.incubator.apache.org/ ). It's a next-gen open source cluster-computing system built on top of the Berekely Data Analytics Stack (BDAS - https://amplab.cs.berkeley.edu/software/ ), which includes Mesos, Spark, SparkStreaming, Shark, and GraphX (to name a few). Mesos is the foundation of the stack, and Spark started out as a research project because they needed something to run on Mes…

There was some big news this week regarding Spark: http://gigaom.com/2013/09/25/databricks-raises-14m-from-andr...

The creators Matei Zaharia, Ion Stoica et al raised a substantial amount. With Tachyon (in-memory file system that supports lineage and is thus fairly robust) and more recently MLBase, one should look at Spark beyond its excellent performance and really look at the overall package and versatility it provides.

I think many people somewhat pointlessly get caught up in arguments about what constitutes big data and what doesn't. As someone who's used Spark substantially for machine learning as well as other complex types of processing beyond your standard joins, filters, etc Spark is incredibly useful even for a few GB of data because it allows one to iterate rapidly.

With MLBase and all, I think Spark will really have an impact because your average engineer will be able to run some standard ML algorithms out of the box at scale. That is huge. That's what matters with data (big or not) -- it's the insights you can gain.

Edited: typos. Also, some more on mlbase: http://www.mlbase.org which will be released tomorrow if am not mistaken.

Re: Data-Processing Frameworks Benchmark: Redshift, Hive, Shark, Impala

#16

Post by ceyhunkazel »

bcoates wrote: post #13
ceyhunkazel wrote: post #2

SAP HANA would be a better option than Redshift. You can get cloud version of HANA. It support R, JavaScript, ArgGIS and more SQL data types.

Redshift is $0.43/TB-hour, it looks like HANA on AWS is around $59/TB-hour. You get a lot for the money (HANA software, in-memory, tons more CPU) but your workload had better really need it at that price difference!

I think you calculated wrong. https://aws.amazon.com/marketplace/pp/B009KA3CRY

EC2 Instance Type Software EC2 Total

8XL cc2.8xlarge $0.99/hr $2.50/hr $3.49/hr

EBS Storage Fees $0.10 / GB / Month for Standard EBS Storage

Re: Data-Processing Frameworks Benchmark: Redshift, Hive, Shark, Impala

#17

Post by espeed »

vosper wrote: post #12
espeed wrote: post #10

Spark is a big deal ( http://spark.incubator.apache.org/ ). It's a next-gen open source cluster-computing system built on top of the Berekely Data Analytics Stack (BDAS - https://amplab.cs.berkeley.edu/software/ ), which includes Mesos, Spark, SparkStreaming, Shark, and GraphX (to name a few). Mesos is the foundation of the stack, and Spark started out as a research project because they needed something to run on Mes…

I've been following Spark for a while, and you excellent comment has motivated me to take another look - thanks you for the reading materials. Have you used Spark in production?

We will be using it in production for a current project.

Re: Data-Processing Frameworks Benchmark: Redshift, Hive, Shark, Impala

#18

Post by espeed »

izendejas wrote: post #15
espeed wrote: post #10

Spark is a big deal ( http://spark.incubator.apache.org/ ). It's a next-gen open source cluster-computing system built on top of the Berekely Data Analytics Stack (BDAS - https://amplab.cs.berkeley.edu/software/ ), which includes Mesos, Spark, SparkStreaming, Shark, and GraphX (to name a few). Mesos is the foundation of the stack, and Spark started out as a research project because they needed something to run on Mes…

There was some big news this week regarding Spark: http://gigaom.com/2013/09/25/databricks-raises-14m-from-andr... The creators Matei Zaharia, Ion Stoica et al raised a substantial amount. With Tachyon (in-memory file system that supports lineage and is thus fairly robust) and more recently MLBase, one should look at Spark beyond its excellent performance and really look at the overall package and versatility it prov…

Congrats to Matei and Prof Stoica on the new company.
Post reply on HN