Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google BigQuery brings Big Data analytics to all businesses

googledevelopers.blogspot.in

Post reply on HN
Print view View original

11–15 of 15 posts

Re: Google BigQuery brings Big Data analytics to all businesses

#11

Post by DenisM »

haberman wrote: post #7
DenisM wrote: post #4

Earlier quoted context omitted.

So, what are the publicly available data sets? I see there is wikipedia in one of your screenshots, but short of that I couldn't find a list. I think if I saw something enticing I would sign up just to play with it.

You can find the list of public data sets and descriptions of them here: https://developers.google.com/bigquery/docs/sample-tables

Thanks.

It's a bit thin, so I suggest you guys pump a lot of public datasets into it, and then do a series of blog posts about "look what you can discover from these public datasets with out awesome Q engine in a matter of seconds".

Re: Google BigQuery brings Big Data analytics to all businesses

#12

Post by mattmiller »

haberman wrote: post #3

Hi everyone, I work on the BigQuery team. From a technical perspective, think of this as an append-only cloud SQL service built on Dremel ( http://research.google.com/pubs/pub36632.html ). You upload huge data sets, we run SQL queries over them in seconds, even for billions of rows. And without having to specify/build any indexes (we actually scan the billions of rows, though only the columns we need to). All interac…

I would love to use it but my company would veto this product based on security concerns. At a minimum we would require a VPN connection to the cloud and the ability to limit (or cut off entirely) access to the web interface. You guys could have a huge product if these concerns are addressed, but based on Google's history I do not think they will be.

Also, does anyone know how this performs compared to Hive?

Re: Google BigQuery brings Big Data analytics to all businesses

#14

Post by mwhooker »

mattmiller wrote: post #12
haberman wrote: post #3

Hi everyone, I work on the BigQuery team. From a technical perspective, think of this as an append-only cloud SQL service built on Dremel ( http://research.google.com/pubs/pub36632.html ). You upload huge data sets, we run SQL queries over them in seconds, even for billions of rows. And without having to specify/build any indexes (we actually scan the billions of rows, though only the columns we need to). All interac…

I would love to use it but my company would veto this product based on security concerns. At a minimum we would require a VPN connection to the cloud and the ability to limit (or cut off entirely) access to the web interface. You guys could have a huge product if these concerns are addressed, but based on Google's history I do not think they will be. Also, does anyone know how this performs compared to Hive?

just started playing around with it. We've been using hive on EMR with tables stored in S3 (json formatted). using a single m1.large to run run queries over an hour of data was taking 10-15 minutes. Bigquery returns the same query in seconds. For example, extracting referrer domains on big query:

  > Query complete (7.6s elapsed, 583 MB processed)
granted that's with an under-provisioned emr "cluster", so I don't want to assign too much meaning to the results, but they are promising.

I'll run some more comparisons on a larger cluster and update later.

Re: Google BigQuery brings Big Data analytics to all businesses

#15

Post by JPKab »

pappnase12 wrote: post #9
JPKab wrote: post #2

This unquestionably lowers the barrier to entry for crunching large data sets. I'm looking forward to messing around with it. Are there any other alternatives to this service? Something like a PigAsAService or HiveAsAService offering?

Well, we are working on a project that provides Hive (and Hadoop Streaming) as a service. It's http://www.hadoopondemand.com and uses amazon ec2. We have just started our private beta and you are very welcome to join. And there is also amazon's offering EMR ( http://aws.amazon.com/elasticmapreduce/ ) which also provides an interface to Hive and Pig. EDIT: link to amazon's offering

Thanks. I look forward to checking out your project.
Post reply on HN