Google BigQuery brings Big Data analytics to all businesses

#1

Post by boundlessdreamz » Tue, May 01, 2012, 5:29 PM UTC

Google BigQuery brings Big Data analytics to all businesses
googledevelopers.blogspot.in

Re: Google BigQuery brings Big Data analytics to all businesses

#2

Post by JPKab » Tue, May 01, 2012, 7:48 PM UTC

This unquestionably lowers the barrier to entry for crunching large data sets. I'm looking forward to messing around with it. Are there any other alternatives to this service? Something like a PigAsAService or HiveAsAService offering?

Re: Google BigQuery brings Big Data analytics to all businesses

#3

Post by haberman » Tue, May 01, 2012, 8:37 PM UTC

Hi everyone, I work on the BigQuery team. From a technical perspective, think of this as an append-only cloud SQL service built on Dremel (http://research.google.com/pubs/pub36632.html). You upload huge data sets, we run SQL queries over them in seconds, even for billions of rows. And without having to specify/build any indexes (we actually scan the billions of rows, though only the columns we need to).

All interaction with the system happens through REST interfaces. Even our own UI uses only our publicly-available REST APIs.

There's a certain amount of free quota available. If you sign up you can try queries against public data sets like Wikipedia edits. Also it looks like the GitHub guys have been experimenting with analyzing GitHub data with BigQuery: https://github.com/blog/1112-data-at-github

I've just joined the team recently, but I really believe in what we're doing. I'd be happy to answer any questions I can.

Re: Google BigQuery brings Big Data analytics to all businesses

#4

Post by DenisM » Tue, May 01, 2012, 8:42 PM UTC

haberman wrote: post #3

Hi everyone, I work on the BigQuery team. From a technical perspective, think of this as an append-only cloud SQL service built on Dremel ( http://research.google.com/pubs/pub36632.html ). You upload huge data sets, we run SQL queries over them in seconds, even for billions of rows. And without having to specify/build any indexes (we actually scan the billions of rows, though only the columns we need to). All interac…

So, what are the publicly available data sets? I see there is wikipedia in one of your screenshots, but short of that I couldn't find a list. I think if I saw something enticing I would sign up just to play with it.

Re: Google BigQuery brings Big Data analytics to all businesses

#5

Post by oliverkofoed » Tue, May 01, 2012, 8:47 PM UTC

With all the 'spring cleaning' going on recently at google, my main concern would be the likelihood of this service staying available permanently.

Re: Google BigQuery brings Big Data analytics to all businesses

#6

Post by [deleted] » Tue, May 01, 2012, 8:51 PM UTC

DenisM wrote: post #4

haberman wrote: post #3

Hi everyone, I work on the BigQuery team. From a technical perspective, think of this as an append-only cloud SQL service built on Dremel ( http://research.google.com/pubs/pub36632.html ). You upload huge data sets, we run SQL queries over them in seconds, even for billions of rows. And without having to specify/build any indexes (we actually scan the billions of rows, though only the columns we need to). All interac…

So, what are the publicly available data sets? I see there is wikipedia in one of your screenshots, but short of that I couldn't find a list. I think if I saw something enticing I would sign up just to play with it.

[deleted]

Re: Google BigQuery brings Big Data analytics to all businesses

#7

Post by haberman » Tue, May 01, 2012, 8:52 PM UTC

DenisM wrote: post #4

haberman wrote: post #3

Hi everyone, I work on the BigQuery team. From a technical perspective, think of this as an append-only cloud SQL service built on Dremel ( http://research.google.com/pubs/pub36632.html ). You upload huge data sets, we run SQL queries over them in seconds, even for billions of rows. And without having to specify/build any indexes (we actually scan the billions of rows, though only the columns we need to). All interac…

So, what are the publicly available data sets? I see there is wikipedia in one of your screenshots, but short of that I couldn't find a list. I think if I saw something enticing I would sign up just to play with it.

You can find the list of public data sets and descriptions of them here: https://developers.google.com/bigquery/docs/sample-tables

Re: Google BigQuery brings Big Data analytics to all businesses

#8

Post by pgrote » Tue, May 01, 2012, 9:05 PM UTC

oliverkofoed wrote: post #5

With all the 'spring cleaning' going on recently at google, my main concern would be the likelihood of this service staying available permanently.

As a pay service focused on businesses, I think they would keep it going.

I cannot get to it right now. :)

"Error: Server Error The server encountered an error and could not complete your request. If the problem persists, please report your problem and mention this error message and the query that caused it."

Re: Google BigQuery brings Big Data analytics to all businesses

#9

Post by pappnase12 » Tue, May 01, 2012, 10:05 PM UTC

JPKab wrote: post #2

This unquestionably lowers the barrier to entry for crunching large data sets. I'm looking forward to messing around with it. Are there any other alternatives to this service? Something like a PigAsAService or HiveAsAService offering?

Well, we are working on a project that provides Hive (and Hadoop Streaming) as a service. It's http://www.hadoopondemand.com and uses amazon ec2. We have just started our private beta and you are very welcome to join. And there is also amazon's offering EMR (http://aws.amazon.com/elasticmapreduce/) which also provides an interface to Hive and Pig.

EDIT: link to amazon's offering

Re: Google BigQuery brings Big Data analytics to all businesses

#10

Post by batista » Tue, May 01, 2012, 10:23 PM UTC

pgrote wrote: post #8

oliverkofoed wrote: post #5

With all the 'spring cleaning' going on recently at google, my main concern would be the likelihood of this service staying available permanently.

As a pay service focused on businesses, I think they would keep it going. I cannot get to it right now. :) "Error: Server Error The server encountered an error and could not complete your request. If the problem persists, please report your problem and mention this error message and the query that caused it."

>As a pay service focused on businesses, I think they would keep it going.

Unless if not enough businesses end up paying for it, so if yours does use it and they cancel it, you're screwed. Or if Google decides that while it makes a nice revenue, they'd rather killer to concentrate on something else...

That's the problem with putting out tons of products (including highly touted stuff like Wave) and then killing them, nobody trust you to maintain a product their business will depend on anymore... Contrast that with Amazon AWS.