Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google launches Public Datasets program

cloud.google.com

Post reply on HN
Print view View original

11–20 of 98 posts

Re: Google launches Public Datasets program

#11

Post by kuschku »

vgt wrote: post #7
kuschku wrote: post #6

Can we extract the datasets to work with them on our own (cheaper to query) servers?

Of course! BigQuery export operations are free. Although compared with other cloud technologies, BigQuery is incredibly cost-effective :)

Well, I’ve checked the prices, and dedicated servers from old-style hosters are still far cheaper if you’re going to query data 24/7.

Re: Google launches Public Datasets program

#12

Post by frikk »

polartx wrote: post #4

I work and play in data. By far the best resource I've encountered is https://app.enigma.io/ Signups are free. The aggregated public data is plentiful and easily discovered, indexed, filtered, and exported. Free account have API limitations, but as far as govt data is concerned, I don't find that its updated often enough to peg my API rate limiter anyway.

Awesome. Do you have any other resources to share? I run occasional local hackathons and am always looking for inspiring data sets.

Re: Google launches Public Datasets program

#14

Post by kyrra »

nl wrote: post #10

This is a much more useful list: https://www.reddit.com/r/bigquery/wiki/datasets I'm not sure why Google hosts it on Reddit. There's some interesting (and more up-to-date) stuff on there.

This page may interest you: https://support.google.com/cloud/answer/3466163?hl=en

I think the Google Cloud team watches the community and helps promote the forums that have the most active for a given product.

Re: Google launches Public Datasets program

#15

Post by jonesb6 »

ortusdux wrote: post #2

"HACKER NEWS - A dataset that contains all stories and comments from Hacker News since its launch in 2006." I know what I'm doing this weekend.

Would be interesting to find out the value (upvotes) of sponsored content vs. unsponsered content, as well as comments from green users vs normal users.

Re: Google launches Public Datasets program

#17

Post by pbh101 »

I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of concern was "Why won't Google just shut this down when the going gets rough?" The question was mostly batted away along the lines of "this is lunch money to Google anyway." But just a couple years later, it did indeed bite the dust.

Covered in Wired: http://www.wired.com/2008/01/google-to-provi/

Update: And the shutdown article, 11 months later: http://www.wired.com/2008/12/googlescienceda/

Re: Google launches Public Datasets program

#19

Post by fhoffa »

ortusdux wrote: post #2

"HACKER NEWS - A dataset that contains all stories and comments from Hacker News since its launch in 2006." I know what I'm doing this weekend.

Some resources to get you started:

- How to use the Hacker News dataset https://medium.com/google-cloud/big-data-stories-in-seconds-...

- Discussion of the HN dataset announcement here https://news.ycombinator.com/item?id=10440502

- An iPython notebook: https://github.com/fhoffa/notebooks/blob/master/analyzing%20...

- More: http://debarghyadas.com/writes/looking-back-at-9-years-of-ha...

Disclaimer: I'm Felipe Hoffa, and I work at Google. (https://twitter.com/felipehoffa)

Re: Google launches Public Datasets program

#20

Post by kuschku »

Bytes wrote: post #13
kuschku wrote: post #6

Can we extract the datasets to work with them on our own (cheaper to query) servers?

Most of the datasets seem to be available to download here https://www.reddit.com/r/bigquery/wiki/datasets .

After looking at those:

What’s "big" about any of them?

That’s literally a standard database size, and can be easily done with any postgres install.

All of that data combined can be stored and quickly queried on just 10 dedicated servers, for an overall price of 200€/month.

And with "all" I mean "all 20TB".

And with "quickly" I mean "faster than a network request to Google".

________________

Due to rate limiting, I can't answer you at the moment, vgt. So my answer will happen here, inline:

> Well, like this:

> If someone wants to decide if they want to build a business on bigquery, they want to evaluate it's performance on an average data set before for free.

> Assuming the test data is thought for similar cases it has to be of similar size to the average dataset used with BigQuery.

Post reply on HN