Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google launches Public Datasets program

cloud.google.com

Post reply on HN
Print view View original

21–30 of 98 posts

Re: Google launches Public Datasets program

#21

Post by vgt »

pbh101 wrote: post #17

I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of concern was "Why won't Google just shut this down when the going gets rough?" The question was mostly batted away along the lines of "this is lunch money to Google anyway." But just a couple years later, it did indeed bite the dust. Covered in Wired: http://www.…

Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google relies on it heavily.

Externally, companies like Spotify and Kabam Games rely heavily on BigQuery as well:

https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5s...

https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5s...

So the story of Google Cloud products, which are paid, supported, and Enterprise-grade, and which are covered by a clear 1 year deprecation policy, are very different from the story of free services.

Public Datasets is just a function of BigQuery's unique ACL functionality.

Re: Google launches Public Datasets program

#23

Post by vgt »

kuschku wrote: post #20
Bytes wrote: post #13

Earlier quoted context omitted.

Most of the datasets seem to be available to download here https://www.reddit.com/r/bigquery/wiki/datasets .

After looking at those: What’s "big" about any of them? That’s literally a standard database size, and can be easily done with any postgres install. All of that data combined can be stored and quickly queried on just 10 dedicated servers, for an overall price of 200€/month. And with "all" I mean "all 20TB". And with "quickly" I mean "faster than a network request to Google". ________________ Due to rate limiting, I c…

Apologies, but where do you find the implication that these datasets are "big"? There is "big" in BigQuery, so apologize if you misunderstood.

We try to keep datasets reasonable, so that folks can get the most out of the BigQuery free pricing tier :)

If you want bigger, here's an example of a 10TB dataset:

https://cloud.google.com/genomics/data/1000-genomes

Re: Google launches Public Datasets program

#24

Post by fhoffa »

pbh101 wrote: post #17

I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of concern was "Why won't Google just shut this down when the going gets rough?" The question was mostly batted away along the lines of "this is lunch money to Google anyway." But just a couple years later, it did indeed bite the dust. Covered in Wired: http://www.…

2008.

I remember those days. GMail was still in beta and ' a big area of concern was "Why won't Google just shut this down when the going gets rough?" '

8 years later GMail has a billion users, and not going anywhere but up.

I haven't been to the future (yet), but I'll happily take any bet you have against BigQuery.

Disclaimer: I'm Felipe Hoffa, and I work at Google. (https://twitter.com/felipehoffa)

Re: Google launches Public Datasets program

#25

Post by vijayr »

polartx wrote: post #4

I work and play in data. By far the best resource I've encountered is https://app.enigma.io/ Signups are free. The aggregated public data is plentiful and easily discovered, indexed, filtered, and exported. Free account have API limitations, but as far as govt data is concerned, I don't find that its updated often enough to peg my API rate limiter anyway.

can you talk more about your work? what kind of projects do you work on? what tools do you use other than enigma?

Re: Google launches Public Datasets program

#26

Post by [deleted] »

jonesb6 wrote: post #15
ortusdux wrote: post #2

"HACKER NEWS - A dataset that contains all stories and comments from Hacker News since its launch in 2006." I know what I'm doing this weekend.

Would be interesting to find out the value (upvotes) of sponsored content vs. unsponsered content, as well as comments from green users vs normal users.

[deleted]

Re: Google launches Public Datasets program

#27

Post by fhoffa »

kuschku wrote: post #20
Bytes wrote: post #13

Earlier quoted context omitted.

Most of the datasets seem to be available to download here https://www.reddit.com/r/bigquery/wiki/datasets .

After looking at those: What’s "big" about any of them? That’s literally a standard database size, and can be easily done with any postgres install. All of that data combined can be stored and quickly queried on just 10 dedicated servers, for an overall price of 200€/month. And with "all" I mean "all 20TB". And with "quickly" I mean "faster than a network request to Google". ________________ Due to rate limiting, I c…

Hi Janne!

I'm sure you'll enjoy a challenge.

Let's talk about Wikidata. Can you download this 8 GB compressed file?

https://dumps.wikimedia.org/wikidatawiki/entities/latest-all...

I want to know the id of the JSON row which length is 102 bytes.

It took me 4 seconds with BigQuery - how can we improve this with "any postgres install"?

https://lists.wikimedia.org/pipermail/wikidata/2016-March/00...

Re: Google launches Public Datasets program

#29

Post by kuschku »

fhoffa wrote: post #27
kuschku wrote: post #20

Earlier quoted context omitted.

After looking at those: What’s "big" about any of them? That’s literally a standard database size, and can be easily done with any postgres install. All of that data combined can be stored and quickly queried on just 10 dedicated servers, for an overall price of 200€/month. And with "all" I mean "all 20TB". And with "quickly" I mean "faster than a network request to Google". ________________ Due to rate limiting, I c…

Hi Janne! I'm sure you'll enjoy a challenge. Let's talk about Wikidata. Can you download this 8 GB compressed file? https://dumps.wikimedia.org/wikidatawiki/entities/latest-all... I want to know the id of the JSON row which length is 102 bytes. It took me 4 seconds with BigQuery - how can we improve this with "any postgres install"? https://lists.wikimedia.org/pipermail/wikidata/2016-March/00...

What you just did isn't how data is usually processed — a single 8GB JSON never occurs in real life cases, except maybe as transitional medium while you migrate from one database to another.

In most cases, you'll have it stored in a relational or graph database, easily accessible.

Especially in the sciences.

But sure, I can go through the data and tell you the amount of cats with specific properties, or similar.

I personally use pgsql currently to analyse data from Android crash reports.

Like, "list me all exceptions which have reports from more than 50 uniqie users which do not all use Samsung phones"

Or, my favourite, as I store for each report the exception, for which I store the stack trace elements, for which I store methods, files, lines of files and classes,

Show me a breakdown of operating systems of users affected by exceptions occuring in this method which have sent more than 40 fatal crashes each in the past month.

It manages that faster than just the latency to bigquery would be — 18ms over a test dataset of several gigabyte by now.

________________

Seriously, @dang, can you maybe consider deactivating downvotes unless people also post a comment explaining why they consider a comment not constructive?

Getting mass downvotes within of seconds of posting is very annoying, and just destroys the discussion culture.

Post reply on HN