Live data from Hacker News

Google launches Public Datasets program

cloud.google.com

41–50 of 98 posts

Re: Google launches Public Datasets program

#41
post #28
post #9

I get 404s for all Looker iframes...

where are you seeing Looker being used?

Looker is powering all the iframes (any 404s should be fixed now). Bigquery is hosting the data, but Looker is generating the queries and visualizations of the data.

Re: Google launches Public Datasets program

#42
post #2

"HACKER NEWS - A dataset that contains all stories and comments from Hacker News since its launch in 2006." I know what I'm doing this weekend.

I'm dying to get comment scores, not just submission scores.

There's a lot you could do with that to find the best comments, which is really why HN is so awesome.

Re: Google launches Public Datasets program

#43
post #42
post #2

"HACKER NEWS - A dataset that contains all stories and comments from Hacker News since its launch in 2006." I know what I'm doing this weekend.

I'm dying to get comment scores, not just submission scores. There's a lot you could do with that to find the best comments, which is really why HN is so awesome.

> There's a lot you could do with that to find the best comments.

In any given situation, how often does best comment you could write coincide with the one that would get you the most upvotes? I'm guessing not frequently.

Re: Google launches Public Datasets program

#44
post #10

This is a much more useful list: https://www.reddit.com/r/bigquery/wiki/datasets I'm not sure why Google hosts it on Reddit. There's some interesting (and more up-to-date) stuff on there.

That's a list of random data from random people, curated by Googler felipehoffa in an unofficial capacity.

The ones in the submitted page are maintained by Google itself in the public-data project.

Re: Google launches Public Datasets program

#45
post #24
post #17

I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of concern was "Why won't Google just shut this down when the going gets rough?" The question was mostly batted away along the lines of "this is lunch money to Google anyway." But just a couple years later, it did indeed bite the dust. Covered in Wired: http://www.…

2008. I remember those days. GMail was still in beta and ' a big area of concern was "Why won't Google just shut this down when the going gets rough?" ' 8 years later GMail has a billion users, and not going anywhere but up. I haven't been to the future (yet), but I'll happily take any bet you have against BigQuery. Disclaimer: I'm Felipe Hoffa, and I work at Google. ( https://twitter.com/felipehoffa )

Can you give me an update on Wave and Orkut? I never use this Reader thing, but I hear others do.

Re: Google launches Public Datasets program

#46
post #42

Earlier quoted context omitted.

I'm dying to get comment scores, not just submission scores. There's a lot you could do with that to find the best comments, which is really why HN is so awesome.

> There's a lot you could do with that to find the best comments. In any given situation, how often does best comment you could write coincide with the one that would get you the most upvotes? I'm guessing not frequently.

Your best is different from his/her best.

Re: Google launches Public Datasets program

#47
post #42
post #2

"HACKER NEWS - A dataset that contains all stories and comments from Hacker News since its launch in 2006." I know what I'm doing this weekend.

I'm dying to get comment scores, not just submission scores. There's a lot you could do with that to find the best comments, which is really why HN is so awesome.

Blame HN for that. It was removed from public access right after I made a blog post about it, although it was coincidental I think.

Re: Google launches Public Datasets program

#48
post #40
post #21

Earlier quoted context omitted.

Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google relies on it heavily. Externally, companies like Spotify and Kabam Games rely heavily on BigQuery as well: https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5s... https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5s... So the story of Google Clou…

Google employees using something internally doesn't mean that it won't go away for everyone else... Also, they could decide to honor existing contracts and stop issuing new ones...

BigQuery is a truly fully-managed service. This means that SREs are on call 24/7. It's not software that one downloads. Hence it really doesn't make a lot of sense to have it available to just a few folks, incremental overhead is low.

I'd love to hear industry standards around deprecation of paid services. This is really a way of doing business in the entire world, and I'm not sure why Google is being held to an unreasonable and impossible standard of "never deprecate anything ever!". For example, here is Microsoft's:

https://en.wikipedia.org/wiki/Category:Discontinued_Microsof...

Re: Google launches Public Datasets program

#49
post #39
post #5

One large difference between this program and alternative programs is that data already resides in Google BigQuery: - You do not need to spin up a database to work with BigQuery - You can simply start writing SQL on top of BigQuery - You may leverage Dataflow and MapReduce connectors to work with this data directly in Hadoop, Spark, or Dataflow - BigQuery has a free tier - one Terabyte of data processed per month Fin…

And if you've got public data in BigQuery that you want to make explorable/shareable/visualizable to anybody (no SQL required), let Looker know and we'll see what we can do. (disclosure: work at Looker)

Looker has been a great launch partner for this program. Really helped us develop and deliver great interactive dashboards.

Re: Google launches Public Datasets program

#50
post #24

Earlier quoted context omitted.

2008. I remember those days. GMail was still in beta and ' a big area of concern was "Why won't Google just shut this down when the going gets rough?" ' 8 years later GMail has a billion users, and not going anywhere but up. I haven't been to the future (yet), but I'll happily take any bet you have against BigQuery. Disclaimer: I'm Felipe Hoffa, and I work at Google. ( https://twitter.com/felipehoffa )

Can you give me an update on Wave and Orkut? I never use this Reader thing, but I hear others do.

[deleted]
Post reply on HN