Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google launches Public Datasets program

cloud.google.com

Post reply on HN
Print view View original

71–80 of 98 posts

Re: Google launches Public Datasets program

#71

Post by dingaling »

vgt wrote: post #53
philovivero wrote: post #45

Earlier quoted context omitted.

Can you give me an update on Wave and Orkut? I never use this Reader thing, but I hear others do.

I'm happy to respond to every single person that makes this argument in perpetuity :) There's a difference between experimental free services and Enterprise-grade SLA'd SLO'd fully-supported paid services with a very clear and recorded deprecation policy and an army of customers with contracts and full support from Google CEO and chairman.

Indeed, but who is paying to maintain 546GB of online storage ( plus backups ) for Reddit comments in BigQuery, for example?

If the answer is "Google" then I think people are still right to be cautious.

Or to invert the question; could I put 1TB of my own 'interesting' data into BigQuery and have Google maintain it in perpetuity for free? If not, then why are any of these datasets considered safer?

Re: Google launches Public Datasets program

#72

Post by greglindahl »

fhoffa wrote: post #24
pbh101 wrote: post #17

I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of concern was "Why won't Google just shut this down when the going gets rough?" The question was mostly batted away along the lines of "this is lunch money to Google anyway." But just a couple years later, it did indeed bite the dust. Covered in Wired: http://www.…

2008. I remember those days. GMail was still in beta and ' a big area of concern was "Why won't Google just shut this down when the going gets rough?" ' 8 years later GMail has a billion users, and not going anywhere but up. I haven't been to the future (yet), but I'll happily take any bet you have against BigQuery. Disclaimer: I'm Felipe Hoffa, and I work at Google. ( https://twitter.com/felipehoffa )

gmail is a low-margin business, compared to search advertising. Does it have any non-creepy way to make huge margins? If not, why is getting big mean anything but danger?

Re: Google launches Public Datasets program

#73

Post by bambax »

fhoffa wrote: post #19
ortusdux wrote: post #2

"HACKER NEWS - A dataset that contains all stories and comments from Hacker News since its launch in 2006." I know what I'm doing this weekend.

Some resources to get you started: - How to use the Hacker News dataset https://medium.com/google-cloud/big-data-stories-in-seconds-... - Discussion of the HN dataset announcement here https://news.ycombinator.com/item?id=10440502 - An iPython notebook: https://github.com/fhoffa/notebooks/blob/master/analyzing%20... - More: http://debarghyadas.com/writes/looking-back-at-9-years-of-ha... Disclaimer: I'm Felipe Hoffa,…

How fresh is the data, and how often is it refreshed? It doesn't seem to be described anywhere...

Re: Google launches Public Datasets program

#75

Post by ignoramous »

vgt wrote: post #63
awqrre wrote: post #54

Earlier quoted context omitted.

I was not blaming anybody, it's possible that a service would be more valuable to a company if it's kept internally.

Ah yes, point well taken. I think we've seen this historically at Google, but things have changed in the past few years.

Primarily as a reaction to AWS and Azure?

Google announced Colossus in 2011, is that going to make it to GCP?

Re: Google launches Public Datasets program

#76

Post by nl »

jpatokal wrote: post #44
nl wrote: post #10

This is a much more useful list: https://www.reddit.com/r/bigquery/wiki/datasets I'm not sure why Google hosts it on Reddit. There's some interesting (and more up-to-date) stuff on there.

That's a list of random data from random people, curated by Googler felipehoffa in an unofficial capacity. The ones in the submitted page are maintained by Google itself in the public-data project.

I believe that the GDelt, Freebase and Genomics tables (at least) are officially supported by Google.

Re: Google launches Public Datasets program

#77

Post by lbn »

incepted wrote: post #68
vgt wrote: post #21

Earlier quoted context omitted.

Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google relies on it heavily. Externally, companies like Spotify and Kabam Games rely heavily on BigQuery as well: https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5s... https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5s... So the story of Google Clou…

> It's simply not going away because Google relies on it heavily. Still doesn't mean Google will maintain the public access to this database. They might just as well shut it down and limit its access to Googlers. Like they already did a few years ago.

When you use these datasets you pay for how much the query has to scan. For example, if you do a select on a single 8 byte column with 1 billion rows in the table you are paying for 8GB.

Every month the first TB is free but if you go over that you have to pay $5 per TB. They are not exactly doing this for free.

Re: Google launches Public Datasets program

#78

Post by kuschku »

fhoffa wrote: post #33
kuschku wrote: post #29

Earlier quoted context omitted.

What you just did isn't how data is usually processed — a single 8GB JSON never occurs in real life cases, except maybe as transitional medium while you migrate from one database to another. In most cases, you'll have it stored in a relational or graph database, easily accessible. Especially in the sciences. But sure, I can go through the data and tell you the amount of cats with specific properties, or similar. I pe…

> Especially in the sciences. Take a look at how Stanford is leveraging BigQuery for their genomics analysis: http://www.eventbrite.com/e/interactive-cloud-analytics-extr... > It manages that faster than just the latency to bigquery would be — 18ms over a test dataset of several gigabyte by now. Yes!!! Exactly! That's the whole point. If something is taking you less than a second, don't bring BigQuery into the mix. B…

> If something is taking you less than a second, don't bring BigQuery into the mix. But when you start hitting processes that take hours or days to run... try BigQuery

But that's my entire point? Why are all the test datasets small enough to be faster in pgsql than the latency to Google would take?

I mean, if I wanted to showcase my software, I'd use a huge dataset with a complex problem — say protein folding, or superconductor molecular analysis — and show it in comparison on bigquery and a standard local database.

Although these two examples are bad, as I know from my university that they can be solved in the same time for less money locally than by using bigquery

Re: Google launches Public Datasets program

#79

Post by teddyh »

minimaxir wrote: post #36
jonesb6 wrote: post #15

Earlier quoted context omitted.

Would be interesting to find out the value (upvotes) of sponsored content vs. unsponsered content, as well as comments from green users vs normal users.

There isn't sponsored content on HN. (if you are referring to the job ads, those do not receive upvotes and follow a steady rank decay.)

What? The lack of upvotes (or steady decay) does not make it “not content”.
Post reply on HN