Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google launches Public Datasets program

cloud.google.com

Post reply on HN
Print view View original

61–70 of 98 posts

Re: Google launches Public Datasets program

#61

Post by apahwa »

dwmintz wrote: post #41
apahwa wrote: post #28

Earlier quoted context omitted.

where are you seeing Looker being used?

Looker is powering all the iframes (any 404s should be fixed now). Bigquery is hosting the data, but Looker is generating the queries and visualizations of the data.

Interesting, I'm surprised Google is using Looker rather than their own tool

edit: ah, I see, it is just one of the partners they have with BigQuery and they are apparently using it for their website (I'm guessing because Looker lets you iframe graphs directly into the page for free).

Re: Google launches Public Datasets program

#62

Post by krishna2 »

fhoffa wrote: post #35
tmannen wrote: post #31

I just noticed the Freebase data available as one of the public datasets, so been wondering what happened to the Freebase team? I liked what Freebase, DBpedia etc were doing few years ago with "semantic db's". I remember Freebase Gridworks being extremely useful for data cleanup. Haven't had a reason to follow the space...is it dead? Haven't seen much semantic talk off late. Where would one go to catch up on the late…

Freebase data is being moved into Wikidata. I loaded Wikidata into BigQuery too! Fun with movies and cats: https://medium.com/google-cloud/oscars-2016-movies-that-got-... Implementation notes: https://lists.wikimedia.org/pipermail/wikidata/2016-March/00... More notes: https://lists.wikimedia.org/pipermail/wikidata/2016-March/00...

Super interesting. Thanks for all the curation and relevant set of links that you are promptly posting/replying. StackOverflow data is available for free too - that would be one awesome dataset to let folks get their hands on. Any plans for that?

Re: Google launches Public Datasets program

#63

Post by vgt »

awqrre wrote: post #54
vgt wrote: post #48

Earlier quoted context omitted.

BigQuery is a truly fully-managed service. This means that SREs are on call 24/7. It's not software that one downloads. Hence it really doesn't make a lot of sense to have it available to just a few folks, incremental overhead is low. I'd love to hear industry standards around deprecation of paid services. This is really a way of doing business in the entire world, and I'm not sure why Google is being held to an unre…

I was not blaming anybody, it's possible that a service would be more valuable to a company if it's kept internally.

Ah yes, point well taken. I think we've seen this historically at Google, but things have changed in the past few years.

Re: Google launches Public Datasets program

#64

Post by bcook »

toomuchtodo wrote: post #59
jrockway wrote: post #58

Earlier quoted context omitted.

It's mildly amusing: > SELECT sum(length(text)) FROM [bigquery-public-data:hacker_news.comments] where author="jrockway" 4830955 That's almost 5 MB of comments I've written.

Get outside more :)

Being outside does not preclude internet access... :)

Re: Google launches Public Datasets program

#65

Post by vgt »

Veratyr wrote: post #55
vgt wrote: post #5

One large difference between this program and alternative programs is that data already resides in Google BigQuery: - You do not need to spin up a database to work with BigQuery - You can simply start writing SQL on top of BigQuery - You may leverage Dataflow and MapReduce connectors to work with this data directly in Hadoop, Spark, or Dataflow - BigQuery has a free tier - one Terabyte of data processed per month Fin…

> Finally, for folks who would like to share their datasets, BigQuery offers free hosting and credits to help get a pipeline going. Is this available in an automated fashion or is it something one has to contact Google for?

You can always create a dataset and make it public by setting open ACLs to it. The only downside is paying $0.02 per GB per month.

To participate in the Public Datasets program, you must contact us - there's a lot of legal mumbo jumbo associated with us hosting your datasets that I wish weren't the case :)

Re: Google launches Public Datasets program

#66

Post by duncanawoods »

vgt wrote: post #21
pbh101 wrote: post #17

I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of concern was "Why won't Google just shut this down when the going gets rough?" The question was mostly batted away along the lines of "this is lunch money to Google anyway." But just a couple years later, it did indeed bite the dust. Covered in Wired: http://www.…

Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google relies on it heavily. Externally, companies like Spotify and Kabam Games rely heavily on BigQuery as well: https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5s... https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5s... So the story of Google Clou…

>> Externally, companies like Spotify

I hate people bringing them up - its just being a sucker for marketing. Spotify and Netflix are clearly brands recruited to promote their respective platforms. The only question is how large a discount they are getting among other special treatment in return for their name. I wouldn't be surprised if its conditional on a gag criticising the platform too. These are the sort of situations that are revealed years later to have required a totally isolated and custom solution that bore no resemblance to the service it was intended to promote.

Special partnerships prove nothing because the economics and service you get won't be remotely similar.

Re: Google launches Public Datasets program

#67

Post by vgt »

dwmintz wrote: post #38

Great to see folks digging into this. Looker (where I work) is what you're seeing visualize and make the underlying datasets explorable. Our founder did a great blog post explaining how it works and how quickly these datasets yield interesting insights with just a few lines of code. Full blog post explaining and walking through the process (and giving access to explore the data fully yourself, no SQL required) is her…

Kudos to Looker, great partners here! We're just getting started :)

Re: Google launches Public Datasets program

#68

Post by incepted »

vgt wrote: post #21
pbh101 wrote: post #17

I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of concern was "Why won't Google just shut this down when the going gets rough?" The question was mostly batted away along the lines of "this is lunch money to Google anyway." But just a couple years later, it did indeed bite the dust. Covered in Wired: http://www.…

Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google relies on it heavily. Externally, companies like Spotify and Kabam Games rely heavily on BigQuery as well: https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5s... https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5s... So the story of Google Clou…

> It's simply not going away because Google relies on it heavily.

Still doesn't mean Google will maintain the public access to this database. They might just as well shut it down and limit its access to Googlers. Like they already did a few years ago.

Re: Google launches Public Datasets program

#69

Post by incepted »

fhoffa wrote: post #24
pbh101 wrote: post #17

I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of concern was "Why won't Google just shut this down when the going gets rough?" The question was mostly batted away along the lines of "this is lunch money to Google anyway." But just a couple years later, it did indeed bite the dust. Covered in Wired: http://www.…

2008. I remember those days. GMail was still in beta and ' a big area of concern was "Why won't Google just shut this down when the going gets rough?" ' 8 years later GMail has a billion users, and not going anywhere but up. I haven't been to the future (yet), but I'll happily take any bet you have against BigQuery. Disclaimer: I'm Felipe Hoffa, and I work at Google. ( https://twitter.com/felipehoffa )

There are not really comparable.

The value of an email service is how many users it has.

The value of a big database is... who knows? If Google one day decides the content of that database is a strategic asset, they might just as well decide to shut it down to the public and continue using it internally.

Re: Google launches Public Datasets program

#70

Post by flashman »

I just ran a Markov text generator over the USA Name Database and created a few thousand girls' names, including:

Aracella, Ashla, Blakelyne, Carylou, Damariah, Enchantrelle, Francenza, Iridia, Jalexius, Lilliotte and Scotlanta.

Hey, maybe I can start a bespoke baby name business...

Post reply on HN