Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google launches Public Datasets program

cloud.google.com

Post reply on HN
Print view View original

91–98 of 98 posts

Re: Google launches Public Datasets program

#91

Post by dwmintz »

apahwa wrote: post #61
dwmintz wrote: post #41

Earlier quoted context omitted.

Looker is powering all the iframes (any 404s should be fixed now). Bigquery is hosting the data, but Looker is generating the queries and visualizations of the data.

Interesting, I'm surprised Google is using Looker rather than their own tool edit: ah, I see, it is just one of the partners they have with BigQuery and they are apparently using it for their website (I'm guessing because Looker lets you iframe graphs directly into the page for free).

Yup, Looker is a Premier Partner for the GCP launch (https://cloud.google.com/partners/?q=Looker#search). And then we (I work at Looker) are specifically partnering with GCP for the Public Datasets project.

Because LookML (Looker's modeling language) makes it easy to explore and visualize big datasets, we're building out models and dashboards so visitors can get a sense of BigQuery/Looker's power and find insights from the datasets quickly (whether or not they write SQL).

Frankly, there's a ton of public data that's "available" in the sense that you can technically download and clean a CSV, but isn't actually easy to extract meaning from. So we figured we'd select some interesting datasets, do the cleaning, uploading and modeling, and then let folks have at it for free.

If there are specific datasets you'd be interested in seeing, let us know and we'll see what we can do.

Re: Google launches Public Datasets program

#93

Post by vgt »

ignoramous wrote: post #75
vgt wrote: post #63

Earlier quoted context omitted.

Ah yes, point well taken. I think we've seen this historically at Google, but things have changed in the past few years.

Primarily as a reaction to AWS and Azure? Google announced Colossus in 2011, is that going to make it to GCP?

You can use Colossus today! It's a very low level service, leveraged by other GCP services.

For example, Google Cloud Storage and BigQuery Storage both sit on Colossus. Read more on BQ here:

https://cloud.google.com/blog/big-data/2016/01/bigquery-unde...

Re: Google launches Public Datasets program

#94

Post by vgt »

dcdevito wrote: post #82

Sorry Google but my trust in cloud solutions reside in AWS and Azure. Why? Because when Amazon and Microsoft announce something I know there's a good chance it will still exist 24 months later

This narrative gets repeated time and time again, and it really doesn't hold up to even surface debate:

- Google TI and Cloud merged. Same teams. Per Urs Holzle in Jan2014

- Just last week Sundar (CEO), Eric Schmidt (chairman), Jeff Dean, Urs Holzle, and Diane were on stage talking about how they spent $10 billion last year investing in infrastructure and how serious Google is about cloud (https://www.youtube.com/watch?v=HgWHeT_OwHc&list=PLIivdWyY5s...)

- In many cases, publically available technologies are actually replacing internal technologies (BigQuery and Dremel, Dataflow and FlumeJava/Millwheel for example).

- Disney, Coca Cola, Snapchat, Spotify, Home Depot are just some of the customers running on Google Cloud. It'd be very foolish to abandon these.

- If you watch the day two GCP NEXT keynote, you'll see how Google is the cloud leader in environmental responsibility and open source. We're the good guys :) (https://www.youtube.com/watch?v=axhdIa_co2o&list=PLIivdWyY5s...)

Happy to debate further!

Re: Google launches Public Datasets program

#95

Post by vgt »

duncanawoods wrote: post #66
vgt wrote: post #21

Earlier quoted context omitted.

Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google relies on it heavily. Externally, companies like Spotify and Kabam Games rely heavily on BigQuery as well: https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5s... https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5s... So the story of Google Clou…

>> Externally, companies like Spotify I hate people bringing them up - its just being a sucker for marketing. Spotify and Netflix are clearly brands recruited to promote their respective platforms. The only question is how large a discount they are getting among other special treatment in return for their name. I wouldn't be surprised if its conditional on a gag criticising the platform too. These are the sort of sit…

You can listen and read Spotify engineers talking about their experiences with Google, including running 2,000,000 qps load tests without ever telling anyone at Google.

If you read some of their blogs, you can see how brutally honest they are about the pros and cons of our services:

https://labs.spotify.com/2016/03/10/spotifys-event-delivery-...

https://www.youtube.com/watch?v=LTVFg6YOjWo&index=7&list=PLI...

https://www.youtube.com/watch?v=xT6tQAIywFQ&index=1&list=PLI...

Finally, does this sound like an engineer who got paid to speak? https://twitter.com/mpjme/status/702167231623516160

Re: Google launches Public Datasets program

#96

Post by kuschku »

fhoffa wrote: post #86
kuschku wrote: post #78

Earlier quoted context omitted.

> If something is taking you less than a second, don't bring BigQuery into the mix. But when you start hitting processes that take hours or days to run... try BigQuery But that's my entire point? Why are all the test datasets small enough to be faster in pgsql than the latency to Google would take? I mean, if I wanted to showcase my software, I'd use a huge dataset with a complex problem — say protein folding, or sup…

Look... if you don't have problems that take hours to solve in your current environment, you haven't found big data problems (yet). And that's OK. Not everyone works with big data. But if one day you do, please ping me, it will be fun to do a follow up.

Yeah, that's why I had hoped the test datasets would be big data problems — so I could see what that actually looks like ;)

Re: Google launches Public Datasets program

#97

Post by mark_l_watson »

Public data is good! I have used their book text data on two projects to generate common ngrams. These newly released data sets look useful and using BigQuery is reasonable if you just need parts of data sets.

Amazon does something similar: keeping useful data in S3 for cheap access when running on AWS.

Post reply on HN