31–40 of 98 posts
I liked what Freebase, DBpedia etc were doing few years ago with "semantic db's". I remember Freebase Gridworks being extremely useful for data cleanup. Haven't had a reason to follow the space...is it dead? Haven't seen much semantic talk off late. Where would one go to catch up on the latest news?
Any chance we'll be seeing the Common Crawl data on there anytime soon?
bq-public-data@google.com
Earlier quoted context omitted.
Hi Janne! I'm sure you'll enjoy a challenge. Let's talk about Wikidata. Can you download this 8 GB compressed file? https://dumps.wikimedia.org/wikidatawiki/entities/latest-all... I want to know the id of the JSON row which length is 102 bytes. It took me 4 seconds with BigQuery - how can we improve this with "any postgres install"? https://lists.wikimedia.org/pipermail/wikidata/2016-March/00...
What you just did isn't how data is usually processed — a single 8GB JSON never occurs in real life cases, except maybe as transitional medium while you migrate from one database to another. In most cases, you'll have it stored in a relational or graph database, easily accessible. Especially in the sciences. But sure, I can go through the data and tell you the amount of cats with specific properties, or similar. I pe…
Take a look at how Stanford is leveraging BigQuery for their genomics analysis:
http://www.eventbrite.com/e/interactive-cloud-analytics-extr...
> It manages that faster than just the latency to bigquery would be — 18ms over a test dataset of several gigabyte by now.
Yes!!! Exactly! That's the whole point. If something is taking you less than a second, don't bring BigQuery into the mix. But when you start hitting processes that take hours or days to run... try BigQuery. Going from hours to seconds changes your life.
> I personally use pgsql currently to analyse data from Android crash reports.
Cool! Guess what Motorola uses...
http://www.slideshare.net/PatrickDeglon/predictive-analytics...
Who is the King of Hacker News? https://cloud.google.com/bigquery/public-data/hacker-news#wh...
I just noticed the Freebase data available as one of the public datasets, so been wondering what happened to the Freebase team? I liked what Freebase, DBpedia etc were doing few years ago with "semantic db's". I remember Freebase Gridworks being extremely useful for data cleanup. Haven't had a reason to follow the space...is it dead? Haven't seen much semantic talk off late. Where would one go to catch up on the late…
I loaded Wikidata into BigQuery too!
Fun with movies and cats: https://medium.com/google-cloud/oscars-2016-movies-that-got-...
Implementation notes: https://lists.wikimedia.org/pipermail/wikidata/2016-March/00...
More notes: https://lists.wikimedia.org/pipermail/wikidata/2016-March/00...
"HACKER NEWS - A dataset that contains all stories and comments from Hacker News since its launch in 2006." I know what I'm doing this weekend.
Would be interesting to find out the value (upvotes) of sponsored content vs. unsponsered content, as well as comments from green users vs normal users.
Full blog post explaining and walking through the process (and giving access to explore the data fully yourself, no SQL required) is here: http://looker.com/blog/hacking-hacker-news
Felipe and all the other folks at Google have done a great job getting this project off the ground and we're psyched to partner with them. We're working on some new public datasets now, but if you have particular ones you'd like to explore, let us know.
What should we look at next? Census? Medicare?
One large difference between this program and alternative programs is that data already resides in Google BigQuery: - You do not need to spin up a database to work with BigQuery - You can simply start writing SQL on top of BigQuery - You may leverage Dataflow and MapReduce connectors to work with this data directly in Hadoop, Spark, or Dataflow - BigQuery has a free tier - one Terabyte of data processed per month Fin…
I remember the earlier Google attempt at the same thing. I was in undergrad and an engineer came to my Uni to give a presentation on the topic. A big area of concern was "Why won't Google just shut this down when the going gets rough?" The question was mostly batted away along the lines of "this is lunch money to Google anyway." But just a couple years later, it did indeed bite the dust. Covered in Wired: http://www.…
Google BigQuery is a paid service, Dremel under the hood, used internally at Google by ~80% of Googlers every week. It's simply not going away because Google relies on it heavily. Externally, companies like Spotify and Kabam Games rely heavily on BigQuery as well: https://www.youtube.com/watch?v=LTVFg6YOjWo&list=PLIivdWyY5s... https://www.youtube.com/watch?v=6Nv18xmJirs&list=PLIivdWyY5s... So the story of Google Clou…