How We Work on Queries at GitHub

#1

Post by samlambert » Fri, Aug 29, 2014, 8:08 PM UTC

How We Work on Queries at GitHub
samlambert.com

Re: How We Work on Queries at GitHub

#2

Post by nathantotten » Fri, Aug 29, 2014, 8:56 PM UTC

Now, does anyone have the time to figure out how many users Github has in their users table? :) http://dheera.net/projects/blur

Re: How We Work on Queries at GitHub

#3

Post by ckluis » Fri, Aug 29, 2014, 8:58 PM UTC

Awesome. I’m going to share this to see if we can do something similar where I work. Neat idea.

Re: How We Work on Queries at GitHub

#4

Post by petercooper » Fri, Aug 29, 2014, 9:04 PM UTC

nathantotten wrote: post #2

Now, does anyone have the time to figure out how many users Github has in their users table? :) http://dheera.net/projects/blur

The mosaic effect they used doesn't appear to be real, as no number has a horizontal gap between two parts.

However, GitHub user IDs are/always used to be sequential, create a new account and see what its user ID is. It's around 8.6 million at the moment.

Re: How We Work on Queries at GitHub

#5

Post by minimaxir » Fri, Aug 29, 2014, 9:06 PM UTC

nathantotten wrote: post #2

Now, does anyone have the time to figure out how many users Github has in their users table? :) http://dheera.net/projects/blur

Atleast 3,815,207 users. (who have performed some meaningful GitHub action)

Via BigQuery:

   SELECT COUNT(DISTINCT actor) FROM [githubarchive:github.timeline];

Re: How We Work on Queries at GitHub

#6

Post by blaincate » Fri, Aug 29, 2014, 9:56 PM UTC

from : http://ghtorrent.org/downloads.html

atleast :

   4,151,457 repos   

   2,480,478 users

popular repos :

twbs/bootstrap 40662

jquery/jquery 34633

joyent/node 34522

mbostock/d3 30247

h5bp/html5-boilerplate 28736

popular users:

visionmedia 10712

torvalds 9984

paulirish 5885

schacon 4431

mattt 4053

pjhyett 3732

src code: https://github.com/akuchlous/githublike

Re: How We Work on Queries at GitHub

#7

Post by simonw » Fri, Aug 29, 2014, 10:06 PM UTC

"Once we have decided if we want to modify our schema we can perform an incremental rollout across our cluster. I will cover this more in another post." - looking forward to that.

Re: How We Work on Queries at GitHub

#8

Post by famousactress » Fri, Aug 29, 2014, 10:08 PM UTC

So, maybe a toy example but I can see the query included a join. Curious how you guys clone enough tables (and their keys) to troubleshoot things like that? Seems like it gets a lot more complex than the example suggests pretty quickly. Wondering if you have neat tools for that.

[Edit: Just noticed poster is author. Hi Sam and welcome to HN :)]

Re: How We Work on Queries at GitHub

#9

Post by samlambert » Fri, Aug 29, 2014, 10:09 PM UTC

famousactress wrote: post #8

So, maybe a toy example but I can see the query included a join. Curious how you guys clone enough tables (and their keys) to troubleshoot things like that? Seems like it gets a lot more complex than the example suggests pretty quickly. Wondering if you have neat tools for that. [Edit: Just noticed poster is author. Hi Sam and welcome to HN :)]

Basically you can hit /mysql clone for any table and it will make its way to an isolated db for the user running the command.

It would be cool to be able to pass the script a query and have it clone all the tables.

Re: How We Work on Queries at GitHub

#10

Post by samlambert » Fri, Aug 29, 2014, 10:17 PM UTC

famousactress wrote: post #8

So, maybe a toy example but I can see the query included a join. Curious how you guys clone enough tables (and their keys) to troubleshoot things like that? Seems like it gets a lot more complex than the example suggests pretty quickly. Wondering if you have neat tools for that. [Edit: Just noticed poster is author. Hi Sam and welcome to HN :)]

Hey! Thank you for the welcome :)