Update strategy of master first is interesting. I've always seen the other way with update standby, flip to standby, verify, update original master.
Auto inc db keys once again cause horribleness. Nothing new there I suppose.
And as mentioned the multi dyno + DB read status page is craaaazy. Why oh why isnt this a couple static objects. Automagically generate and push if you want. Give 'em a 60 second TTL and call it a day. Put them behind a different CDN & DNS then the rest of your site for bonus points.
GitHub availability this week
51–60 of 62 posts
Re: GitHub availability this week
#52"As traffic to the status site began to ramp up, we increased the number of dynos running from 8 to 64 and finally 90."
Wait, why isn't there some caching layer? eg. Generate a static page or use Varnish.
This part makes no sense at all.
At most you're then firing up another 5 dynos (or none) to handle the traffic. 90 is ridiculous.
Re: GitHub availability this week
#53Interesting to read about github using MySQL instead of Postgres. Anyone know why? I am just curious because of all the MySQL bashing I hear in the echo chamber.
They use both, according to Zach Holman ( http://zachholman.com/talk/unsucking-your-teams-development-... )
The only postgres we use is from internal Heroku apps. We use Mongo in a few places too.
Re: GitHub availability this week
#54"As traffic to the status site began to ramp up, we increased the number of dynos running from 8 to 64 and finally 90." Wait, why isn't there some caching layer? eg. Generate a static page or use Varnish. This part makes no sense at all. At most you're then firing up another 5 dynos (or none) to handle the traffic. 90 is ridiculous.
[deleted]
Re: GitHub availability this week
#55Earlier quoted context omitted.
Anyone tested S3's static page hosting under heavy load? I would think you could just update the static file as a result of some events fired by your internal monitoring process.
S3 is great for static content. I was taking the AWS ops course and the instructor mentioned some very large organizations redirect their site to S3 when under DDOS so they can remain on-line. In fact, he said that AWS recommended this solution to them?! Can you fathom someone who is under DDOS, and you tell them, hey, just redirect that our way ;)
You pay for the bandwidth on AWS. Of course they would be glad to redirect a DDOS their way. It's pure gold for AWS.
Re: GitHub availability this week
#56The lack of any negative response on this thread is a testament both to the thoroughness of the post-mortem, and the outstanding quality of GitHub in general. In GitHub we trust. I can't imagine putting my code anywhere else right now.
I like github too but please remember that things come and go. Some time ago it was SourceForge that was hot.
Re: GitHub availability this week
#57Genuine question: github is built upon git, which is a rock solid system for storing dataand in these reports we read that github relies a lot on MySQL, so... Did the github guys ponder using git as their data store? Just an example, in git one can add comments on commits, would it be possible to use it for the github comment function? Or maybe it is?
Re: GitHub availability this week
#58The lack of any negative response on this thread is a testament both to the thoroughness of the post-mortem, and the outstanding quality of GitHub in general. In GitHub we trust. I can't imagine putting my code anywhere else right now.
I like github too but please remember that things come and go. Some time ago it was SourceForge that was hot.
... but never as well loved.
Re: GitHub availability this week
#59"16 of these repositories were private, and for seven minutes from 8:19 AM to 8:26 AM PDT on Tuesday, Sept 11th, were accessible to people outside of the repository's list of collaborators or team members" ouch!
One of those repos was mine. :( Fortunately it was a fresh Rails app without anything important. However, it does make me rethink the security of storing my code on github.
I store proprietary code on github, but I would never recommend storing actual secrets (like keys or passwords).
Re: GitHub availability this week
#60Genuine question: github is built upon git, which is a rock solid system for storing dataand in these reports we read that github relies a lot on MySQL, so... Did the github guys ponder using git as their data store? Just an example, in git one can add comments on commits, would it be possible to use it for the github comment function? Or maybe it is?
Generally, Git will be way too slow for that. Git is typically our bottleneck, since you're dealing with so much overhead and disk access to perform functions.
Databases are best for, well, performing relational queries. In the case of commenting on a commit, if you store them only in the repository it becomes non-trivial to ask "show me all of the comments by this user" unless you have an intermediary cache layer (in which case you're back where you started).