Earlier quoted context omitted.
Mostly because of legacy reasons, at this point.
Do you have a source for this information?
I have the source code, yes :)
GitHub availability this week
21–30 of 62 posts
Re: GitHub availability this week
#22If Github hasn't gotten their custom HA solution right, will you? Digging into their fix, they disabled automatic failover -- so all DB failures will now require manual intervention. While addressing this particular (erroneous) failover condition, it does raise minimum down time for true failures. Also, their mysql replicant's misconfiguration upon switching masters is also tied to their (stopgap) approach to prevent…
It blows my mind that they aren't simply using Jekyll to generate and update the status page. I mean... they wrote it, right?
Re: GitHub availability this week
#23Earlier quoted context omitted.
Mostly because of legacy reasons, at this point.
Do you have a source for this information?
https://github.com/technoweenie
He's on the Github team. I assume he's speaking Ex Cathedra.
Re: GitHub availability this week
#24I'd like to welcome the github ops/dbas to the club of people who've learned the hard way that automated database failover usually causes more downtime than it prevents. Here's sortof the seminal post on the matter in the mysql community: http://www.xaprb.com/blog/2009/08/30/failure-scenarios-and-s... Though it turns into an MMM pile-on the tool doesn't matter so much as the scenarios. Automated failover is simply un…
automated failover in the case of too much load is usually not what you want to do. automated failure in the case of hw/network failure is usually what you want to do. differentiating the former from the latter is left as an exercise for the reader.
Re: GitHub availability this week
#25If Github hasn't gotten their custom HA solution right, will you? Digging into their fix, they disabled automatic failover -- so all DB failures will now require manual intervention. While addressing this particular (erroneous) failover condition, it does raise minimum down time for true failures. Also, their mysql replicant's misconfiguration upon switching masters is also tied to their (stopgap) approach to prevent…
"There is also lesson to be learned in the fact that their status page had scaling issues due to db connection limits. Static files are the most dependable!"
Seriously, why would a status page need to query a db?
Re: GitHub availability this week
#26Interesting to read about github using MySQL instead of Postgres. Anyone know why? I am just curious because of all the MySQL bashing I hear in the echo chamber.
They use both, according to Zach Holman (http://zachholman.com/talk/unsucking-your-teams-development-...)
Re: GitHub availability this week
#27The part of this post that really blew my mind: We host our status site on Heroku to ensure its availability during an outage. However, during our downtime on Tuesday our status site experienced some availability issues. As traffic to the status site began to ramp up, we increased the number of dynos running from 8 to 64 and finally 90. This had a negative effect since we were running an old development database addo…
At the time of the outage, the status site was seeing upwards of 30,000/req minute. AS we scaled up dynos, we would see temporary performance improvements until the status site would stop responding again. In the short term, this led to us massively increasing dynos as quickly as we could as it appeared that CPU burn was a significant cause of the slowness (at the time). This was in part caused by all the dynos repea…
What prevented you from just caching the status page and then refilling the cache manually every X seconds ? I'm sure a status that is a few seconds old given the system wide meltdown wouldn't have been an unreasonable compromise ?
Re: GitHub availability this week
#28If Github hasn't gotten their custom HA solution right, will you? Digging into their fix, they disabled automatic failover -- so all DB failures will now require manual intervention. While addressing this particular (erroneous) failover condition, it does raise minimum down time for true failures. Also, their mysql replicant's misconfiguration upon switching masters is also tied to their (stopgap) approach to prevent…
"There is also lesson to be learned in the fact that their status page had scaling issues due to db connection limits. Static files are the most dependable!" Seriously, why would a status page need to query a db?
I assume that the status server is not actively checking every Github server/service whenever someone pings it. It probably polls the servers every X seconds. The best place to store that type of data is in a DB.
Where else would you put it?
Re: GitHub availability this week
#29Earlier quoted context omitted.
At the time of the outage, the status site was seeing upwards of 30,000/req minute. AS we scaled up dynos, we would see temporary performance improvements until the status site would stop responding again. In the short term, this led to us massively increasing dynos as quickly as we could as it appeared that CPU burn was a significant cause of the slowness (at the time). This was in part caused by all the dynos repea…
What prevented you from just caching the status page and then refilling the cache manually every X seconds ? I'm sure a status that is a few seconds old given the system wide meltdown wouldn't have been an unreasonable compromise ?
Or memcache, with one worker dyno dedicated to updating it, cron-like.
Re: GitHub availability this week
#30Earlier quoted context omitted.
Sure. Maybe I should do a writeup for it on my blog at some point in the near future :). The two main issues we encountered both had to do with search for products/categories on our sites. The first was that Galera/WSREP doesn't support MyISAM replication (It has beta support, but I wouldn't trust it). This meant that we had to transition our fulltext data to something else. The something else in this case was Solr w…
Consider this an expression of extreme interest on my part.
+1