Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub availability this week

github.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: GitHub availability this week

#41

Post by adgar »

wfarr wrote: post #14
cwb71 wrote: post #9

The part of this post that really blew my mind: We host our status site on Heroku to ensure its availability during an outage. However, during our downtime on Tuesday our status site experienced some availability issues. As traffic to the status site began to ramp up, we increased the number of dynos running from 8 to 64 and finally 90. This had a negative effect since we were running an old development database addo…

At the time of the outage, the status site was seeing upwards of 30,000/req minute. AS we scaled up dynos, we would see temporary performance improvements until the status site would stop responding again. In the short term, this led to us massively increasing dynos as quickly as we could as it appeared that CPU burn was a significant cause of the slowness (at the time). This was in part caused by all the dynos repea…

30,000req/minute is 500qps. That's... just not a lot for a large service.

Re: GitHub availability this week

#42

Post by koglerjs »

cagenut wrote: post #13

I'd like to welcome the github ops/dbas to the club of people who've learned the hard way that automated database failover usually causes more downtime than it prevents. Here's sortof the seminal post on the matter in the mysql community: http://www.xaprb.com/blog/2009/08/30/failure-scenarios-and-s... Though it turns into an MMM pile-on the tool doesn't matter so much as the scenarios. Automated failover is simply un…

http://www.shoebuy.com/gym-shorts.htm

Re: GitHub availability this week

#43

Post by ghshephard »

cagenut wrote: post #13

I'd like to welcome the github ops/dbas to the club of people who've learned the hard way that automated database failover usually causes more downtime than it prevents. Here's sortof the seminal post on the matter in the mysql community: http://www.xaprb.com/blog/2009/08/30/failure-scenarios-and-s... Though it turns into an MMM pile-on the tool doesn't matter so much as the scenarios. Automated failover is simply un…

Automated database failover is absolutely mandatory for HA environments (as in, there is no way to run a 5 9s system without it) but, poorly done, results in actually reducing your uptime (which is a separate concept from HA).

I've been in a couple of environment in which developers have successfully rolled out automated database failover, and, my takeaway, is that's it usually not worth the cost - and with very, very few exceptions, most organizations can take the downtime of several minutes to do manual failover.

In general, when rolling out these operational environment, they are only ready when you've found, and demonstrated 10-12 failure cases, and come up with workarounds.

In other words - if you can't demonstrate how your environment will fail, then it's not ready for an HA deployment.

Re: GitHub availability this week

#44

Post by cageface »

WestCoastJustin wrote: post #12

Here are the makings of a bad week (Monday of all things) - MySQL schema migration causes high load, automated HA solution causes cascading database failure - MySQL cluster becomes out of sync - HA solution segfaults - Redis and MySQL become out of sync - Incorrect users have access to private repositories! Cleanup and recovery takes time, all I can say is, I'm glad it was not me who had that mess to clean up . I'm s…

This kind of thing is one of the main reasons I prefer to do app development instead of backend work now. I don't get calls at 3am any more.

Re: GitHub availability this week

#45

Post by code0 »

jluxenberg wrote: post #18

"16 of these repositories were private, and for seven minutes from 8:19 AM to 8:26 AM PDT on Tuesday, Sept 11th, were accessible to people outside of the repository's list of collaborators or team members" ouch!

I am really curious about the technical reasons how this might have happened.

Re: GitHub availability this week

#46

Post by Xorlev »

ghshephard wrote: post #43
cagenut wrote: post #13

I'd like to welcome the github ops/dbas to the club of people who've learned the hard way that automated database failover usually causes more downtime than it prevents. Here's sortof the seminal post on the matter in the mysql community: http://www.xaprb.com/blog/2009/08/30/failure-scenarios-and-s... Though it turns into an MMM pile-on the tool doesn't matter so much as the scenarios. Automated failover is simply un…

Automated database failover is absolutely mandatory for HA environments (as in, there is no way to run a 5 9s system without it) but, poorly done, results in actually reducing your uptime (which is a separate concept from HA). I've been in a couple of environment in which developers have successfully rolled out automated database failover, and, my takeaway, is that's it usually not worth the cost - and with very, ver…

Every HA deployment I've done, the HA manager inevitably had issues to begin with. It takes time, patience, and a few late nights.

Re: GitHub availability this week

#47

Post by dustym »

spicyj wrote: post #39
dustym wrote: post #33

Earlier quoted context omitted.

We use S3 behind 1 second max-age cloudfront to serve The Verge liveblog. It's been nothing but rock solid. We essentially create a static site and push up JSON blobs. See here: http://product.voxmedia.com/post/25113965826/introducing-syl...

This is really interesting -- thanks for sharing. It seems to me that you could probably have nginx running on a regular box and then CloudFront as a caching CDN to avoid the S3 update delay.

Probably could figure that out, yeah. But we didn't want to take any chances given how important it was to get our live blog situation under control.

[edit]

Which is to say, we wanted a rock solid network and to essentially be a drop in a bucket of traffic, even at the insane burst that The Verge live blog gets.

Re: GitHub availability this week

#49

Post by ghshephard »

Xorlev wrote: post #46
ghshephard wrote: post #43

Earlier quoted context omitted.

Automated database failover is absolutely mandatory for HA environments (as in, there is no way to run a 5 9s system without it) but, poorly done, results in actually reducing your uptime (which is a separate concept from HA). I've been in a couple of environment in which developers have successfully rolled out automated database failover, and, my takeaway, is that's it usually not worth the cost - and with very, ver…

Every HA deployment I've done, the HA manager inevitably had issues to begin with. It takes time, patience, and a few late nights.

With the possible exception of life safety systems, credit card processing, stock exchanges, and other "High $ per second applications" - I just don't see getting HA right on transactional databases as worth the effort. Properly rehearsed, a good Ops/DBA (and, in the right environment) NOC team can execute a decent failover in just a few minutes - and there aren't that many environments (with the exceptions listed above) - that can't take two or three 5 minute outages a year.

The alternative is your HA manager decides to act wacky on you, and your database downtime is extended.

For some reason - this rarely (almost never, in my practical experience) is a problem with HA systems in networking. With just a modicum of planning, HA Routers, Switches, and Load Balancers Just Seem to Work (tm).

Likewise, HA Storage arrays are bullet proof to the point at which a lot of reasonably conservative companies are comfortable picking up a single array/frame.

But HA transactional databases - still don't seem to be there.

Re: GitHub availability this week

#50

Post by donavanm »

dustym wrote: post #33
mbell wrote: post #10

Earlier quoted context omitted.

Anyone tested S3's static page hosting under heavy load? I would think you could just update the static file as a result of some events fired by your internal monitoring process.

We use S3 behind 1 second max-age cloudfront to serve The Verge liveblog. It's been nothing but rock solid. We essentially create a static site and push up JSON blobs. See here: http://product.voxmedia.com/post/25113965826/introducing-syl...

Could you say more about using both the Cache-Control and a query string of epoch time? In particular the query string has me puzzled. On it's face it seems to decrease your cache hit ratio, with no/little benefit. Im assuming the epoch time is the clients local time. The clock skew across the client population increases the number of cache keys active at any one time. The incrementing query string also forces a new cache key once per second. Those would force a cache miss and complete request to S3 even when content has not changed. It's even worse with the skew as you now force a cache miss per second for each unique epoch time in your client clocks. Without the query string the cache could do a conditional GET for live.json. That would save latency & bytes as the origin could respond with a 304 instead of the complete 200.
Post reply on HN