Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub availability this week

github.com

Post reply on HN
Print view View original

31–40 of 62 posts

Re: GitHub availability this week

#31

Post by aaronblohowiak »

gsibble wrote: post #28
autotravis wrote: post #25

Earlier quoted context omitted.

"There is also lesson to be learned in the fact that their status page had scaling issues due to db connection limits. Static files are the most dependable!" Seriously, why would a status page need to query a db?

I assume that the status server is not actively checking every Github server/service whenever someone pings it. It probably polls the servers every X seconds. The best place to store that type of data is in a DB. Where else would you put it?

> It probably polls the servers every X seconds.

And then you could write out a new static file, just once, and send it to your edge server of choice.

Re: GitHub availability this week

#32

Post by wonnage »

gsibble wrote: post #28
autotravis wrote: post #25

Earlier quoted context omitted.

"There is also lesson to be learned in the fact that their status page had scaling issues due to db connection limits. Static files are the most dependable!" Seriously, why would a status page need to query a db?

I assume that the status server is not actively checking every Github server/service whenever someone pings it. It probably polls the servers every X seconds. The best place to store that type of data is in a DB. Where else would you put it?

You could just as easily store the result in a plain file somewhere... a database seems like overkill.

Re: GitHub availability this week

#33

Post by dustym »

mbell wrote: post #10
cwb71 wrote: post #9

The part of this post that really blew my mind: We host our status site on Heroku to ensure its availability during an outage. However, during our downtime on Tuesday our status site experienced some availability issues. As traffic to the status site began to ramp up, we increased the number of dynos running from 8 to 64 and finally 90. This had a negative effect since we were running an old development database addo…

Anyone tested S3's static page hosting under heavy load? I would think you could just update the static file as a result of some events fired by your internal monitoring process.

We use S3 behind 1 second max-age cloudfront to serve The Verge liveblog. It's been nothing but rock solid. We essentially create a static site and push up JSON blobs. See here:

http://product.voxmedia.com/post/25113965826/introducing-syl...

Re: GitHub availability this week

#35

Post by moe »

mbell wrote: post #10
cwb71 wrote: post #9

The part of this post that really blew my mind: We host our status site on Heroku to ensure its availability during an outage. However, during our downtime on Tuesday our status site experienced some availability issues. As traffic to the status site began to ramp up, we increased the number of dynos running from 8 to 64 and finally 90. This had a negative effect since we were running an old development database addo…

Anyone tested S3's static page hosting under heavy load? I would think you could just update the static file as a result of some events fired by your internal monitoring process.

"Heavy load"?

30 kRPM is 500 hits/sec. Nginx will serve >2000/sec from a m1.small. For S3 that is about the equivalent of a mosquito fart.

Re: GitHub availability this week

#37

Post by _f75i »

jluxenberg wrote: post #18

"16 of these repositories were private, and for seven minutes from 8:19 AM to 8:26 AM PDT on Tuesday, Sept 11th, were accessible to people outside of the repository's list of collaborators or team members" ouch!

One of those repos was mine. :( Fortunately it was a fresh Rails app without anything important. However, it does make me rethink the security of storing my code on github.

Re: GitHub availability this week

#38

Post by rhizome »

aaronblohowiak wrote: post #6

If Github hasn't gotten their custom HA solution right, will you? Digging into their fix, they disabled automatic failover -- so all DB failures will now require manual intervention. While addressing this particular (erroneous) failover condition, it does raise minimum down time for true failures. Also, their mysql replicant's misconfiguration upon switching masters is also tied to their (stopgap) approach to prevent…

I think people tend to overestimate the value of nines to the user. It's chiefly a management/VC/busybody metric that has gained importance mainly due to it being a high level and easy to understand abstraction. "Well how much was it down?" Then they spend zillions on failover software, hardware and talent that could be supplanted by one fewer nine and a simpler architecture.

And really, just to get a dig in here, I believe Arrington shares a big part of the blame for this state of affairs with all of his Dvorak-caliber ignorant harping about Twitter back in the day.

Re: GitHub availability this week

#39

Post by spicyj »

dustym wrote: post #33
mbell wrote: post #10

Earlier quoted context omitted.

Anyone tested S3's static page hosting under heavy load? I would think you could just update the static file as a result of some events fired by your internal monitoring process.

We use S3 behind 1 second max-age cloudfront to serve The Verge liveblog. It's been nothing but rock solid. We essentially create a static site and push up JSON blobs. See here: http://product.voxmedia.com/post/25113965826/introducing-syl...

This is really interesting -- thanks for sharing. It seems to me that you could probably have nginx running on a regular box and then CloudFront as a caching CDN to avoid the S3 update delay.

Re: GitHub availability this week

#40

Post by boundlessdreamz »

technoweenie wrote: post #16
cschep wrote: post #11

Interesting to read about github using MySQL instead of Postgres. Anyone know why? I am just curious because of all the MySQL bashing I hear in the echo chamber.

Mostly because of legacy reasons, at this point.

That sounds like you would have chosen differently if you had to choose now. Is that so?
Post reply on HN