Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Outage

github.com

Post reply on HN
Print view View original

261–270 of 297 posts

Re: GitHub Outage

#261

Post by mitchtbaum »

debacle wrote: post #136
beat wrote: post #124

Earlier quoted context omitted.

Having used both, I find github's tools (mostly) better. And bitbucket isn't 100% reliable either.

Self-hosted is.

"Fool me once, shame on you. Fool me twice, shame on me."

I can learn from my mistakes and improve progressively.

If I rely on another instead, I cannot, and I cannot (necessarily and confidently) see to it that they learn from their mistakes and improve.

By relying on another (at least an unreliable \ uncommunicative \ uncooperative one), I cannot improve my chances of them not making those same mistakes again, taking me down with them.

Re: GitHub Outage

#262

Post by hayleox »

madmod wrote: post #80

For the impatient. I would make a gist but... while true; do curl -s https://status.github.com/api/status.json | grep good && tput bel || echo -n .; sleep 1; done

Ah, I like yours better for GitHub specifically, but for any website, my .bashrc got this as its latest addition tonight:

  # watch for a website to come back online
  # example: github down? do `mashf5 github.com`
  function mashf5() {
      watch -d -n 5 "curl --head --silent --location $1 | grep '^HTTP/'"
  }

Re: GitHub Outage

#263

Post by ivthreadp110 »

AdamJacobMuller wrote: post #239
gjtorikian wrote: post #207

Earlier quoted context omitted.

Hate to be the bearer of bad news, but ikawe was joking about the bower thing. Source: I work at GitHub.

> Source: I work at GitHub. So, you're saying you broke it. Everybody, over here, gang up on this guy.

Shit happens some time. Who cares who broke it- what matters is they're fixing it. I hope github does make public a official reason tomorrow. Not looking to blame anyone- I just want to know what they're doing internally to make sure it does not happen again... adaptation. Even if it's from some external hack or ddos attack- how can they plan on building redundancies in. I am pretty sure GitHub's dev team is talented- I'm being such an anti-troll.

Re: GitHub Outage

#264

Post by AdamJacobMuller »

ivthreadp110 wrote: post #263
AdamJacobMuller wrote: post #239

Earlier quoted context omitted.

> Source: I work at GitHub. So, you're saying you broke it. Everybody, over here, gang up on this guy.

Shit happens some time. Who cares who broke it- what matters is they're fixing it. I hope github does make public a official reason tomorrow. Not looking to blame anyone- I just want to know what they're doing internally to make sure it does not happen again... adaptation. Even if it's from some external hack or ddos attack- how can they plan on building redundancies in. I am pretty sure GitHub's dev team is talented…

My comment was quite joking in nature.

> I hope github does make public a official reason tomorrow

I do too, but, mostly I find post-mortems quite interesting to read.

If your services are at some 95% uptime or lower, you're doing something (serveral things) very wrong and its probably not that interesting to me.

Getting from 95% -> 99% you probably did some interesting things there.

Going into multiple .9s beyond that, you're likely doing quite a bit of interesting stuff but what I find more interesting is where you went wrong. Figuring out not just where you went wrong, but, what your incorrect assumptions were and WHY they were wrong. "We believed X could never fail because of Y, and even if X did fail, it would not cause production impact because of Z!"

Re: GitHub Outage

#265

Post by AdamJacobMuller »

chuhnk wrote: post #94
ikawe wrote: post #27

Sorry, I think I caused this. =[ bower jquery#1.11.3 not-cached git://github.com/jquery/jquery-dist.git#1.11.3 bower jquery#1.11.3 resolve git://github.com/jquery/jquery-dist.git#1.11.3 bower foundation#~5.5.2 cached git://github.com/zurb/bower-foundation.git#5.5.3 bower foundation#~5.5.2 validate 5.5.3 against git://github.com/zurb/bower-foundation.git#~5.5.2 bower ember#^2.3.0 ECMDERR Failed to execute "git ls-remo…

I once ran rm -rf in the production mysql data directory. Shit happens.

Been there. Done that. No backups.

Recovered all the data by using open file handles in /proc/.

Not a fun two hours.

Shit happens. Live and learn.

Re: GitHub Outage

#267

Post by antsar »

hosay123 wrote: post #211
rdl wrote: post #116

They're also on Prolexic right now, so...big DDoS, likely state-sponsored?

How can you spot Prolexic? I just checked their DNS/IPv4 and it seems to be pointing regular hosting

    $ traceroute github.com
    traceroute to github.com (192.30.252.130), 30 hops max, 60 byte packets
    ...
     9  level3-pni.iad1.us.voxel.net (4.53.116.2)  17.609 ms  15.057 ms  10.113 ms
    10  unknown.prolexic.com (209.200.144.192)  9.186 ms  9.462 ms  9.315 ms
    11  unknown.prolexic.com (209.200.144.197)  17.753 ms  17.767 ms  18.851 ms
    12  unknown.prolexic.com (209.200.169.98)  9.922 ms  9.542 ms unknown.prolexic.com (209.200.169.96)  11.471 ms
    13  192.30.252.215 (192.30.252.215)  13.569 ms 192.30.252.207 (192.30.252.207)  9.660 ms 192.30.252.215 (192.30.252.215)  13.150 ms
    14  github.com (192.30.252.130)  9.051 ms  8.833 ms *

Re: GitHub Outage

#268

Post by peterkelly »

philip1209 wrote: post #230

This is reminding me of last year's major Facebook outage. If I recall correctly, that outage was a bug in service discovery that took down all data centers (a CLI accepted a negative value when the Zookeeper variant treated it as an unsigned int , then all service discovery went down). I feel like service discovery is the biggest point of failure at large companies, and it would explain why services across so many d…

Except that when Facebook is down, productivity goes increases, and when GitHub is down, productivity decreases :)

Re: GitHub Outage

#269

Post by AdamJacobMuller »

Interestingly. Our (minimal) traffic to github shifted transit carriers when this outage happened.

Between 2016-01-28 00:39:47+00 and 2016-01-28 00:43:26+00 there was a flurry of BGP updates that caused that.

I'm not sure on the exact timing of the outage, this could either be a symptom or a cause.

Re: GitHub Outage

#270

Post by marco-lavag »

AdamJacobMuller wrote: post #265
chuhnk wrote: post #94

Earlier quoted context omitted.

I once ran rm -rf in the production mysql data directory. Shit happens.

Been there. Done that. No backups. Recovered all the data by using open file handles in /proc/. Not a fun two hours. Shit happens. Live and learn.

Have you written a blog post about that /proc/ trick? sounds like an interesting read
Post reply on HN