271–280 of 297 posts
Apparently they can see into the future and know for a fact that they'll still be having issues in a few hours. January 28, 2016 00:00 EST The status is still red at the beginning of the day January 27, 2016 20:02 EST We're continuing to investigate a significant network disruption affecting all github.com services.
Earlier quoted context omitted.
You shouldn't notice a webhook deluge because the site isn't generating events. I'm watching our webhook services though and will let you know if that changes.
Hi Kyle! It looks like webhooks are wedged.. no?
Developers: "I can't get any work done because GitHub is down!" Linus Torvalds: [ facepalm ]
I honestly can't get much work done now, I've been looking for RabbitMQ auth plugin sample code and every link I'm clicking now shows me a unicorn.
> January 28, 2016 > 00:00 EST The status is still red at the beginning of the day They seem to have some time issues as well since it's not yet January 28th in EST.
Earlier quoted context omitted.
Been there. Done that. No backups. Recovered all the data by using open file handles in /proc/. Not a fun two hours. Shit happens. Live and learn.
Have you written a blog post about that /proc/ trick? sounds like an interesting read
Earlier quoted context omitted.
The bigger problem here is the number of build packs only relying on Github as a source of truth. We need to find an abstraction to distribute storage so that there is no single point of failure.
Exactly. So I had to install Gems from RubyGems, not that big of a deal, and I had to look them up on RubyGems since Google gives me GitHub first, but that's OK too, but then all the documentation seems to be on.... Github. Except not, it's also on RDoc. (Though RDoc kinda sucks, compared to GitHub...) Pretty big win imho, Rails got a few more points with that. :D Now I'm wondering how NodeJS is faring at this...
Earlier quoted context omitted.
So... you were doing a production redeploy, and it crashed? The URL "git://github.com/components/ember.git" [1] suggests this is an internal GitHub Bower build log, but your post history doesn't mention anything about GitHub (let alone whether you work there), so I'm not 100.00% sure. [1] https://webcache.googleusercontent.com/search?q=cache:3e00jl... Assuming this is, in fact, a GH Bower log, the first thing that ca…
Hate to be the bearer of bad news, but ikawe was joking about the bower thing. Source: I work at GitHub.
But now my "technical breakdown info" box has no tidbits in it. :P
I'm glad you're back up now (sortakinda - what sort of traffic are you sustaining right now? :D), but a rough idea of what asploded would be really cool to know about.
Speaking of which, I'd like to take a moment to make a strong point about the fact that disaster-recovery situations don't get blogged about enough. Vague "we fixed it" datapoints get buried in status update logs like it's something to hide and hope nobody brings up.
In situations like these, the only constructive perspective is for everyone to accept that something went horribly wrong and not make a fuss about it, and if such a mentality can be established, this creates an environment within which we can share technical breakdowns of "we found ourselves in XYZ position and then we did these thirty highly specific things in heroically record time to be up and running again", and I think sharing this type of info would potentially be more educational than setup tutorials or the "we switched to X and it improved Y by 1400%" type things the Net's full of. Sure, you'd have to generalize and probably give a lot of backstory about infrastructure, but it's becoming trendy (in a sense) for companies to describe their operations in precisely this way, so it's not completely nonviable.
(Note my use of the angle of "we found ourselves in XYZ position" - maybe a small highlight of what led up to the disaster would be included (worth considering if the information would be educational), maybe not. In a blog context, moderating comments to keep the discussion on-track and constructive may be necessary, but IMO would be worth it.)