Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cause of today's Github outage

github.com

Post reply on HN
Print view View original

71–80 of 119 posts

Re: Cause of today's Github outage

#71

Post by scorpion032 »

random42 wrote: post #59

I am a software developer, so I know "shit happens", but having the same configuration for database as testing environment, (same superuser name and password), which is not isolated from test environment, is pretty criminal even for a first time mistakeIMHO, especially for a product like github whom business, small and big trust with there business critical piece ("repository"). If I were running some critical code,…

I think the fact that they have been able to recover all critical data soon and will even update the event table over the next few days clearly displays their competence.

Github isn't facebook and not seeing the last week's "activities" timeline for a few days isn't really that much of a problem.

Re: Cause of today's Github outage

#72

Post by chesser »

latch wrote: post #26

I agree with what's been said so far. 1 - Shit happens 2 - We've all done stupid stuff 3 - Testing environment shouldn't have access to production What hasn't been said is how refreshing it is to see an honest and quick explanation. I know this type of approach is getting more and more common (see the foursquare outage), but in the grand scale of things, its still quite rare.

I have a slightly higher threshold for considering something refreshing.

It took several levels of basic mistakes for this to happen AND for the restore to be as slow as it is.

Using MySQL with no transactions, no binary backups, no way to do a quick restore, and no separation of dev/production. DROP as opposed to DELETE cannot be rolled back and is therefore scary -- unless you aren't using transactions in the first place, in which case, WHEE!

Re: Cause of today's Github outage

#73

Post by xentronium »

aaronbrethorst wrote: post #10

Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing. From Chris' Twitter stream ( http://twitter.com/#!/defunkt ): Seriously, I blame whoever wrote our crappy continuous integration software. Oh that's me

I didn't even notice, maybe because it was Sunday and none of my data seems to have been lost. Well, except for the wiped events but I can live with that.

Re: Cause of today's Github outage

#74

Post by michael_dorfman »

peterwwillis wrote: post #44

If someone ever produces a good book of best practices for sysadm/syseng, please provide examples like these of why it's important to follow these best practices. Yes, we've all made silly mistakes. But if you're in that design meeting and somebody asks, should we do ABC in case of XYZ, try not to think about how complicated or time consuming it might be to do ABC. Think about the worst case. If not doing it could at…

While a book of screw-ups might be amusing, I think it might be more instructive to look at the "old school" ways that screw-ups like this were avoided.

I worked, back in the dark ages, for a health insurer that had two parallel environments-- one for testing/development, and one for production. None of the developers were allowed anywhere near the production environment. There was one full time employee, a former developer, whose primary job was to move code into production-- which he would only do when he received a signed document authorizing the change. Said document included the telephone numbers where the developer responsible for the code would be for the next 24 hours, so that one of the operators could call you in the middle of the night if your code caused any problems to the system.

At the time, I used to think that this was ridiculous. After managing a staff of programmers myself, I'm not so sure.

Re: Cause of today's Github outage

#75

Post by chesser »

alextgordon wrote: post #24
binaryfinery wrote: post #13

Earlier quoted context omitted.

I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises some basic questions about password storage. I don't run a site anything like github, but my production and test databases have different passwords and none of them are stored in a way that the test environment could get access to the live db, nor could the test…

If github spent their time so that their site raised absolutely no basic questions, then they'd still be in beta by now.

To me, that sounds like encouraging a race to the bottom.

When you're in the business of storing other people's data, things like transactions, binary backups, QUICK RESTORES, and separating dev from production shouldn't be afterthoughts. They are core attributes.

I would understand this for a demo, but not after a couple of years and half a million users. They have paying customers.

"Reliable code hosting

We spend all day and night making sure your repositories are secure, backed up and always available."

Re: Cause of today's Github outage

#76

Post by michael_dorfman »

seanmcq wrote: post #4

Lesson, don't let your CI machine talk to your production servers (firewalls are good at this).

Nice, but still too specific.

Lesson: don't let anything talk to your production servers.

Make it as difficult as possible for anyone to log in. Treat production as if it were a loaded gun. You shouldn't be touching it unless you absolutely have to, and even then, you need to be vigilantly aware of the consequences of your actions.

Re: Cause of today's Github outage

#77

Post by donw »

This is why it's important to isolate production from other environments. Three rules have kept me from ever borking a production database:

1. Production DB credentials are only stored on the production appservers, and copied in at deploy time.

2. The production DB can only be accessed from the IPs of the production webfarm.

3. Staging, Testing, Development, and Everything Else live on separate networks and machines than production.

Re: Cause of today's Github outage

#78

Post by 1337p337 »

haploid wrote: post #67
1337p337 wrote: post #34

Earlier quoted context omitted.

I suspect that this is why people tend to understand when GitHub goes down, and this allows them to be more open. All (or nearly all) of GitHub's users are tech-savvy, so we tend to understand the problems.

It also doesn't hurt that every user's repositories are "backed up" to localhost. Sounds like the biggest impact was on non-github-users trying to read source or documentation on other people's projects moreso than actual users.

Definitely the important part. Linus Torvalds once joked: "Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it." Distributed SCMs like git are a great way to

But GitHub's main value over something like, say, gitosis, is all of the metadata. This stuff is mainly useful to github users. Projects you're watching, pull requests, bug reports, fun (if only of dubious general use) graphs. That stuff doesn't come across the wire with the individual repos. The big (in terms of bytes) part, the "events" table, is helpful for keeping up with people and projects (if sometimes noisy). I use the RSS feed of that, though, so I didn't even notice the interruption until I heard about it here.

Re: Cause of today's Github outage

#79

Post by 1337p337 »

random42 wrote: post #59

I am a software developer, so I know "shit happens", but having the same configuration for database as testing environment, (same superuser name and password), which is not isolated from test environment, is pretty criminal even for a first time mistakeIMHO, especially for a product like github whom business, small and big trust with there business critical piece ("repository"). If I were running some critical code,…

I can only hope it shocks some sense into kids that use GitHub for distribution rather than putting a tarball named $name-$version.tgz (or bz2 or xz or whatever). As much as I love GitHub, is has been the bane of my automated-build existence. I don't want to ever have to make a build script that guesses at a SHA1 (or punts by doing a clone at depth zero) again.
Post reply on HN