Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cause of today's Github outage

github.com

Post reply on HN
Print view View original

81–90 of 119 posts

Re: Cause of today's Github outage

#81

Post by 1337p337 »

resdirector wrote: post #42
SkyMarshal wrote: post #31

Earlier quoted context omitted.

> Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Or sent a test email to thousands of customers in your prod database encouraging them to use web check-in for their non-existent flight tomorrow. Yeah, did that five years ago, talk about heart-attack-inducing. Quickly remedied by sending a second e…

I have a strict rule for myself: never use any curse words in any comments, variable names, dummy accounts etc.

I don't think they're useful for variable names, but I use them (very sparingly) in comments. A great way to spot what needs refactoring is to find the piece that caused the most frustration to the author. Under a tight deadline, apologies to future maintainers can also be helpful in injecting some levity.

Re: Cause of today's Github outage

#82

Post by 1337p337 »

jrockway wrote: post #49
resdirector wrote: post #42

Earlier quoted context omitted.

I have a strict rule for myself: never use any curse words in any comments, variable names, dummy accounts etc.

I have the opposite strict rule: use as many curse words in comments, variable names, and dummy accounts as possible. That way you'll find out quickly when someone else notices!

I recommend against that. :) A team I was on had a demo for a client, and a shaky database schema. I had used, as a test account, the username "MOTHERFUCKINYEAH" for the same purpose. The purging of this account caused a few 500 errors, and we almost lost the client, and although I didn't get fired, we were all shuffled around after that to less critical projects, and one of us actually got sent out of the state.

Re: Cause of today's Github outage

#83

Post by chesser »

alextgordon wrote: post #24
binaryfinery wrote: post #13

Earlier quoted context omitted.

I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises some basic questions about password storage. I don't run a site anything like github, but my production and test databases have different passwords and none of them are stored in a way that the test environment could get access to the live db, nor could the test…

If github spent their time so that their site raised absolutely no basic questions, then they'd still be in beta by now.

> If github spent their time so that their site raised absolutely no basic questions, then they'd still be in beta by now.

Beta is the stage you reach AFTER you flip on transactions, institute and test quick restores, and separate dev from production.

Re: Cause of today's Github outage

#84

Post by burgerbrain »

axod wrote: post #80
burgerbrain wrote: post #3

Good thing git is distributed. I've been working on my code all day and never even noticed!

So what's the point of using github if you don't notice it's offline?

Github provides a convenient place for me to publish my work to the world, allowing others to pull it as they desire (my work machine is a laptop, and can never be relied upon to be up, or at any particular address). Because git is distributed my workflow is not effected in the slightest by "somebody else's" (github's) completely separate repo being down.

Contrast this with something older like subversion and sourceforge. If sourceforge went down you were shit out of luck.

Re: Cause of today's Github outage

#85

Post by chesser »

storborg wrote: post #32
gregwebs wrote: post #16

Earlier quoted context omitted.

I don't know why you first got down-voted. I have never worked on a project where a testing environment could access the production db.

I think he got downvoted for calling the GitHub team "rookies". Sure, they may not have tons of experience running 5-nines systems, but they've clearly built a great product that a lot of people love and respect.

They may not be rookies as Ruby hackers, but the evidence clearly points to them being rookies from the standpoint of data robustness. That's embarrassing enough when it's not your core business offering.

They even brag about it on their main page.

Re: Cause of today's Github outage

#86

Post by chesser »

scorpion032 wrote: post #71
random42 wrote: post #59

I am a software developer, so I know "shit happens", but having the same configuration for database as testing environment, (same superuser name and password), which is not isolated from test environment, is pretty criminal even for a first time mistakeIMHO, especially for a product like github whom business, small and big trust with there business critical piece ("repository"). If I were running some critical code,…

I think the fact that they have been able to recover all critical data soon and will even update the event table over the next few days clearly displays their competence. Github isn't facebook and not seeing the last week's "activities" timeline for a few days isn't really that much of a problem.

> I think the fact that they have been able to recover all critical data soon and will even update the event table over the next few days clearly displays their competence.

Taking a few days to restore a table indicates they only had a text dump of it. The details of how they screwed up indicate they were new to the idea of data robustness, despite it being the core of their business.

Being a competent home user and being a competent data-robustness company involve slightly different levels of expertise.

Re: Cause of today's Github outage

#88

Post by joelg87 »

lambda wrote: post #51
SkyMarshal wrote: post #31

Earlier quoted context omitted.

> Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Or sent a test email to thousands of customers in your prod database encouraging them to use web check-in for their non-existent flight tomorrow. Yeah, did that five years ago, talk about heart-attack-inducing. Quickly remedied by sending a second e…

Heck, you think that's heart attack inducing? How about the DreamHost case in which they typed the wrong year in their billing code; charged many of their users for an extra year of service, to the tune of $7.5 million: http://blog.dreamhost.com/2008/01/15/um-whoops/ http://blog.dreamhost.com/2008/01/16/the-aftermath/ http://blog.dreamhost.com/2008/01/17/the-final-update/

Wow. That is a great read.

“What’s the harm in keeping it flexible?” $7.5M in harm, that’s what! "flexibility" is rarely desired in programming!

Re: Cause of today's Github outage

#89

Post by Jabbles »

nigelsampson wrote: post #11
aaronbrethorst wrote: post #10

Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing. From Chris' Twitter stream ( http://twitter.com/#!/defunkt ): Seriously, I blame whoever wrote our crappy continuous integration software. Oh that's me

Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.

Like when the google engineer added "/" to the list of bad URLs, thereby marking every single website (!) as possibly dangerous.

http://googleblog.blogspot.com/2009/01/this-site-may-harm-yo...

Now that would be a bad feeling.

Post reply on HN