Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cause of today's Github outage

github.com

Post reply on HN
Print view View original

31–40 of 119 posts

Re: Cause of today's Github outage

#31

Post by SkyMarshal »

nigelsampson wrote: post #11
aaronbrethorst wrote: post #10

Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing. From Chris' Twitter stream ( http://twitter.com/#!/defunkt ): Seriously, I blame whoever wrote our crappy continuous integration software. Oh that's me

Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.

> Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database.

Or sent a test email to thousands of customers in your prod database encouraging them to use web check-in for their non-existent flight tomorrow.

Yeah, did that five years ago, talk about heart-attack-inducing. Quickly remedied by sending a second email to the same test set, thankfully, but that's the kind of mistake you never forget.

Re: Cause of today's Github outage

#32

Post by storborg »

gregwebs wrote: post #16
woan wrote: post #8

rookie mistake, better security if they isolate their networks too...

I don't know why you first got down-voted. I have never worked on a project where a testing environment could access the production db.

I think he got downvoted for calling the GitHub team "rookies". Sure, they may not have tons of experience running 5-nines systems, but they've clearly built a great product that a lot of people love and respect.

Re: Cause of today's Github outage

#33

Post by catch23 »

devmonk wrote: post #27
gregwebs wrote: post #20

Earlier quoted context omitted.

If your database is on a socket then that would mean tests are running on a production machine, which just seems crazy. If it is on TCP than that means access isn't restricted properly. At any rate how is this Rails specific?

"how is this Rails specific?" I spent years in Java prior to Ruby and through the various frameworks in Java, I never remember making the mistake or working with anyone who made the mistake of blowing away the production data when running tests. Since working in Rails, have had at least 2-3 experiences on the team with truncated/dropped tables or mangled data in production, and it isn't just the current team. Many ot…

I think it could happen just as easily in Java, especially in continuous integration scripts, which are nearly fully automated. I've nearly done something in Hibernate before just by running ant scripts that looked at my environment variables to see which database it should truncate. If I forget to change back the environment variable and my machines were able to access the prod servers, same thing could happen. Only reason it didn't happen for me is because I usually have pretty strict rules in my pg_hba.conf file that prevents foreign hosts from connecting.

Re: Cause of today's Github outage

#34

Post by 1337p337 »

nigelsampson wrote: post #11
aaronbrethorst wrote: post #10

Ouch. I think we've all done this once or twice, in some fashion or another. I'm just happy they're so open about it. Learning experience == good thing. From Chris' Twitter stream ( http://twitter.com/#!/defunkt ): Seriously, I blame whoever wrote our crappy continuous integration software. Oh that's me

Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.

I suspect that this is why people tend to understand when GitHub goes down, and this allows them to be more open. All (or nearly all) of GitHub's users are tech-savvy, so we tend to understand the problems.

Re: Cause of today's Github outage

#35

Post by bytesong »

SkyMarshal wrote: post #28
bytesong wrote: post #23

Minutes before the outage, my account would seem non-existent and all my repositories gone. They really scared the hell out of me. This will be a good reminder for me to always keep a local copy.

Since it's git-based, is it even possible to not keep a local copy? I thought everything that goes into github has to be committed to your local git repo first?

Well, there is always a local copy since you have to commit to the local repo first before you push it to github. That is, of course, as long as you don't remove it for no apparent reason.

Re: Cause of today's Github outage

#36

Post by ethangunderson »

blutonium wrote: post #7
geophile wrote: post #2

Anyone know what database system they use?

MySQL. Not that using anything else would have prevented this or would help restore the table faster...

No system could prevent it, but there are databases(Oracle and MongoDB off the top of my head, and I'm sure there are others) that implement slave delays specifically to help mitigate human error like this.

Re: Cause of today's Github outage

#37

Post by smokinn »

nettdata wrote: post #17
seanmcq wrote: post #4

Lesson, don't let your CI machine talk to your production servers (firewalls are good at this).

In my environment, our Dev's (individuals or environments/subnets) don't have access to PROD or QA, and our CIT boxes are in DEV. Likewise, QA and PROD only have access to their own environments. We have a build master that promotes a reviewed deployment package to QA and/or PROD environments, where the appropriate QA or PROD operations folks do the actual deployment. It's a luxury to have the resources available for…

I'm not so sure it's a luxury. Maybe it is if your startup is servicing a group of techies who have knowledge of what problems lay in the background but most clients don't know and don't care.

It's not that hard to keep up really. It's a question of a day or two of setup and then the hardish part of constant discipline to not take the "easy way out" and poke holes into the segregation you've set up. Mainly it takes a single team lead or CTO or whatever to be really explicit that you just don't break the steps and you'll avoid a LOT of problems. You'll still have problems, problems are inevitable, but in general you'll have mitigated them and with a proper backup and merge procedure you'll minimize downtime.

Re: Cause of today's Github outage

#38

Post by invisible »

tlb wrote: post #22

Forthright and classy. Compare to register.com, which had a big DNS outage Friday (affecting anybots.com) and never admitted to a problem.

I believe they had a message on their homepage during the event about being DDoSed, but yeah no after-remarks is kind of ugly in my opinion. They could at least post and hide it from the general customer.

1) http://seclists.org/nanog/2010/Nov/415

Re: Cause of today's Github outage

#39

Post by ammmir »

simply checking if you're talking to a production instance could avert something like this. having some metadata in the db about whether the data stored there is acting as production and at what version and deployment level, so tests can have a sanity check before destructive activities.

Re: Cause of today's Github outage

#40

Post by jacques_chester »

Well done on coming clean. However this is why the dinosaur pens have such arduous red tape -- to try and catch serious errors before they hit production. A mate of mine works in that world and he regularly stops code going into production that would hose mission-critical government data.

I prefer my agility to remain on the dev-and-test side of the fence.

Post reply on HN