Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cause of today's Github outage

github.com

Post reply on HN
Print view View original

101–110 of 119 posts

Re: Cause of today's Github outage

#101

Post by arethuza »

viraptor wrote: post #93
michael_dorfman wrote: post #74

Earlier quoted context omitted.

While a book of screw-ups might be amusing, I think it might be more instructive to look at the "old school" ways that screw-ups like this were avoided. I worked, back in the dark ages, for a health insurer that had two parallel environments-- one for testing/development, and one for production. None of the developers were allowed anywhere near the production environment. There was one full time employee, a former de…

It depends on the software a bit, but I got a taste of "programmers not allowed on production" environment (in a very small scale). The problem I run into is that you have no idea what's going on on the production environment. Some characteristics are maybe reproducible on the dev site, but actual users will always do something differently. Sometimes not being able to poke the live system in a specific way while it's…

Even in fairly conservative financial environments I've seen developers getting read-only access to production application servers and databases to troubleshoot specific problems.

Re: Cause of today's Github outage

#102

Post by mml »

catch23 wrote: post #33
devmonk wrote: post #27

Earlier quoted context omitted.

"how is this Rails specific?" I spent years in Java prior to Ruby and through the various frameworks in Java, I never remember making the mistake or working with anyone who made the mistake of blowing away the production data when running tests. Since working in Rails, have had at least 2-3 experiences on the team with truncated/dropped tables or mangled data in production, and it isn't just the current team. Many ot…

I think it could happen just as easily in Java, especially in continuous integration scripts, which are nearly fully automated. I've nearly done something in Hibernate before just by running ant scripts that looked at my environment variables to see which database it should truncate. If I forget to change back the environment variable and my machines were able to access the prod servers, same thing could happen. Only…

Not really. Reasonably complex java apps would use jndi to store a single db connect name, which the app server maps internally to db credentials.

That way, the app just says "connect me to /x" and the container gives the app /x, which might be the test or prod db connection, the app never knows, or cares. You get the db that the operations guys configured for -that- environment.

Jndi is horrible in many ways, and so are app containers, but there are a few niceties that ruby-land could learn from.

Re: Cause of today's Github outage

#103

Post by chesser »

alextgordon wrote: post #96
chesser wrote: post #75

Earlier quoted context omitted.

To me, that sounds like encouraging a race to the bottom. When you're in the business of storing other people's data, things like transactions, binary backups, QUICK RESTORES, and separating dev from production shouldn't be afterthoughts. They are core attributes. I would understand this for a demo, but not after a couple of years and half a million users. They have paying customers. "Reliable code hosting We spend a…

If we're going to use hindsight, we might as well look at the cause and the effect. Yes, they missed out some pretty obvious things. And what happened? A few hours of downtime because their restore was slow, plus a tiny bit of inconsequential data loss. Hardly a catastrophe. The fact is that every site has some sort of problems. Many of them will be completely obvious like this one. And while github could have gone t…

Your post is a good illustration of the type of attitude we want to scrupulously avoid when appointing someone to be in charge of a database.

Re: Cause of today's Github outage

#104

Post by risotto »

chesser wrote: post #100
risotto wrote: post #97

Earlier quoted context omitted.

The half a million users validates their techniques, no matter how much armchair quarterbacking. The code repos were never in danger, and they've been killing it in the market because they are racing to add awesome extra features, not racin to the bottom. I don't disagree that they can do more in their db operations, and that it's fine for us paying customers to demand more, but the reality of startups is that it ver…

> The half a million users validates their techniques, Let me introduce you to my friend, GeoCities. Funny thing about free hosting. > no matter how much armchair quarterbacking. Transactions aren't armchair quarterbacking. Binary backups aren't armchair quarterbacking. Separating development from production isn't armchair quarterbacking. Kindly, you have no idea what you're talking about. You can't give them credit…

Are you sitting in a chair? Do you not work at Github? Then you're armchair quarterbacking. We all are. Even if you are a quarterback for another team (I'm a DBA myself)

Anyway, it's a great discussion so we can learn from other peoples mistakes. I will be triple checking my restores later today, and likely halt the project I'm working on to get cold standbys shored up asap. (but it's hard to prioritize house keeping over customer centric features in the race I'm running along side the Github crew)

Re: Cause of today's Github outage

#105

Post by zaphar »

risotto wrote: post #97
chesser wrote: post #75

Earlier quoted context omitted.

To me, that sounds like encouraging a race to the bottom. When you're in the business of storing other people's data, things like transactions, binary backups, QUICK RESTORES, and separating dev from production shouldn't be afterthoughts. They are core attributes. I would understand this for a demo, but not after a couple of years and half a million users. They have paying customers. "Reliable code hosting We spend a…

The half a million users validates their techniques, no matter how much armchair quarterbacking. The code repos were never in danger, and they've been killing it in the market because they are racing to add awesome extra features, not racin to the bottom. I don't disagree that they can do more in their db operations, and that it's fine for us paying customers to demand more, but the reality of startups is that it ver…

    "If you don't thing Github cuts the mustard, svn on
     Google Code probably won't have problems like this..."
Neither will Mercurial on Google Code ;-)

Disclaimer: I work for google on the Project Hosting product.

Re: Cause of today's Github outage

#106

Post by risotto »

zaphar wrote: post #105
risotto wrote: post #97

Earlier quoted context omitted.

The half a million users validates their techniques, no matter how much armchair quarterbacking. The code repos were never in danger, and they've been killing it in the market because they are racing to add awesome extra features, not racin to the bottom. I don't disagree that they can do more in their db operations, and that it's fine for us paying customers to demand more, but the reality of startups is that it ver…

"If you don't thing Github cuts the mustard, svn on Google Code probably won't have problems like this..." Neither will Mercurial on Google Code ;-) Disclaimer: I work for google on the Project Hosting product.

Quite fair! Google code is sweet

Re: Cause of today's Github outage

#107

Post by chesser »

risotto wrote: post #104
chesser wrote: post #100

Earlier quoted context omitted.

> The half a million users validates their techniques, Let me introduce you to my friend, GeoCities. Funny thing about free hosting. > no matter how much armchair quarterbacking. Transactions aren't armchair quarterbacking. Binary backups aren't armchair quarterbacking. Separating development from production isn't armchair quarterbacking. Kindly, you have no idea what you're talking about. You can't give them credit…

Are you sitting in a chair? Do you not work at Github? Then you're armchair quarterbacking. We all are. Even if you are a quarterback for another team (I'm a DBA myself) Anyway, it's a great discussion so we can learn from other peoples mistakes. I will be triple checking my restores later today, and likely halt the project I'm working on to get cold standbys shored up asap. (but it's hard to prioritize house keeping…

"a pejorative modifier to refer to a person who experiences something vicariously rather than first-hand, or to a casual critic who lacks practical experience"

Ironically the most fitting application would be to say that they're armchair quarterbacking their own database administering.

Re: Cause of today's Github outage

#108

Post by grandpa »

peterwwillis wrote: post #44

If someone ever produces a good book of best practices for sysadm/syseng, please provide examples like these of why it's important to follow these best practices. Yes, we've all made silly mistakes. But if you're in that design meeting and somebody asks, should we do ABC in case of XYZ, try not to think about how complicated or time consuming it might be to do ABC. Think about the worst case. If not doing it could at…

97 Things Every Software Architect Should Know is a pretty good starting point.

http://97things.oreilly.com/wiki/index.php/97_Things_Every_S...

Re: Cause of today's Github outage

#109

Post by yosh »

1337p337 wrote: post #79
random42 wrote: post #59

I am a software developer, so I know "shit happens", but having the same configuration for database as testing environment, (same superuser name and password), which is not isolated from test environment, is pretty criminal even for a first time mistakeIMHO, especially for a product like github whom business, small and big trust with there business critical piece ("repository"). If I were running some critical code,…

I can only hope it shocks some sense into kids that use GitHub for distribution rather than putting a tarball named $name-$version.tgz (or bz2 or xz or whatever). As much as I love GitHub, is has been the bane of my automated-build existence. I don't want to ever have to make a build script that guesses at a SHA1 (or punts by doing a clone at depth zero) again.

git ls-remote will help you to get the hash of HEAD. It's much cheaper than doing a zero-depth clone.

Re: Cause of today's Github outage

#110

Post by oddi »

donw wrote: post #77

This is why it's important to isolate production from other environments. Three rules have kept me from ever borking a production database: 1. Production DB credentials are only stored on the production appservers, and copied in at deploy time. 2. The production DB can only be accessed from the IPs of the production webfarm. 3. Staging, Testing, Development, and Everything Else live on separate networks and machines…

While this is part of the problem, to me it seems like they didn't have proper restore procedures or at least hadn't tested them enough. There are countless ways to corrupt a database and restoring from backup is part of most remedies.
Post reply on HN