Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cause of today's Github outage

github.com

Post reply on HN
Print view View original

111–119 of 119 posts

Re: Cause of today's Github outage

#111

Post by matwood »

chesser wrote: post #72
latch wrote: post #26

I agree with what's been said so far. 1 - Shit happens 2 - We've all done stupid stuff 3 - Testing environment shouldn't have access to production What hasn't been said is how refreshing it is to see an honest and quick explanation. I know this type of approach is getting more and more common (see the foursquare outage), but in the grand scale of things, its still quite rare.

I have a slightly higher threshold for considering something refreshing . It took several levels of basic mistakes for this to happen AND for the restore to be as slow as it is. Using MySQL with no transactions, no binary backups, no way to do a quick restore, and no separation of dev/production. DROP as opposed to DELETE cannot be rolled back and is therefore scary -- unless you aren't using transactions in the firs…

I don't have much MySQL experience, but have lots of experience with other RDBMSs. Large restores take a long time no matter what. They shouldn't take days, but since we don't know how large the events table was we have no way of knowing if a faster restore was possible. A hot restore which it sounds like what they are doing may take longer by simple fact that it's restoring while the table is in use.

And DROPs (or truncates) are almost always used if the goal is to remove all the data in a table like you would want if rebuilding the entire system. Something like 100M record transaction is not generally considered a good thing.

Re: Cause of today's Github outage

#112

Post by [deleted] »

pixdamix wrote: post #68
seldo wrote: post #54

Earlier quoted context omitted.

My worst was discovering I had written a unique ID generator which was (due to me typing "==" instead of "!="), producing duplicate IDs -- and not only that, it was producing them at exponentially increasing rates -- and every duplicate ID was destroying an association in our database, making it unclear what records belonged to who. It was not a good day.

Mine was for a French social networking site 4 years ago. They used to send mails everyday to say "hey look at the people who you might know". The links on the mail would automatically log the user on the website. When I sent the code live it took 2 days (and more than 50000 mails to found out that when I sent a mail to person Z about person Y the link logged in Z ON Y's account. [1] I knew it was a bad idea to autom…

[deleted]

Re: Cause of today's Github outage

#113

Post by burgerbrain »

chesser wrote: post #107
risotto wrote: post #104

Earlier quoted context omitted.

Are you sitting in a chair? Do you not work at Github? Then you're armchair quarterbacking. We all are. Even if you are a quarterback for another team (I'm a DBA myself) Anyway, it's a great discussion so we can learn from other peoples mistakes. I will be triple checking my restores later today, and likely halt the project I'm working on to get cold standbys shored up asap. (but it's hard to prioritize house keeping…

"a pejorative modifier to refer to a person who experiences something vicariously rather than first-hand, or to a casual critic who lacks practical experience " Ironically the most fitting application would be to say that they're armchair quarterbacking their own database administering.

Unless you work for github, you lack practical experience in github's internal operation.

Re: Cause of today's Github outage

#114

Post by jmulho »

trezor wrote: post #66
blutonium wrote: post #7

Earlier quoted context omitted.

MySQL. Not that using anything else would have prevented this or would help restore the table faster...

Incorrect. The article states that they lost data between backup time and whenever the incident happened. Any reasonable quality transactional database system has a transaction log for a reason . If you were using commercial DB system like Microsoft SQL Server or Oracle (even a decade ago), this would not be an issue. No data loss. Businesses should care about their data, and I guess this is why the commercial databa…

You are right on. The cardinal rule of traditional relational databases is that data loss is unacceptable. A database administrator who allows data loss is by definition incompetent. A database that allows data loss is by definition not a database.

Commercial databases such as Oracle, SQL Server, and DB2 are built from the ground up to make it possible to recover everything up through the last committed transaction before the incident. I have always presumed that the leading open source traditional relational databases (mySQL and PostgreSQL) provide this same capability.

In the github case (mySQL) it seems that full recovery may have been possible, but it was not possible within an acceptable timeframe. This may be a flaw in the software (lack of features to make quick recovery possible), or a flaw in the strategy (not configured to make quick recovery possible).

However, any of these databases can easily be administered in a manner that allows data loss. The github event is not necessarily an indication of a problem with mySQL. It is unquestionably an indication of a problem with the administration of the database.

Can anyone confirm that full recovery from this type of incident (within a reasonable timeframe) should be possible with both mySQL and PostgreSQL if administered properly?

Re: Cause of today's Github outage

#115

Post by Fluxx »

peterwwillis wrote: post #44

If someone ever produces a good book of best practices for sysadm/syseng, please provide examples like these of why it's important to follow these best practices. Yes, we've all made silly mistakes. But if you're in that design meeting and somebody asks, should we do ABC in case of XYZ, try not to think about how complicated or time consuming it might be to do ABC. Think about the worst case. If not doing it could at…

We have a great sysadmin at work, and I've learned 3 very important lessons:

Have redundant everything. Backup everything. Provide tools to talk to production systems, databases, but make them read only and make it hard to mess with production systems unless you're reallllllly sure you know what you're doing.

Re: Cause of today's Github outage

#116

Post by peterwwillis »

grandpa wrote: post #108
peterwwillis wrote: post #44

If someone ever produces a good book of best practices for sysadm/syseng, please provide examples like these of why it's important to follow these best practices. Yes, we've all made silly mistakes. But if you're in that design meeting and somebody asks, should we do ABC in case of XYZ, try not to think about how complicated or time consuming it might be to do ABC. Think about the worst case. If not doing it could at…

97 Things Every Software Architect Should Know is a pretty good starting point. http://97things.oreilly.com/wiki/index.php/97_Things_Every_S...

Thanks, i'll give this a read.

Re: Cause of today's Github outage

#117

Post by peterwwillis »

viraptor wrote: post #93
michael_dorfman wrote: post #74

Earlier quoted context omitted.

While a book of screw-ups might be amusing, I think it might be more instructive to look at the "old school" ways that screw-ups like this were avoided. I worked, back in the dark ages, for a health insurer that had two parallel environments-- one for testing/development, and one for production. None of the developers were allowed anywhere near the production environment. There was one full time employee, a former de…

It depends on the software a bit, but I got a taste of "programmers not allowed on production" environment (in a very small scale). The problem I run into is that you have no idea what's going on on the production environment. Some characteristics are maybe reproducible on the dev site, but actual users will always do something differently. Sometimes not being able to poke the live system in a specific way while it's…

In my experience the best solution for a large-scale dynamic site is a combination of read-only access to production and deploy management.

Deploy management is the combination of a change management system with a deployment tool in a flexible way. So for example, with the right options, in an emergency the manager of a team of developers could deploy anything to the site immediately without requiring approval from change management. The tools are still right there to revert any change, and of course there's snapshots and daily backups for the most critical data.

Except for emergencies, all changes to production would come with an approval from a higher-up with potential code review back-and-forth first. Contact data and reversion capabilities are built in, so everybody knows who did what, how to contact them and how to revert it if they're unavailable. And of course your trending data will tell you when your peak use is and code pushes are typically frozen during that time, minimizing further potential loss.

However, besides having read-only access to production, devs should also have two kinds of testing: "development" and "staging". Development is where the bleeding-edge broken stuff lives and code is written. Staging is an identical machine to those in production. Often you'll see test or qa machines which aren't identical to production, usually because changes aren't pushed to them the way they are in production. The staging machine gets all changes pushed to it like any other production machine, except it lives in a LAN that cannot access production or anything else. A method to reproduce incoming requests and sessions from production system to this staging server will give you a pretty good idea what "real traffic" looks like on this box, if you need it.

Re: Cause of today's Github outage

#118

Post by chesser »

burgerbrain wrote: post #113
chesser wrote: post #107

Earlier quoted context omitted.

"a pejorative modifier to refer to a person who experiences something vicariously rather than first-hand, or to a casual critic who lacks practical experience " Ironically the most fitting application would be to say that they're armchair quarterbacking their own database administering.

Unless you work for github, you lack practical experience in github's internal operation.

Fortunately, there is this thing called "science" which means we can understand things about the world regardless of where we live. As Dawkins would say, there is no such thing as "Chinese Science" or "French Science", just science. Similarly, there is no such thing as "Github MySQL" or "Github separation of production and development systems" in that same sense.

These are categorical mistakes.

Re: Cause of today's Github outage

#119

Post by chesser »

matwood wrote: post #111
chesser wrote: post #72

Earlier quoted context omitted.

I have a slightly higher threshold for considering something refreshing . It took several levels of basic mistakes for this to happen AND for the restore to be as slow as it is. Using MySQL with no transactions, no binary backups, no way to do a quick restore, and no separation of dev/production. DROP as opposed to DELETE cannot be rolled back and is therefore scary -- unless you aren't using transactions in the firs…

I don't have much MySQL experience, but have lots of experience with other RDBMSs. Large restores take a long time no matter what. They shouldn't take days, but since we don't know how large the events table was we have no way of knowing if a faster restore was possible. A hot restore which it sounds like what they are doing may take longer by simple fact that it's restoring while the table is in use. And DROPs (or t…

They take a long time if you have to rebuild the indexes, which is what you have to do if you only have a text dump.

For MySQL, using the standard format (MyISAM), you can just do a file copy if you bothered to do a proper backup of the binary files.

Post reply on HN