[deleted]
Cause of today's Github outage
61–70 of 119 posts
Re: Cause of today's Github outage
#62I moved my repos off GitHub to my EC2 server a month or two back since they're private and I was only using GitHub for keeping a copy of my code offsite. It's faster for simple push/pull and considering the sunk cost of my EC2 server, also free. I was trying to browse some repos on GitHub yesterday during the downtime and was thankful that my own were still available.
Also, if you just need something for replicating code, Amazon is offering free micro servers on new accounts for up to a year I think.
Re: Cause of today's Github outage
#63Minutes before the outage, my account would seem non-existent and all my repositories gone. They really scared the hell out of me. This will be a good reminder for me to always keep a local copy.
I was walking through someone's capistrano scripts, and suddenly everything came back "Not Found". My first thought was that the guy had just starting cleaning up his account. :)
Re: Cause of today's Github outage
#64Kudos to them for having the guts to say publicly that they accidentally destroyed their production database.
It's been a great service, and I think as long as this kind of thing is rare, and none of my code repositories get corrupted or destroyed, I plan to stick with them.
Re: Cause of today's Github outage
#65Second time I've heard of this happening fairly recently. Another incident, same cause. http://www.bigdoor.com/blog/bigdoor-api-service-has-been-res...
This was pretty painful. Their backups were too old, so it was necessary to do InnoDB data recovery rather than a straight-forward restore from backup.
Since InnoDB table spaces never shrink, 80G of their truncated data was still all available in a single monolithic ibdata file. An InnoDB recovery tool named page_parser read their 80G ibdata file and spits out a maze of 16k InnoDB page files organized by an arbitrary index id.
There are two internal InnoDB meta tables called SYS_INDEXES and SYS_TABLES which can give you a mapping from table name to PK index id. Unfortunately after the mass TRUNCATE all the tables got new index mappings, so it became a bit of table hide-and-seek.
The InnoDB recovery tools lack a certain polish and maturity. You need to create a C header file for each table you want recover from the pile of 16k page files. You end up having to build a separate version of the constraints_parser binary for each table. There were bugs with the output of negative numbers, unicode handling, VARCHAR types with >= 127 characters, and some edge cases where InnoDB chooses to store CHAR(15) types as VARCHARs internally. Aleksander at Percona really saved the day, he was able to find and fix these bugs pretty quickly.
I remember that magic moment when I finally was able to successfully checksum a huge block of the recovered data against the too-old-to-be-useful backup.
"I love the smell of CRC32 in the morning. It smells like... victory."
Re: Cause of today's Github outage
#66Anyone know what database system they use?
MySQL. Not that using anything else would have prevented this or would help restore the table faster...
Incorrect. The article states that they lost data between backup time and whenever the incident happened.
Any reasonable quality transactional database system has a transaction log for a reason. If you were using commercial DB system like Microsoft SQL Server or Oracle (even a decade ago), this would not be an issue. No data loss. Businesses should care about their data, and I guess this is why the commercial databases are still doing fine in a landscape increasingly dominated by FOSS everywhere.
I realize licensing costs do matter, but I can't fathom why people put up with the sad excuse of a RDBMS which is MySQL. For any nontrivial tasks it is slow, it's unreliable, attempting to secure your data by taking database backups (which seemingly can't even provide you with transactional safety!!!) renders the DB unusable and locked while backup is performed, having databases bigger than what you can store in memory makes it perform like a flat-file, etc etc ad infitum.
Surely there must be something better people can use which is still free? Postgres for instance?
Re: Cause of today's Github outage
#67Earlier quoted context omitted.
Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.
I suspect that this is why people tend to understand when GitHub goes down, and this allows them to be more open. All (or nearly all) of GitHub's users are tech-savvy, so we tend to understand the problems.
It also doesn't hurt that every user's repositories are "backed up" to localhost.
Sounds like the biggest impact was on non-github-users trying to read source or documentation on other people's projects moreso than actual users.
Re: Cause of today's Github outage
#68Earlier quoted context omitted.
Exactly, I don't think you've really lived till you've experienced that pit of your stomach feeling when you realise you've just wiped out a product website / database. Thankfully for me it was a small website and no one really noticed. I can't imagine what that feeling would be like on something like github.
My worst was discovering I had written a unique ID generator which was (due to me typing "==" instead of "!="), producing duplicate IDs -- and not only that, it was producing them at exponentially increasing rates -- and every duplicate ID was destroying an association in our database, making it unclear what records belonged to who. It was not a good day.
Mine was for a French social networking site 4 years ago. They used to send mails everyday to say "hey look at the people who you might know". The links on the mail would automatically log the user on the website. When I sent the code live it took 2 days (and more than 50000 mails to found out that when I sent a mail to person Z about person Y the link logged in Z ON Y's account.
[1] I knew it was a bad idea to automatically log on the target of the mail. But it was the policy. And it still the case as far as I know. And if you forgot your password don't worry It's stored in plaintext...
Re: Cause of today's Github outage
#69I am a software developer, so I know "shit happens", but having the same configuration for database as testing environment, (same superuser name and password), which is not isolated from test environment, is pretty criminal even for a first time mistakeIMHO, especially for a product like github whom business, small and big trust with there business critical piece ("repository"). If I were running some critical code,…
Can you share more details about how you usually work ?
Re: Cause of today's Github outage
#70Anyone know what database system they use?
MySQL. Not that using anything else would have prevented this or would help restore the table faster...
> MySQL. Not that using anything else would have prevented this or would help restore the table faster...
The slowness of the restore sounds like they have to insert a text dump back into the database, rather than simply copy over the binary files. Even copying over 1 TB from a single hard drive or between EC2 instances would only take a few hours, not days.
And they weren't using binlogs.
And they didn't separate out their test from production environment, which is understandable when you're in demo mode but not after a couple years and half a million users.