Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cause of today's Github outage

github.com

Post reply on HN
Print view View original

91–100 of 119 posts

Re: Cause of today's Github outage

#91

Post by bigiain »

trezor wrote: post #66
blutonium wrote: post #7

Earlier quoted context omitted.

MySQL. Not that using anything else would have prevented this or would help restore the table faster...

Incorrect. The article states that they lost data between backup time and whenever the incident happened. Any reasonable quality transactional database system has a transaction log for a reason . If you were using commercial DB system like Microsoft SQL Server or Oracle (even a decade ago), this would not be an issue. No data loss. Businesses should care about their data, and I guess this is why the commercial databa…

FWIW, even MySQL let's you store "binlogs", which allow you to "rerun" all commands which changed the data since your last backup - if you've configured it to.

Which makes me think, last time I had anything to do with adminning MySQL, binlogs were required for replication, I'm guessing this means github aren't replicating that database anywhere either...

Re: Cause of today's Github outage

#92

Post by pilif »

trezor wrote: post #66
blutonium wrote: post #7

Earlier quoted context omitted.

MySQL. Not that using anything else would have prevented this or would help restore the table faster...

Incorrect. The article states that they lost data between backup time and whenever the incident happened. Any reasonable quality transactional database system has a transaction log for a reason . If you were using commercial DB system like Microsoft SQL Server or Oracle (even a decade ago), this would not be an issue. No data loss. Businesses should care about their data, and I guess this is why the commercial databa…

There are many reasons for using Postgres over mysql, but having a backupable bin-log isn't one of them. MySQL does this too.

This is an oversight in github's MySQL configuration (and their post states that they want to fix it)

Re: Cause of today's Github outage

#93

Post by viraptor »

michael_dorfman wrote: post #74
peterwwillis wrote: post #44

If someone ever produces a good book of best practices for sysadm/syseng, please provide examples like these of why it's important to follow these best practices. Yes, we've all made silly mistakes. But if you're in that design meeting and somebody asks, should we do ABC in case of XYZ, try not to think about how complicated or time consuming it might be to do ABC. Think about the worst case. If not doing it could at…

While a book of screw-ups might be amusing, I think it might be more instructive to look at the "old school" ways that screw-ups like this were avoided. I worked, back in the dark ages, for a health insurer that had two parallel environments-- one for testing/development, and one for production. None of the developers were allowed anywhere near the production environment. There was one full time employee, a former de…

It depends on the software a bit, but I got a taste of "programmers not allowed on production" environment (in a very small scale). The problem I run into is that you have no idea what's going on on the production environment. Some characteristics are maybe reproducible on the dev site, but actual users will always do something differently. Sometimes not being able to poke the live system in a specific way while it's live, will cost you weeks of guessing in the dev environment.

Then again, there is no perfect solution for this, is there? If you tried installing new version with more debugging points, you'd be deploying something unstable over something previously unstable and trying to push more data out which might be problematic in itself. I'm not even going into "rollback to stable didn't work" scenarios :(

Re: Cause of today's Github outage

#94

Post by risotto »

chesser wrote: post #86
scorpion032 wrote: post #71

Earlier quoted context omitted.

I think the fact that they have been able to recover all critical data soon and will even update the event table over the next few days clearly displays their competence. Github isn't facebook and not seeing the last week's "activities" timeline for a few days isn't really that much of a problem.

> I think the fact that they have been able to recover all critical data soon and will even update the event table over the next few days clearly displays their competence. Taking a few days to restore a table indicates they only had a text dump of it. The details of how they screwed up indicate they were new to the idea of data robustness, despite it being the core of their business. Being a competent home user and…

Git itself handles robustness for what counts -- the source code and revision history. This outage did not affect repos as far as I have heard.

Backing up and restoring mysql is an orthogonal problem.

The fact that github can operate at all in a degraded mode very much indicates their competence. Many architectures can not operate without the entire db.

Plus I bet they'll shore up their db operations next. Github is very much a young company that will take a lick or two in their way to greatness. All the startups here will.

Re: Cause of today's Github outage

#95

Post by risotto »

binaryfinery wrote: post #13
jlangenauer wrote: post #6

I think it's a measure of the goodwill in the community for Github (and perhaps, the fact that a lot of us have done something similar in the past) that they won't cop much flak at all for this.

I don't know. I use github, but my paid, private repos are elsewhere. The fact that someone, anyone, can run against the production system and nuke it raises some basic questions about password storage. I don't run a site anything like github, but my production and test databases have different passwords and none of them are stored in a way that the test environment could get access to the live db, nor could the test…

I wonder if this is a rails thing. It's an out of the box pattern to put production, development and testing credentials into one database configuration file.

Also from what I know Github is a flat org where all the devs have ability to do work in production. My company is like this too, and while it sometimes leads to scary mistakes, it also leads to massive productivity over having to go through a release engineering team.

Re: Cause of today's Github outage

#96

Post by alextgordon »

chesser wrote: post #75
alextgordon wrote: post #24

Earlier quoted context omitted.

If github spent their time so that their site raised absolutely no basic questions, then they'd still be in beta by now.

To me, that sounds like encouraging a race to the bottom. When you're in the business of storing other people's data, things like transactions, binary backups, QUICK RESTORES, and separating dev from production shouldn't be afterthoughts. They are core attributes. I would understand this for a demo, but not after a couple of years and half a million users. They have paying customers. "Reliable code hosting We spend a…

If we're going to use hindsight, we might as well look at the cause and the effect.

Yes, they missed out some pretty obvious things. And what happened? A few hours of downtime because their restore was slow, plus a tiny bit of inconsequential data loss. Hardly a catastrophe.

The fact is that every site has some sort of problems. Many of them will be completely obvious like this one. And while github could have gone through and attempted to fix them all, I much prefer they spend their time doing things that have more than a few hours of impact on my life.

No doubt they will fix the issues involved today.

Re: Cause of today's Github outage

#97

Post by risotto »

chesser wrote: post #75
alextgordon wrote: post #24

Earlier quoted context omitted.

If github spent their time so that their site raised absolutely no basic questions, then they'd still be in beta by now.

To me, that sounds like encouraging a race to the bottom. When you're in the business of storing other people's data, things like transactions, binary backups, QUICK RESTORES, and separating dev from production shouldn't be afterthoughts. They are core attributes. I would understand this for a demo, but not after a couple of years and half a million users. They have paying customers. "Reliable code hosting We spend a…

The half a million users validates their techniques, no matter how much armchair quarterbacking.

The code repos were never in danger, and they've been killing it in the market because they are racing to add awesome extra features, not racin to the bottom.

I don't disagree that they can do more in their db operations, and that it's fine for us paying customers to demand more, but the reality of startups is that it very much is a race and features, for availability or recovery or other, are viciously prioritized and many things don't happen until something breaks.

If you don't thing Github cuts the mustard, svn on Google Code probably won't have problems like this...

Disclaimer: I'm personal friends with much of the Github crew.

Re: Cause of today's Github outage

#98

Post by chesser »

risotto wrote: post #94
chesser wrote: post #86

Earlier quoted context omitted.

> I think the fact that they have been able to recover all critical data soon and will even update the event table over the next few days clearly displays their competence. Taking a few days to restore a table indicates they only had a text dump of it. The details of how they screwed up indicate they were new to the idea of data robustness, despite it being the core of their business. Being a competent home user and…

Git itself handles robustness for what counts -- the source code and revision history. This outage did not affect repos as far as I have heard. Backing up and restoring mysql is an orthogonal problem. The fact that github can operate at all in a degraded mode very much indicates their competence. Many architectures can not operate without the entire db. Plus I bet they'll shore up their db operations next. Github is…

> This outage did not affect repos as far as I have heard.

Well, I'm just going by TFA, which stated:

"Newly created users and repositories are being restored, but pull request state changes and similar might be gone."

Repos were affected.

> The fact that github can operate at all in a degraded mode very much indicates their competence. Many architectures can not operate without the entire db.

This is a curious statement to make after you started off by recognizing that git handles itself. The only parts of the repositories that broke were the parts they tried to glue to their database.

It's nice of you to give them a pass. I'd say DBAs are pretty horrified. I suppose it's kind of like how the average person can watch CSI where they say, "I'll create a GUI interface in Visual Basic, see if I can track an IP address" and think it's just fine.

Re: Cause of today's Github outage

#99

Post by risotto »

aaronblohowiak wrote: post #57
jrockway wrote: post #47

Earlier quoted context omitted.

So basically, you don't make mistakes, only other people do. Is it cold up there on that high horse?

It is trivial to put up barriers to mistakes.

After you make them, sure. But trying to predict mistakes before they happen can be difficult to impossible.

Re: Cause of today's Github outage

#100

Post by chesser »

risotto wrote: post #97
chesser wrote: post #75

Earlier quoted context omitted.

To me, that sounds like encouraging a race to the bottom. When you're in the business of storing other people's data, things like transactions, binary backups, QUICK RESTORES, and separating dev from production shouldn't be afterthoughts. They are core attributes. I would understand this for a demo, but not after a couple of years and half a million users. They have paying customers. "Reliable code hosting We spend a…

The half a million users validates their techniques, no matter how much armchair quarterbacking. The code repos were never in danger, and they've been killing it in the market because they are racing to add awesome extra features, not racin to the bottom. I don't disagree that they can do more in their db operations, and that it's fine for us paying customers to demand more, but the reality of startups is that it ver…

> The half a million users validates their techniques,

Let me introduce you to my friend, GeoCities.

Funny thing about free hosting.

> no matter how much armchair quarterbacking.

Transactions aren't armchair quarterbacking. Binary backups aren't armchair quarterbacking. Separating development from production isn't armchair quarterbacking. Kindly, you have no idea what you're talking about.

You can't give them credit for the repositories being mostly intact, when the ONLY parts that broke were the parts they mucked with to tie them into their database.

> but the reality of startups is that it very much is a race and features, for availability or recovery

And those are exactly the places where they screwed up.

Post reply on HN