January 28th Incident Report • Hacker News Board

Re: January 28th Incident Report

#31

Post by Cthulhu_ » Wed, Feb 03, 2016, 10:22 PM UTC

viraptor wrote: post #18

> ... Updating our tooling to automatically open issues for the team when new firmware updates are available will force us to review the changelogs against our environment. That's an awesome idea. I wish all companies published the firmware releases in simple rss feeds, so everyone could easily integrate them with their trackers. (If someone's bored, that may be a nice service actually ;) )

I've played with the idea of some automated software update reporting site ages ago - it'd read rss feeds and scrape websites for the required info. It'd probably need adjustments for each hardware manufacturer / product though, and regular updating. But that could possibly be part of an open source project, give the firmware maintainers the opportunity to help out too.

Re: January 28th Incident Report

#32

Post by [deleted] » Wed, Feb 03, 2016, 10:24 PM UTC

pedalpete wrote: post #13

Does Github run anything like Netflix Simbian Army against it's services? As a company by engineers for engineers with the scale that github has reached, I'm a bit surprised they are lacking a bit more redundancy. Though they may not need the uptime of netflix, an outage of more than a few minutes on github could affect businesses that rely on the service.

[deleted]

Re: January 28th Incident Report

#33

Post by merqurio » Wed, Feb 03, 2016, 10:25 PM UTC

davidcelis wrote: post #23

merqurio wrote: post #21

Earlier quoted context omitted.

Of course it was. I don't know if everybody goes silent after a few days, It's the first outage I'm aware of, but some people at university made see the hypocrisy of using GitHub for open source projects and I feel that if there is a community strong enough to make some impact on GitHub that could be hackernews. Maybe I'm wrong.

If you look back through the years and find a few other stories of "GitHub is down", you'll see that this conversation happens every time. Some people tread into the HackerNews thread and say "More people should be using self-hosted GitLab instances" or "if GitHub would just open source their code, we wouldn't need to be so dependent." But then the conversation stops within days because, the fact is, hosting your own…

Sure ! And I think it's a great business and I'm a happy customer too. My comment wasn't against the enterprise at all!

I just wanted to point that now that you guys are well stablished and have huge impact in the open source community, adopting a more open approach with your end users can be very beneficial for both, GitHub and the user base. I'm sure that lot of people would contribute to your codebase and thing like this: https://github.com/dear-github/dear-github would be less frequent and notorious.

But if the enterprise edition is really the source of income, open sourcing it doesn't have any sense. I agree on that. Maybe, another way to be more open to contributions from the community ? I dunno

Re: January 28th Incident Report

#34

Post by uiri » Wed, Feb 03, 2016, 10:26 PM UTC

davidcelis wrote: post #23

merqurio wrote: post #21

Earlier quoted context omitted.

Of course it was. I don't know if everybody goes silent after a few days, It's the first outage I'm aware of, but some people at university made see the hypocrisy of using GitHub for open source projects and I feel that if there is a community strong enough to make some impact on GitHub that could be hackernews. Maybe I'm wrong.

If you look back through the years and find a few other stories of "GitHub is down", you'll see that this conversation happens every time. Some people tread into the HackerNews thread and say "More people should be using self-hosted GitLab instances" or "if GitHub would just open source their code, we wouldn't need to be so dependent." But then the conversation stops within days because, the fact is, hosting your own…

hosting your own git servers and getting people to actually use them is a huge pain in the ass. More simply put: people just like using GitHub.

Given that this is the case, I fell like GitHub is entrenched enough that they could open source their codebase and not lose any customers. People are paying them for the convenience of someone else hosting their git repository.

Re: January 28th Incident Report

#35

Post by lhc- » Wed, Feb 03, 2016, 10:28 PM UTC

yeukhon wrote: post #24

viraptor wrote: post #14

Earlier quoted context omitted.

Isn't BB's problem basically that there are too many users? GH's outage writeup is cool, because it's a one off and it can be analysed. When BB is just overloaded for a long time and needs more power, it's not going to be very interesting. (unless I missed some specific non capacity related outages?)

Maybe. BitBucket was also an acquisition so for some time I believe there was a lack of resource provided to them and there was a huge technical debt/integration effort required. At this very time, I don't know if Atlassian actually care much about BitBucket. They are probably more concerned about delivering Stash than BitBucket, my wild guess. I was an active BB user a couple years ago, and the project I worked on w…

FYI, Stash is now Bitbucket Server, and the plan as I've heard it is to work towards feature parity between the two.

Re: January 28th Incident Report

#36

Post by sytse » Wed, Feb 03, 2016, 10:28 PM UTC

jdboyd wrote: post #26

merqurio wrote: post #5

I feel it was good incident for the Open Source community, to see how dependent we are on GitHub today. I feel sad whenever I see another large project like Python moving to GitHub, a closed-sourced company. I know, GitLab is there as an alternative, but I would love to see all the big Open Source projects putting pressure over GitHub to make them open their source code, as right they are big player in open source, l…

If github opensource all of their stuff, this still wouldn't prevent issues like this for the projects that want to use a hosted service instead of hosting it themselves, and many projects don't want to host these services them selves anymore. When I worry about dependency on GitHub, I'm thinking about not the inconvenient hours of downtime but the larger threat that they might disappear or turn evil. What I would li…

You propose GitLab syncing between on-premises installations and GitLab.com. Sync is pretty hard. What we currently plan to do is import/export https://gitlab.com/gitlab-org/gitlab-ce/issues/3050 We are thinking about distributed code reviews and issues https://gitlab.com/gitlab-org/gitlab-ce/issues/4084 and federated merge requests https://gitlab.com/gitlab-org/gitlab-ce/issues/4013

Re: January 28th Incident Report

#37

Post by totally » Wed, Feb 03, 2016, 10:28 PM UTC

> Because we have experience mitigating DDoS attacks, our response procedure is now habit and we are pleased we could act quickly and confidently without distracting other efforts to resolve the incident.

The thing that fixed the last problem doesn't always fix the current problem.

Re: January 28th Incident Report

#38

Post by totally » Wed, Feb 03, 2016, 10:36 PM UTC

imbriaco wrote: post #19

danielvf wrote: post #9

For all that work to be done in just two hours is amazing, especially with degraded internal tools, and both hardware and ops teams working simultaneously.

You're absolutely right. We should collectively be using incidents like this as an opportunity to learn, much like the GitHub team does. Our entire industry is held back by the lack of knowledge sharing when it comes to problem response and the fact that so many companies are terrified of being transparent in the face of failure. This is very well written retrospective that gives us a glimpse into the internal review…

relevant:

https://codeascraft.com/2012/05/22/blameless-postmortems/

Re: January 28th Incident Report

#39

Post by merqurio » Wed, Feb 03, 2016, 10:36 PM UTC

BinaryIdiot wrote: post #28

merqurio wrote: post #5

I feel it was good incident for the Open Source community, to see how dependent we are on GitHub today. I feel sad whenever I see another large project like Python moving to GitHub, a closed-sourced company. I know, GitLab is there as an alternative, but I would love to see all the big Open Source projects putting pressure over GitHub to make them open their source code, as right they are big player in open source, l…

Git is a distributed version control system. Github is simply a place to host a repository and some issues. There is nothing stopping anyone from pushing to another remote hub for redundancy. So you want Github to open source where they put your git repo and issues? Who cares about that? It's unimportant because regardless they're still the central endpoint to many open source projects, opened or closed source. If yo…

In my case at least is not outrage, it's just that I would love to feel that they are more responsive to their community and that I think that I'm not the only one who thinks that GitHub community would gladly help to make the product even better. I see it like a win win situation. Maybe not and it's just business.. dunno

Re: January 28th Incident Report

#40

Post by chris_wot » Wed, Feb 03, 2016, 10:36 PM UTC

onetwotree wrote: post #30

Every time I read about a massive systems failure, I think of Jurassic Park and am mildly grateful that the velociraptor padock wasn't depending on the systems operation.

I think you'll find they were.