Live data from Hacker News

Has GitHub been down more since its acquisition by Microsoft?

nimbleindustries.io

121–130 of 149 posts

Re: Has GitHub been down more since its acquisition by Microsoft?

#121
post #97

Earlier quoted context omitted.

As I mentioned, my comment is meant as an exercise. If we were to take the numbers more seriously, due diligence is necessary. That said, if we assume that one incident does not affect the other, then the Poisson nature falls out as a natural consequence of that independence and the assumption of a constant rate (our Null hypothesis). As long as the incidents are spaced out enough, that the possibility of one inciden…

But why even start with an assumption that is so likely to be wrong? We know that incidents are frequently correlated. We know that scale and complexity add fragility. We know that GitHub has gotten bigger and more complex. The chance of the probability distribution holding constant over 5 years of major growth is basically zero.

We are talking about different things. One is about attributing causes to an increase in failure rate, the other is about verifying whether there is any material increase in the rate at all. My comment addresses the latter as a back of the envelope calculation.

Strictly speaking, when looked at through a fine toothed comb, yes the assumptions are very likely wrong. All models are wrong [0], but some of them are useful.

The question is can we get some useful conclusions from such a simple model. In my experience I have been surprised by how often low failure rates are captured well by Poisson processes. Yes the assumptions could be wrong, but are they very likely to lead to wrong conclusions ? Empirical experience and math says otherwise.

There are sound reasons for why this happens. If you are interested, you can pick that up from Feller. These [1] [2] links might also help.

Given the data that we have, its a plenty good first cut, but that's what it is -- a first cut. With more data one can do a more refined analysis.

[0] https://en.wikipedia.org/wiki/All_models_are_wrong

[1] https://en.wikipedia.org/wiki/Poisson_point_process#Approxim...

[2] https://en.wikipedia.org/wiki/Poisson_point_process#Converge...

Re: Has GitHub been down more since its acquisition by Microsoft?

#122
post #14

It's easy to see from the uptime history [1] that there have been many more incidents in April-June than in January-April. Don't know if that has anything to do with the Microsoft acquisition, but it is concerning. [1] https://www.githubstatus.com/uptime?page=1

I remember there were a few incident with AWS taking down the whole internet, Verizon BGP incident, and a few others that is completely not the fault of Github. I wonder if those were included or affected.

And generally speaking Github has been 100x more active post Microsoft acquisition. So I am not surprised at the downtime.

And I will gladly trade another few hours if not more downtime if they could just rollback the side panel design.

Re: Has GitHub been down more since its acquisition by Microsoft?

#123
post #99

Earlier quoted context omitted.

exactly. Before M$ there wasn't much product movement at Github :/

I think they got out of the rut before Microsoft's involvement, but I suspect MS has allowed them to accelerate a lot of it with more investment and a more defined direction. I've heard that for a number of years the Product org at GitHub essentially considered the product "finished", and that it did not need more features. Things like the rise of GitLab and the "Dear GitHub" letter, plus I believe a change of leader…

Thing is though, they were right. All this nonsense features are literally messing up (a big part of) their core business: uptime.

Re: Has GitHub been down more since its acquisition by Microsoft?

#124

Earlier quoted context omitted.

The numbers on Github Enterprise might be different. They probably roll out changes to the free/$4 per month Github first.

Isn't enterprise the self-hosted solution?

I believe Enterprise has a self-hosted option, and you can have Enterprise without the self-hosting.

Re: Has GitHub been down more since its acquisition by Microsoft?

#125
post #52
post #31

Earlier quoted context omitted.

> If I make a deploy that turns out to be buggy Unless your deploy reconfigures some networking component that makes a large part of your network inaccessible. Then you need to fix the network issue before you can rollback to a previous version. That may require someone driving up to a datacenter and logging into a racked server. And then you may need to restore data if the network misconfiguration caused data to be…

> People often deploy new features deactivated, then, when the whole fleet is updated, activate features to different groups and monitor for breaking behavior change. Even better - then they shouldn't even need to make another deploy, just flip the feature flag back off. And if you need to make network changes, then test those out behind a load balancer in parallel to the existing topology, so you can start routing m…

There are countless ways for infrastructure to break on its own, without being tied to a specific deploy or feature flag. A few common examples in the db tier alone:

Have you ever encountered a write rate that exceeds your db replica's ability to keep up with async replication? There's nothing to "roll back" in this case, and it takes time to determine whether the increase in write rate is from legitimate usage growth vs some recent feature (possibly deployed hours/days ago) writing more than expected during peak periods vs DDOS/bot activity.

Have you worked on multi-region infrastructure, where traffic is actively served from multiple geographic regions, with fully automated failover during regional outages? This is impossible to fully automate every possible situation -- even Google and Facebook have outages sometimes! Even just as a first step, it's hard to figure out conclusively which situations should be automated vs which ones need to alert humans.

Have you ever implemented read-after-write consistency for multi-region infrastructure, where multiple async DB replicas, caches, and backend file stores are not automatically in sync, but need to appear in sync to users making writes from non-master regions? The network latency between regions is sufficient to make this a complicated problem even when things are stable, let alone when there's other sources of replication lag to consider. There's no "out of the box" solution for this; every company needs to handle it in a way specific to their infrastructure and product.

Have you ever implemented a realistic dev/test environment for a massive infrastructure involving dozens to hundreds of services, and many different data stores, some of which are sharded? Again, no "out of the box" solution exists. You need to do something custom, and there will be plenty of cases where it doesn't accurately mirror production.

Or for a non-technical one: have you ever worked for a medium-to-large size company whose exit was via acquisition, rather than IPO? In my experience this always results in a major increase in attrition of the acquired company's top engineers. With an IPO, early folks are more incentivized to stay on; there's a better feeling of ownership, and the efforts of good talent can directly impact the stock price. But when it's an acquisition by some corporate behemoth, the opposite dynamic is at play: there's very little that the acquired company can do to impact the parent company's stock price, leading to a feeling of helplessness. Couple that with different policies and values mindset (say, a contract with a government agency that puts children in cages) and you can guess what happens.

Re: Has GitHub been down more since its acquisition by Microsoft?

#126
post #80

Taking the numbers at their face value this is a good exercise in some text book statistical test of hypothesis. Incidents: Before 89. After 126. What is the chance of this happening if the 'rate' of occurrence has not changed ? Assuming an unknown but constant Poisson rate, we get the probability of observing what has been observed to be 0.00225. A fortuitous thing about this test is that one does not need to know w…

This was exactly my thought.

More or less downtime (as reported by a status page) is probably affected more by changes to policies re: how / when incidents are posted publicly between "Github" and "Microsoft Github".

Subjectively speaking (using Github daily) I haven't noticed a difference. In general, Github has never been extremely reliable even pre-Microsoft.

Re: Has GitHub been down more since its acquisition by Microsoft?

#127
post #103

Earlier quoted context omitted.

What the hell are you building that there has to be Cloudflare in the dependency tree at some point?

Any javascript CDN?

I'm sure there are more Node.js modules for that than you can count.

Re: Has GitHub been down more since its acquisition by Microsoft?

#129

I mean makes sense. They are probably migrating their infrastructure over to Microsoft’s and such. Disclaimer: I don’t work at MS so no clue, but have been part of acquisitions

I'm sure MS is super uncomfortable with GitHub's AWS bill. I've just assumed they're fast-tracking a migration to Azure

Re: Has GitHub been down more since its acquisition by Microsoft?

#130

Earlier quoted context omitted.

Sorry... we probably should have put a CDN in front of our humble Linode box earlier. Here's a cache link: https://web.archive.org/web/20200629103627/https://nimbleind... Edit: Cloudflare seems to be working now. Wow, that was really easy to setup.

How many req/s before you fell over?

I often wonder what kind of stack websites that fall over under HN traffic have.

From my perspective HN doesn't actually bring that much traffic, so I find it curious which software is so fragile to fall over when getting HN front page traffic.

Post reply on HN