Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Has GitHub been down more since its acquisition by Microsoft?

nimbleindustries.io

Post reply on HN
Print view View original

51–60 of 149 posts

Re: Has GitHub been down more since its acquisition by Microsoft?

#51

Post by scaryclam »

m12k wrote: post #48
Softcadbury wrote: post #32

Earlier quoted context omitted.

I'm sure these down times are more about infrastructure problems than just database updates and I'm also sure that this infrastructure is a little more complexe than your site.

I'm sure their deployment process is way more complex than mine. I'm also sure that unlike me, they have engineers dedicated to managing the complexity. GitHub is a company that has blogged about how easy they have made deployment and how they deploy many times a day. I don't believe there are any compelling reasons why that ease of deployment shouldn't also extend to re-deploying previous versions of their service,…

I think you're missing the point a bit. Not all changes are simple code deploys. They take time to diagnose, mitigate and fix. This is especially true when you have a lot of services that talk to each other, working on infrastructure that can support so many services and users.

Re: Has GitHub been down more since its acquisition by Microsoft?

#52

Post by m12k »

rbanffy wrote: post #31
m12k wrote: post #25

What I don't get is how they can be down for hours on end. If I make a deploy that turns out to be buggy, I'll roll my site back to a stable previous version within minutes. Sure, that's not always possible if I've made incompatible database schema changes, but in my experience those are very, very rare (i.e. I almost always add db columns, and only rarely delete or rename columns - and when I do, I do so after those…

> If I make a deploy that turns out to be buggy Unless your deploy reconfigures some networking component that makes a large part of your network inaccessible. Then you need to fix the network issue before you can rollback to a previous version. That may require someone driving up to a datacenter and logging into a racked server. And then you may need to restore data if the network misconfiguration caused data to be…

> People often deploy new features deactivated, then, when the whole fleet is updated, activate features to different groups and monitor for breaking behavior change.

Even better - then they shouldn't even need to make another deploy, just flip the feature flag back off. And if you need to make network changes, then test those out behind a load balancer in parallel to the existing topology, so you can start routing more traffic to the new setup, but can stop doing so if any problems arise. I'm not saying any of this is trivial, but the point is, best practices exist to start deploying pretty much any kind of change in a way that can be undone in minutes or even seconds. When you have access to the resources and talent that Github has, then there's zero acceptable reasons why your site would ever be down or degraded for hours on end - zero.

Re: Has GitHub been down more since its acquisition by Microsoft?

#54

Post by jillesvangurp »

esperent wrote: post #23
scribu wrote: post #14

It's easy to see from the uptime history [1] that there have been many more incidents in April-June than in January-April. Don't know if that has anything to do with the Microsoft acquisition, but it is concerning. [1] https://www.githubstatus.com/uptime?page=1

They're still averaging 99.81% up-time over the last three months. This means they have been down for a total of 4 hours out of 90 days. I don't think it's all that concerning.

The gold standard here is 5 9s, not 2.

They have paying customers that are being inconvenienced by this. We lost time over this multiple times in the last few weeks.

So, not good. IMHO they are having some major issues with their release process that they need to address. Standards have slipped there; they used to be better at this.

Re: Has GitHub been down more since its acquisition by Microsoft?

#55

Post by waheoo »

danpalmer wrote: post #9
battery423 wrote: post #2

It could also mean that they are more active and creating more chaos due to more changes. shameless and useless advertising.

On my team our downtime is very closely correlated with how much we're working and changing the product. I'd be surprised if GitHub is much different.

Or in other words, facebook is more stable on the holidays.

Re: Has GitHub been down more since its acquisition by Microsoft?

#56

Post by ToolsDevler »

waheoo wrote: post #53
bryanrasmussen wrote: post #47

Earlier quoted context omitted.

I think it's pretty fair to say we've all been pretty down since GitHub's acquisition by Microsoft.

I'm on sourcehut. Whats a microsoft?

It sounds like a kind of toilet paper

Re: Has GitHub been down more since its acquisition by Microsoft?

#57

Post by rattray »

rb12345 wrote: post #44
rattray wrote: post #19

Earlier quoted context omitted.

Wow, that's stark. I count 6 in April-June (2 major outages) and only 1 in January-March (0 major outages. I wonder if COVID has affected this somehow. Anecdotally I've heard of at least one other ~peer company with a large rise in incidents/outages since April. Strange since both companies previously had a strong culture of remote work (maybe 1/4 to 1/3 of eng was remote) going into the pandemic, so I'd be quite sur…

If GitHub's been migrating to Azure infrastructure on the backend, it's possible that everyone else moving to all-WfH helped to cause it. Teams use massively increased during lockdown for obvious reasons and that's definitely Azure-based. If Teams managed to overload Azure capacity, I could see that having knock-on effects on GitHub.

The other company does not use Azure.

Re: Has GitHub been down more since its acquisition by Microsoft?

#58

Post by risyachka »

colinbartlett wrote: post #29
robertlagrant wrote: post #18

Has nimbleindustries.io been down more since Github's acquisition by Microsoft?

Sorry... we probably should have put a CDN in front of our humble Linode box earlier. Here's a cache link: https://web.archive.org/web/20200629103627/https://nimbleind... Edit: Cloudflare seems to be working now. Wow, that was really easy to setup.

Its hard for me to trust your products if you didn't even have a Cloudflare in front.

EDIT: I don't imply the product is bad, sorry

Re: Has GitHub been down more since its acquisition by Microsoft?

#60

Post by robertlagrant »

colinbartlett wrote: post #29
robertlagrant wrote: post #18

Has nimbleindustries.io been down more since Github's acquisition by Microsoft?

Sorry... we probably should have put a CDN in front of our humble Linode box earlier. Here's a cache link: https://web.archive.org/web/20200629103627/https://nimbleind... Edit: Cloudflare seems to be working now. Wow, that was really easy to setup.

:-) Cloudflare is amazing, yes. Everyone should use it by default, I say.
Post reply on HN