Who would have guessed that handing control of a company to Microsoft would make it less reliable.
GitHub is degraded/down
121–130 of 171 posts
Re: GitHub is degraded/down
#122Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?
It has been since the Microsoft acquisition.
I chalk most of the early ones to moving services over to Azure.
Lately though, I don't know. Azure is running pretty close to capacity, so maybe it's part of the problem
Re: GitHub is degraded/down
#123Earlier quoted context omitted.
Here in Minnesota, several coworkers and I are having trouble with our ISP. And a website we host is (apparently) being DDoSed. And now GitHub's down, and you're reporting some Azure issue. Is something going on...?
No, I don't think so (and I'm also in Minnesota). I'm going to guess that the increased load is just pushing services over the edge. Edit: Also, interestingly enough, I am now reliably hitting Cloudflare's ORD (Chicago) datacenter instead of MSP. If you visit https://snazz.xyz/cdn-cgi/trace (or any other Cloudflare-backed website), what comes after the COLO= for you?
You can test Cloudflare at https://cloudflare-test.judge.sh/#snazz.xyz
Re: GitHub is degraded/down
#124Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?
1) Microsoft took over
2) M$ migrates some ADO (Azure DevOps) features to Github (e.g., Github Actions)
3) If Github was not on Azure before M$ bought it (very likely, but needs citation) they will probably migrate to Azure at some point
Re: GitHub is degraded/down
#125Earlier quoted context omitted.
It's almost as if keeping a complex service like Github online and available to millions of users is hard.
Building skyscrapers is hard. Does that make it OK for them to fall down regularly?
GitHub hasn't collapsed killing thousands or needing to be completely rebuilt though, so that analogy doesn't work. This is more like there's a flood in the lobby so maintenance has closed the front door for a bit.
Re: GitHub is degraded/down
#126Earlier quoted context omitted.
Where do you store all that data?
Should only be 2-3 PB. Storing it all on https://tardigrade.io
Only 3 petabytes? At $0.01/GB/Month, it seems that works out to about $360,000/year in storage costs alone. Cheers, but you're paying for this just as a public service?
Re: GitHub is degraded/down
#127Earlier quoted context omitted.
I'm sure you've done a great job building up your infrastructure, but if you have the level of traffic Github has, what would your uptime be?
Who's to say? It's not GitHub scale, and even if everyone in this thread moved to SourceHut, it still wouldn't be GitHub scale, but it would be serving your needs just fine. I feel totally comfortable recommending SourceHut over GitHub as a service which can be expected to have better uptime and performance, because it is a fact - even if we operate at different scales. And I believe sr.ht would beat out GitHub at th…
> Who's to say?
And then you go on to say it. I'm glad that SourceHut exists, and I like many of its principles, and it's probably better designed too, but walking into a thread where someone is having an outage and then claiming that you'd do much better is in poor taste no matter how good you are or how many of your services work offline.
Re: GitHub is degraded/down
#128CircleCI's having a lot of trouble, too. Intermittent 500s, and even the images on the 500 page don't load. Seems like something bigger might be going on.
What usually happens with CircleCI after a GitHub outage is that they're hit with a flood of GitHub webhooks when GitHub comes back online. Then Circle starts to slow down under the load of all of the new jobs that have been queued. One usually causes the other, so we shouldn't infer anything larger from that datapoint.
In this case, CircleCI and Github were down at the same time, and seemed to come out of it around the same time too. I don't know what to infer from that either.
Re: GitHub is degraded/down
#129Earlier quoted context omitted.
Building skyscrapers is hard. Does that make it OK for them to fall down regularly?
GitHub hasn't collapsed killing thousands or needing to be completely rebuilt though, so that analogy doesn't work. This is more like there's a flood in the lobby so maintenance has closed the front door for a bit.
Planes sometimes crash without killing people or needing to be completely rebuilt; that doesn't mean that this is clearly undesirable.
Re: GitHub is degraded/down
#130Earlier quoted context omitted.
I'm not sure the irony is lost on you that; people generally prefer services like github (and; similarly EC2) precisely _because_ it's not on-prem and that if it's down there are hundreds of talented engineers working to resolve those issues. Unfortunately anecdotal experience here is spotty. Services I run for my team are several 9's higher in terms of actual availability (note: I did not say "uptime"); and contrari…
Does the GH Enterprise option not provide some degree of support with initial setup and configuration? Is the enterprise support more or less responsive than the public channels? I would expect those same engineers are also responsible for maintaining the enterprise offering. Also, does isolation of a private GH instance from the public instances not provide some degree of added reliability considering the potential…
My typical response from GitHub Enterprise support has been fantastic over the years. I'll also note that after GitHub Actions/Packages came out on Github.com the lead time for support response did increase substantially for non critical tickets due the increased support burden for those new services. My most recent tickets have been answered promptly so they must have figured out the staffing issues.
You should be well aware that actual architecture of GitHub Enterprise isn't truly highly available (1 active and 1 or more standby instances) unless you are a huge customer to support clustering mode. Which means you likely need to take downtime to implement upgrades since they typically require rebooting the VM.