I don't think it's just GitHub. I noticed everything seemed slow but thought it was just my ISP. Turns out from looking at downdetector.com there seems to be a large spike in reported problems across many sites and providers, all occurring at the same time.
GitHub is degraded/down
71–80 of 171 posts
Re: GitHub is degraded/down
#72Earlier quoted context omitted.
Here in Minnesota, several coworkers and I are having trouble with our ISP. And a website we host is (apparently) being DDoSed. And now GitHub's down, and you're reporting some Azure issue. Is something going on...?
No, I don't think so (and I'm also in Minnesota). I'm going to guess that the increased load is just pushing services over the edge. Edit: Also, interestingly enough, I am now reliably hitting Cloudflare's ORD (Chicago) datacenter instead of MSP. If you visit https://snazz.xyz/cdn-cgi/trace (or any other Cloudflare-backed website), what comes after the COLO= for you?
Re: GitHub is degraded/down
#73Earlier quoted context omitted.
If you view the historical uptime, it does seem like there have been more incidents in the past three months, but otherwise the waters look calm (as reported at least): https://www.githubstatus.com/uptime?page=1
Yes, and so this is actually a thing that bugs me, because I use Github every day. Over the past several months, there have been numerous days in which I've had problems (`You can not comment at this time`, 500s, etc.) and no corresponding status report. It seems like the historical uptime page paints a far rosier picture than I am actually experiencing.
If you want to see more the state of github "extras", you'd need to select "github actions" or "webhooks", which have a fair amount of downtime (about once a week or so, which seems about right).
Interesting how the most stable component of the company is the open source one of course ^^
Re: GitHub is degraded/down
#74I also I tried setting up an EC2 instance in North Cali and just pinging it from Kansas City via Google fiber experienced 37% packet loss.
Re: GitHub is degraded/down
#75Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?
If you're looking for somewhere else, SourceHut has had no unplanned outages in 2020, despite being kept online by an army of one. The software and infrastructure is just more resilient and better maintained. Our ops guide is available here: https://man.sr.ht/ops/ It's also the highest performance software forge by objective measures: https://forgeperf.org/ Full disclosure: I am the founder of SourceHut.
Re: GitHub is degraded/down
#76Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?
It might also be covid related. People are working from home, people responsible for system upkeep might not be immediately responsive, more demand on the servers for whatever reason, etc.
Re: GitHub is degraded/down
#77Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?
If you're looking for somewhere else, SourceHut has had no unplanned outages in 2020, despite being kept online by an army of one. The software and infrastructure is just more resilient and better maintained. Our ops guide is available here: https://man.sr.ht/ops/ It's also the highest performance software forge by objective measures: https://forgeperf.org/ Full disclosure: I am the founder of SourceHut.
Re: GitHub is degraded/down
#78Earlier quoted context omitted.
The "we deploy production directly from Github" pathology is the one I'm talking about. Not being able to "collaborate" for a brief outage is fine. Not being able to deploy code isn't.
You gotta CI/CD from somewhere . Whatever that somewhere is, it can go down. You can, of course, override your CI/CD and do a manual deploy. It's not a matter of "can." It's a matter of not fully understanding all the checks the CI/CD system does to the code, and all the build steps for prod builds, and therefore not having the confidence to deploy to prod without CI/CD holding your hand. (Which I don't at-all blame…
Re: GitHub is degraded/down
#79Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?
If you're looking for somewhere else, SourceHut has had no unplanned outages in 2020, despite being kept online by an army of one. The software and infrastructure is just more resilient and better maintained. Our ops guide is available here: https://man.sr.ht/ops/ It's also the highest performance software forge by objective measures: https://forgeperf.org/ Full disclosure: I am the founder of SourceHut.
When most people talk about “modern web” or modern anything in software they think it means “using all the latest tools”.
That often means things like ES6 and Webpack, which have nice surfaces, but which create nightmares under the hood.
That’s the opposite of what modern architecture was. It was about embracing the constraints of materials. Given the properties of concrete, what is the limit of what you can do with it. Go there, and no further. And don’t cover it up, just finish the dang slab and get on with the rest of the house.
ES6 means transpiling, which means webpack, which means a massive machine of hidden complexity, which if you’re lucky exposes a nice smooth surface where everything is arrow functions and named exports. And if you’re unlucky is a flimsy piece of cardboard over the nightmare underneath.
You (SourceHut) seem to be building a UI that actually takes note of how the browser is. And you are trying to push the big numbers... how reliable your service can be, how many endpoints can one person maintain, while letting the materials of the web (forms, urls) dictate the details.
That’s true modernism.
So, bravo. I’m glad to see you out in the world. It takes courage to step outside of the norm and I’m rooting for you.
Re: GitHub is degraded/down
#80Earlier quoted context omitted.
If you're looking for somewhere else, SourceHut has had no unplanned outages in 2020, despite being kept online by an army of one. The software and infrastructure is just more resilient and better maintained. Our ops guide is available here: https://man.sr.ht/ops/ It's also the highest performance software forge by objective measures: https://forgeperf.org/ Full disclosure: I am the founder of SourceHut.
I'm sure you've done a great job building up your infrastructure, but if you have the level of traffic Github has, what would your uptime be?
And I believe sr.ht would beat out GitHub at their scale anyway. The services are an order of magnitude more lightweight. And the design is more fault tolerant: we use a distributed architecture, so one part of the system can go down without affecting anything else - as if GitHub's issues could go down without anything else being affected. And many of our tools are based on email, a global fault-tolerant system, which would allow you to get your work done more or less unaffected even if SourceHut was experiencing a total outage. We'd automatically get caught back up with what you were up to in the meanwhile once we're online, too.
I've spoken to GitHub engineers about some of the internal architectural design of GitHub, too, I'm confident that SourceHut's technical design beats out GitHub's in terms of scalability. And, despite already winning by a good margin, I'm still spending a lot of effort to push the envelope further on performance and scalability.