Live data from Hacker News

GitHub is degraded/down

githubstatus.com

71–80 of 171 posts

Re: GitHub is degraded/down

#71

I don't think it's just GitHub. I noticed everything seemed slow but thought it was just my ISP. Turns out from looking at downdetector.com there seems to be a large spike in reported problems across many sites and providers, all occurring at the same time.

Yes, saw similar notice from bitbucket

Re: GitHub is degraded/down

#72
post #21
post #17

Earlier quoted context omitted.

Here in Minnesota, several coworkers and I are having trouble with our ISP. And a website we host is (apparently) being DDoSed. And now GitHub's down, and you're reporting some Azure issue. Is something going on...?

No, I don't think so (and I'm also in Minnesota). I'm going to guess that the increased load is just pushing services over the edge. Edit: Also, interestingly enough, I am now reliably hitting Cloudflare's ORD (Chicago) datacenter instead of MSP. If you visit https://snazz.xyz/cdn-cgi/trace (or any other Cloudflare-backed website), what comes after the COLO= for you?

Getting rerouted to ORD instead of DEN as well. Coworkers have been complaining about slow internet all day, there’s been customer complaints about slow load times with our products as well.

Re: GitHub is degraded/down

#73
post #3

Earlier quoted context omitted.

If you view the historical uptime, it does seem like there have been more incidents in the past three months, but otherwise the waters look calm (as reported at least): https://www.githubstatus.com/uptime?page=1

Yes, and so this is actually a thing that bugs me, because I use Github every day. Over the past several months, there have been numerous days in which I've had problems (`You can not comment at this time`, 500s, etc.) and no corresponding status report. It seems like the historical uptime page paints a far rosier picture than I am actually experiencing.

If you do nothing, it lands by default on the "git operations" view, which is by far the most stable, since, well, it consists of executing the battle-tested git program.

If you want to see more the state of github "extras", you'd need to select "github actions" or "webhooks", which have a fair amount of downtime (about once a week or so, which seems about right).

Interesting how the most stable component of the company is the open source one of course ^^

Re: GitHub is degraded/down

#74
There are larger problems I think... our call center is having trouble with Five9 in San Francisco.

I also I tried setting up an EC2 instance in North Cali and just pinging it from Kansas City via Google fiber experienced 37% packet loss.

Re: GitHub is degraded/down

#75

Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?

If you're looking for somewhere else, SourceHut has had no unplanned outages in 2020, despite being kept online by an army of one. The software and infrastructure is just more resilient and better maintained. Our ops guide is available here: https://man.sr.ht/ops/ It's also the highest performance software forge by objective measures: https://forgeperf.org/ Full disclosure: I am the founder of SourceHut.

your $2/month pricing. Is it $2/person/month?

Re: GitHub is degraded/down

#76
post #8

Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?

It might also be covid related. People are working from home, people responsible for system upkeep might not be immediately responsive, more demand on the servers for whatever reason, etc.

Personnel may be partially unavailable, but - home or office - developers are doing the same job as they always did. At least from the users point of view it didn't change that much.

Re: GitHub is degraded/down

#77

Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?

If you're looking for somewhere else, SourceHut has had no unplanned outages in 2020, despite being kept online by an army of one. The software and infrastructure is just more resilient and better maintained. Our ops guide is available here: https://man.sr.ht/ops/ It's also the highest performance software forge by objective measures: https://forgeperf.org/ Full disclosure: I am the founder of SourceHut.

Yeah I'm sure there are no possible scaling issues between your service (how many people use it / repos acitve) vs github or gitlab....

Re: GitHub is degraded/down

#78
post #55

Earlier quoted context omitted.

The "we deploy production directly from Github" pathology is the one I'm talking about. Not being able to "collaborate" for a brief outage is fine. Not being able to deploy code isn't.

You gotta CI/CD from somewhere . Whatever that somewhere is, it can go down. You can, of course, override your CI/CD and do a manual deploy. It's not a matter of "can." It's a matter of not fully understanding all the checks the CI/CD system does to the code, and all the build steps for prod builds, and therefore not having the confidence to deploy to prod without CI/CD holding your hand. (Which I don't at-all blame…

Part of the point of having release infrastructure is so that every engineer doesn't have to know how to manage a deployment, even. Having worked both with and without it, I have to say I favor the former. The occasional loss of productivity due to a service provider outage, or even due to whatever the hell's going on today, is vastly outweighed by the day-to-day baseline requiring next to no time spent on the mechanics of taking code from review approval to live in prod.

Re: GitHub is degraded/down

#79

Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?

If you're looking for somewhere else, SourceHut has had no unplanned outages in 2020, despite being kept online by an army of one. The software and infrastructure is just more resilient and better maintained. Our ops guide is available here: https://man.sr.ht/ops/ It's also the highest performance software forge by objective measures: https://forgeperf.org/ Full disclosure: I am the founder of SourceHut.

Sidebar, I just want to say, you are one of the few people I’ve observed doing actual “modern” web development.

When most people talk about “modern web” or modern anything in software they think it means “using all the latest tools”.

That often means things like ES6 and Webpack, which have nice surfaces, but which create nightmares under the hood.

That’s the opposite of what modern architecture was. It was about embracing the constraints of materials. Given the properties of concrete, what is the limit of what you can do with it. Go there, and no further. And don’t cover it up, just finish the dang slab and get on with the rest of the house.

ES6 means transpiling, which means webpack, which means a massive machine of hidden complexity, which if you’re lucky exposes a nice smooth surface where everything is arrow functions and named exports. And if you’re unlucky is a flimsy piece of cardboard over the nightmare underneath.

You (SourceHut) seem to be building a UI that actually takes note of how the browser is. And you are trying to push the big numbers... how reliable your service can be, how many endpoints can one person maintain, while letting the materials of the web (forms, urls) dictate the details.

That’s true modernism.

So, bravo. I’m glad to see you out in the world. It takes courage to step outside of the norm and I’m rooting for you.

Re: GitHub is degraded/down

#80

Earlier quoted context omitted.

If you're looking for somewhere else, SourceHut has had no unplanned outages in 2020, despite being kept online by an army of one. The software and infrastructure is just more resilient and better maintained. Our ops guide is available here: https://man.sr.ht/ops/ It's also the highest performance software forge by objective measures: https://forgeperf.org/ Full disclosure: I am the founder of SourceHut.

I'm sure you've done a great job building up your infrastructure, but if you have the level of traffic Github has, what would your uptime be?

Who's to say? It's not GitHub scale, and even if everyone in this thread moved to SourceHut, it still wouldn't be GitHub scale, but it would be serving your needs just fine. I feel totally comfortable recommending SourceHut over GitHub as a service which can be expected to have better uptime and performance, because it is a fact - even if we operate at different scales.

And I believe sr.ht would beat out GitHub at their scale anyway. The services are an order of magnitude more lightweight. And the design is more fault tolerant: we use a distributed architecture, so one part of the system can go down without affecting anything else - as if GitHub's issues could go down without anything else being affected. And many of our tools are based on email, a global fault-tolerant system, which would allow you to get your work done more or less unaffected even if SourceHut was experiencing a total outage. We'd automatically get caught back up with what you were up to in the meanwhile once we're online, too.

I've spoken to GitHub engineers about some of the internal architectural design of GitHub, too, I'm confident that SourceHut's technical design beats out GitHub's in terms of scalability. And, despite already winning by a good margin, I'm still spending a lot of effort to push the envelope further on performance and scalability.

Post reply on HN