Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub is degraded/down

githubstatus.com

Post reply on HN
Print view View original

101–110 of 171 posts

Re: GitHub is degraded/down

#101

Post by ivanfon »

hellepardo wrote: post #12
vikramkr wrote: post #8

Earlier quoted context omitted.

It might also be covid related. People are working from home, people responsible for system upkeep might not be immediately responsive, more demand on the servers for whatever reason, etc.

I would agree that the coronavirus could be a factor here. At the same time, I've been noticing issues since probably December or January (before the coronavirus started being a real problem), which makes it seem like maybe there are multiple issues. Of course, I'm not actually internal to Microsoft or Github, so I have no idea and it's all opaque to me.

Same here - seems like these issues have been going on since last year, especially with Actions.

Re: GitHub is degraded/down

#102

Post by bwooceli »

exabrial wrote: post #74

There are larger problems I think... our call center is having trouble with Five9 in San Francisco. I also I tried setting up an EC2 instance in North Cali and just pinging it from Kansas City via Google fiber experienced 37% packet loss.

same with Five9 from the KC Metro, also accessing azure resources. No issues on my Google Fiber. But we have agents all over the midwest, all having issues.

Re: GitHub is degraded/down

#105

Post by bob1029 »

dijit wrote: post #64
bob1029 wrote: post #57

Today marks the 3rd time I've broached the topic w/ management of getting the self-host enterprise option... Compared to (public) GitHub's problems this year so far, our AWS EC2 instances are orders of magnitude more reliable. Sure, the internet can still go down, but my VPN into us-east-1 from Texas has been unbroken for weeks now. At this point I'd almost prefer to pull it all in-house and manage it myself so the e…

I'm not sure the irony is lost on you that; people generally prefer services like github (and; similarly EC2) precisely _because_ it's not on-prem and that if it's down there are hundreds of talented engineers working to resolve those issues. Unfortunately anecdotal experience here is spotty. Services I run for my team are several 9's higher in terms of actual availability (note: I did not say "uptime"); and contrari…

Does the GH Enterprise option not provide some degree of support with initial setup and configuration? Is the enterprise support more or less responsive than the public channels? I would expect those same engineers are also responsible for maintaining the enterprise offering.

Also, does isolation of a private GH instance from the public instances not provide some degree of added reliability considering the potential for DDOS or simply extreme load?

I absolutely grant you the IT infrastructure concerns. BUT Amazon is our vendor on that. GitHub provides the software. It's not like I'd be standing up a new series of physical hosts to run an on-prem GitHub built from source and managing all of the hell around that. This would be simply putting a GH-provided image on a EC2 instance and making sure we have frequent snapshots.

Re: GitHub is degraded/down

#106

Post by EForEndeavour »

MattGaiser wrote: post #53
robertakarobin wrote: post #23

I don't think it's just GitHub. I noticed everything seemed slow but thought it was just my ISP. Turns out from looking at downdetector.com there seems to be a large spike in reported problems across many sites and providers, all occurring at the same time.

Maybe the cloud providers are running out of capacity?

Maybe The Cloud is full.

Re: GitHub is degraded/down

#107

Post by mwcampbell »

ddevault wrote: post #93
mwcampbell wrote: post #89

Earlier quoted context omitted.

Thanks for being so transparent about your operations. Maybe this is somewhere in the manual and I missed it, but do you have some way of automating the configuration of your hosts and VMs? For example, do you use something like Ansible?

No, I provision them manually. Being based on Alpine Linux makes this less time-consuming and more deterministic. At some point I might invest in something completely automated, but right now the manual approach is simpler - and if it's not broke, don't fix it.

Ah, OK. Also, have you written anywhere about why you chose to use colocation rather than VPS (or "cloud") hosting, or leased dedicated hosting for the CI system? If you could use someone else's hardware rather than having to select, buy, and set up your own, then at least in theory, you could spend more time on other things. But I'm sure you have your reasons for making the choice that you did. I'm just curious about what those reasons are, if you're inclined to share.

Re: GitHub is degraded/down

#108

Post by rustyboy »

bob1029 wrote: post #57

Today marks the 3rd time I've broached the topic w/ management of getting the self-host enterprise option... Compared to (public) GitHub's problems this year so far, our AWS EC2 instances are orders of magnitude more reliable. Sure, the internet can still go down, but my VPN into us-east-1 from Texas has been unbroken for weeks now. At this point I'd almost prefer to pull it all in-house and manage it myself so the e…

I work for a fortune 100 company and GHE has been a nightmare. Not only do you need a team to maintain it, but then that team needs to be equipped enough to help support any other internal services (CI/CD pipelines) that integrate with it. Github.com is just an infinitely more enjoyable experience.

Re: GitHub is degraded/down

#110

Post by api »

vikramkr wrote: post #8
hellepardo wrote: post #2

Is it just me, or has Github's quality of service been continually degrading over the past several months? What is going on internally? Is this because of the Microsoft acquisition? Increased usage? An internal transition to Azure? ...is it time to move away from Github?

It might also be covid related. People are working from home, people responsible for system upkeep might not be immediately responsive, more demand on the servers for whatever reason, etc.

Cloud servers everywhere are also under much heavier load. They may have moved much of it into Azure, which is famously overloaded right now.
Post reply on HN