Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

githubstatus.com

Post reply on HN
Print view View original

51–60 of 123 posts

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#51

Post by kortex »

ryanbrunner wrote: post #42
arpinum wrote: post #20

These incidents have to hurt Azure's brand value. It's a monster task to run something as big as GitHub, if they ever get it stable it will lend a lot of credibility to Microsoft's cloud skills.

There's not really all that much pointing to an infrastructure level failure - it's possible, but it's just as likely it's an application-level failure somewhere in Github's code. The API is returning 500s and not 503s and the failure is relatively quick, so it's not obviously a server outage.

It's yellow lights across the board, literally nothing is green. That's usually indicative of some sort of software infrastructure level failure or cascade failure, not an application-level failure, which usually manifests as one or two specific services going down (depending on how you define "infrastructure" and "application" - with IAC, arguably the software defined infrastructure _is_ an application). I doubt its a physical hardware issue. It's rarely hardware (except when your DS catches on fire).

No red lights, so it's probably not something catastrophic like that facebook DNS SNAFU, but it definitely smells infrastructure- or deployment-scoped. Like either small DNS issue, or some load balancers are sending traffic to servers which cannot handle it programmatically (schema change?) so they are barfing.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#53

Post by zinekeller »

arpinum wrote: post #20

These incidents have to hurt Azure's brand value. It's a monster task to run something as big as GitHub, if they ever get it stable it will lend a lot of credibility to Microsoft's cloud skills.

Serious questions:

1) Is GitHub runing under Azure's technology stack?

2) Is GitHub under Azure's mamagement (in contrast to Visual Studio's team)?

I'm not sure about two but I'm pretty sure that GitHub doesn't run under Azure at all, considering that GitHub has fully separate networking from MSN's/Azure's (and GitHub's machines do pingback unlike most of Microsoft's machines which don't).

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#54

Post by rvz »

bastardoperator wrote: post #39
rvz wrote: post #11

Here we go again. GitHub going completely down at least once a month as I said. [0] So nothing has changed. That is excluding the smaller intermittent issues. Let's see if anyone implemented a self-hosted backup or failsafe just in case. Oh dear. [0] https://news.ycombinator.com/item?id=30149071

The entire point of git is that it's decentralized, lol. If I've cloned locally like millions of people do daily, I have a backup.

> The entire point of git is that it's decentralized, lol.

No-one here is criticizing git itself. That is not the point.

It is GitHub that is defeating the whole point of it all, once their hosted central server goes down.

The majority of these projects went all in on GitHub, including using GitHub actions, npm packages, hosting their whole website, etc hence as soon as it goes down, they can't push or update anything; especially if it was very urgent. It has become a giant single point of failure for nearly everything.

There is a reason why the Linux Kernel, Mozilla, Qt, Chromium, GNOME, ReactOS, etc self-host their own repositories and have fail-safes repositories if Github goes down and becomes unreliable.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#55

Post by pdenton »

intsunny wrote: post #33

Whew, outage timestamps in UTC. Now I won't have to know what time is it California, and if California currently has PST, PDT, PTSD, etc

As someone with diagnosed PTSD, I never thought I'd psychologically level with an entire state ;)

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#57

Post by adelarsq »

Wavelets wrote: post #12

Whew, glad I decided to scroll HN right now. I've been puzzling over why I'm getting "! [remote rejected] master -> master (Internal Server Error)" as well while trying to push and decided to take a break.

Time to take some coffee and configure Vim

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#58

Post by ctvo »

5e92cb50239222b wrote: post #44
avar wrote: post #35

I'm finding that pushes do go through eventually, this is probably grossly irresponsible, so I don't recommend its use, but I remembered I had this old alias to "push harder" in my ~/.gitconfig: [alias] thrust = "!f() { until git push $@; do sleep 0.5; done; }; f" I've done a few pushes so far, and found that it's going through in <10 tries or so.

It's fine. Maybe it will force them to finally start paying attention to the quality of their work. If crap I'm writing for a living was misbehaving that frequently, I'd be sweeping the streets by now (or doing some other work that's actually useful to society).

It's OK to be frustrated since we rely on GitHub so much, but this is unkind. Software is complex. GitHub operates at a scale few of us work at. There are people at the other end doing their best traversing complex internal systems (organization and tech).

I would argue GitHub has done more for societal good than most tech ventures, by the way.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#59

Post by egberts1 »

kortex wrote: post #51
ryanbrunner wrote: post #42

Earlier quoted context omitted.

There's not really all that much pointing to an infrastructure level failure - it's possible, but it's just as likely it's an application-level failure somewhere in Github's code. The API is returning 500s and not 503s and the failure is relatively quick, so it's not obviously a server outage.

It's yellow lights across the board, literally nothing is green. That's usually indicative of some sort of software infrastructure level failure or cascade failure, not an application-level failure, which usually manifests as one or two specific services going down (depending on how you define "infrastructure" and "application" - with IAC, arguably the software defined infrastructure _is_ an application). I doubt its…

Only load balancer (as an infrastructure) can hit the lights across the board. Not much else.

Re: Incident with GitHub Actions, API requests, Codespaces, Git operations, Issues

#60

Post by arpinum »

gtirloni wrote: post #41
arpinum wrote: post #20

These incidents have to hurt Azure's brand value. It's a monster task to run something as big as GitHub, if they ever get it stable it will lend a lot of credibility to Microsoft's cloud skills.

GitHub is pretty stable. What are you talking about? I doubt most GitHub users know it's on Azure.

Github Actions does not have good track record. https://www.githubstatus.com/history . You don't need a majority of GitHub users to understand it's owned by Microsoft for there to be an impact on brand value.
Post reply on HN