Earlier quoted context omitted.
I can't tell if this is sarcasm or not.
I'm pretty sure if I e-mailed my sales rep right now they would tell me that Azure Dev Ops doesn't have these problems.
Still can't tell if this is sarcasm or not.
Ask HN: Is GitHub down?
121–130 of 194 posts
Re: Ask HN: Is GitHub down?
#122I noticed Github's OIDC token changed about a half an hour ago. Security incident?
Interesting observation, but I'd be surprised if that was related to a regional network fault
Re: Ask HN: Is GitHub down?
#123Everytime github goes down, and my push/pull is rejected, I immediately assume they’ve discovered I’m incompetent and fired me. And I’m the head of engineering at my company.
I resonate with this.
Re: Ask HN: Is GitHub down?
#124Earlier quoted context omitted.
Pretty much every company has been shown to have fake status pages at this point.
Status pages are updated by humans and the humans need to (1) realize there's a problem and (2) understand the magnitude of the problem and (3) put that on the status page. It's not fake, it's just a human process. And automating this would be error prone just the same.
Very good points. Meanwhile I have clients asking me why they can't have a status page to which I reply: you can, but ultimately to be completely fail proof it will be a human updating it slowly. To which they reply: but GitHub or X does it...
Very infuriating, that.
Re: Ask HN: Is GitHub down?
#125So down right now...I wonder why they still use https://www.githubstatus.com/ that reports everything is alright when it's not!
If it takes someone to manually change it from green to red, that does seem to defeat the purpose.
Not really, things fail in unexpected ways. Automated anomaly detection is notoriously error prone, leading to a lot of false positive and false negatives, in the trivial case of monitoring a single timeseries. For a system the size of GitHub, you need to monitor a whole host of things and if it's quasi impossible to do one timeseries well, there's basically no hope of doing automated many timeseries anomaly detection with a signal-to-noise ratio that's better than "humans looking at the thing and realizing it's not going well".
There's stuff like this that can't be automated well. The automated result is far worse than the human-based alternative.
Re: Ask HN: Is GitHub down?
#126[deleted]
Re: Ask HN: Is GitHub down?
#127I noticed Github's OIDC token changed about a half an hour ago. Security incident?
They added a second token on Tuesday: https://github.blog/changelog/2023-06-27-github-actions-upda...
Re: Ask HN: Is GitHub down?
#128Looks like we are back online.
Re: Ask HN: Is GitHub down?
#129often people point out how unreliable self-hosted services are, well, hosted services are just as unreliable if not more.
this, ladies and gentlemen, is why you should always self host critical infrastructure
Re: Ask HN: Is GitHub down?
#130Update We have identified the root cause of the outage and are working toward mitigation Posted 4 minutes ago. Jun 29, 2023 - 18:02 UTC
Seems like it's coming back online in fits and starts