Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Actions and Pages

githubstatus.com

Post reply on HN
Print view View original

231–240 of 407 posts

Re: Incident with Actions and Pages

#231

Post by bobthepanda »

Doohickey-d wrote: post #113
echelon wrote: post #83

Earlier quoted context omitted.

In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s. Maybe the Github Actions infrastructure isn't run like that. edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262

Im curious about this: because in my experience (working on smaller services though), a small number of errors is always there, as a "baseline". Recently there was this: https://news.ycombinator.com/item?id=47252971 "10% of Firefox crashes are caused by bitflips" Which makes me think a small amount of random issues which happen even though nothing is broken, is normal everywhere. Especially once move things around on…

It’s where monitoring for 9s is more important at that scale than absolute errors. So long as degradation is graceful or retried it should not be a massive problem.

It does require constant tuning and adjustment though.

Re: Incident with Actions and Pages

#232

Post by llbbdd »

gilrain wrote: post #35
cebert wrote: post #30

Earlier quoted context omitted.

GitHub had a blog post about this recently. They reported a significant uptick in volume (repos created, PRs, etc.), which they attribute to AI usage and tooling.

Do you really believe their competition hasn’t seen the same increase? Because their competition certainly hasn’t seen the same instability issues.

What competition?

Re: Incident with Actions and Pages

#234

Post by ibejoeb »

What problem is github solving that has led it to become critical infrastructure for so many? Is it that everyone is remote and VPNs are too much of a hassle to give everyone access to a build server? Is the serving as the authoritative auth for development services? Does it provide better compliance reporting? It just isn't apparent to me what github offers that you can't get elsewhere with at the same cost and effort. I've been in some pretty large orgs with distributed personnel, but this just hasn't ever been a problem.

Re: Incident with Actions and Pages

#235

Post by yoyohello13 »

bouk wrote: post #14

Insane, we have to come up with contingency plans now for long-duration GitHub outages because we can't safely do deployments. For a service we're paying thousands of $ per year for even though we host runners ourselves...

Self host gitlab. If you already host runners it’s not a big lift.

Re: Incident with Actions and Pages

#236

Post by sylware »

matt_kantor wrote: post #56
sylware wrote: post #32

microsoft github should work at restoring interop with noscript/basic HTML browsers...

I agree, but that's not at all related to this outage.

Yeah, just reminding people here about that.

I am trying to refrain my "off topic" rants... but such microsoft github abuse is generating so much hate due to their dominant market position, it is hard.

Re: Incident with Actions and Pages

#237

Post by pdpi »

paularmstrong wrote: post #212
BrunoBernardino wrote: post #91

If you don't want to self-host Gitea/Forgejo, I recommend SourceHut for private repos and Codeberg for public ones. Happy to answer any questions you might have for either based on my experience!

What's wrong with codeberg for private repos?

They only allow private repos as an exception, and only insofar as they're ancillary to open source projects.

From their FAQs[0]:

> Codeberg's mission is to promote free/libre software. Keeping software private is obviously not our primary use case, but we acknowledge that private repositories are useful or necessary at times.

0. https://docs.codeberg.org/getting-started/faq/

Re: Incident with Actions and Pages

#238

Post by dawnerd »

gilrain wrote: post #35
cebert wrote: post #30

Earlier quoted context omitted.

GitHub had a blog post about this recently. They reported a significant uptick in volume (repos created, PRs, etc.), which they attribute to AI usage and tooling.

Do you really believe their competition hasn’t seen the same increase? Because their competition certainly hasn’t seen the same instability issues.

I’ve interviewed a lot of people and when asking about their git experience they’ve said they use GitHub. To a lot of devs they are the same thing.

Re: Incident with Actions and Pages

#239

Post by lokar »

hnlmorg wrote: post #223
dncornholio wrote: post #210

Earlier quoted context omitted.

> If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well. Isn't that what monitoring actually is? The issue seems to be in their testing, not monitoring.

No, monitoring for HTTP response code is a subset of observability and not one that generally gives you the best insights into which subsystems are misbehaving nor why. There are synthetic tests, where you can generate API request calls or even simulate an entire user journey. These allow you to control the user agent, the payloads, and thus you know anything errors back are actual errors. These are triggered by the…

Even a synthetic probe needs a few failures to trigger an alert.

You should not alert on cpu, ram, etc

Re: Incident with Actions and Pages

#240

Post by Xorlev »

ibejoeb wrote: post #234

What problem is github solving that has led it to become critical infrastructure for so many? Is it that everyone is remote and VPNs are too much of a hassle to give everyone access to a build server? Is the serving as the authoritative auth for development services? Does it provide better compliance reporting? It just isn't apparent to me what github offers that you can't get elsewhere with at the same cost and effo…

GitHub solved the original "code collaboration" problem, and now it's a default easy way to outsource repo management. It also has the most integrations. A lot of companies grew up using GitHub.

GitHub was, once upon a time, quite stable. Things have changed: more features, more usage, and automated agents.

Post reply on HN