Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Actions and Pages

githubstatus.com

Post reply on HN
Print view View original

201–210 of 407 posts

Re: Incident with Actions and Pages

#201

Post by halapro »

jillesvangurp wrote: post #150

I've been against self hosting internal tools for a long time mainly because of the devops and other overhead. But AI based devops makes it so easy now to spin up whatever you want now that I'm reconsidering that. I use a lot of ansible for several of our deployments. At this point, most of that is managed via codex. For Git, all you technically need is ssh access and some backup strategy for your server. It would be…

> For Git, all you technically need is

Technically Dropbox is just rsync.

Also https://xkcd.com/1319/ but for maintenance.

Re: Incident with Actions and Pages

#202

Post by compumike »

echelon wrote: post #83
jaapz wrote: post #62

Earlier quoted context omitted.

All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.

In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s. Maybe the Github Actions infrastructure isn't run like that. edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262

Re: "page for all 500s": there's a world of difference between "page me with a critical alert at 3am" and "notify me on Monday morning when my normal workday starts". At the extremes:

If my DB health check endpoint is returning 500s for N consecutive checks over M minutes, yeah, please wake me up at 3am!

If one user hit a weird edge case in form validation and got a one-off 500, please don't! We can fix that on Monday.

Not always easy to distinguish those clearly or configure those business hours rules, but for my team at https://heyoncall.com/ that is the goal -- otherwise your team burns out fast. Waking up someone at 3am has a real cost, so you better be sure it's worth it.

Re: Incident with Actions and Pages

#203

Post by hvb2 »

echelon wrote: post #83
jaapz wrote: post #62

Earlier quoted context omitted.

All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.

In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s. Maybe the Github Actions infrastructure isn't run like that. edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262

> A noisy pager forces the team to fix the 500s.

I'm sure you're not in ops. Or in a dev org of a service with decent request rates.

What you're asking for is a service to fail silently. There's no way a service with a decent request rate to have 0 500s. Not when it still sees development.

A 50 year old bank API? Maybe...

Re: Incident with Actions and Pages

#204

Post by theta_d »

TheDong wrote: post #108
echelon wrote: post #83

Earlier quoted context omitted.

In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s. Maybe the Github Actions infrastructure isn't run like that. edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262

Do you know of a single service at a single company that actually does that? I know all of Gmail, every GCE service I can think of, every AWS service I can think of, Amazon.com, Netflix, and Github all do not page on just a single 500. I know none of those are particularly "high performance" though. Curious where your experience is coming from.

The sub-service at IBM cloud I worked on had an insanely small error budget such that pages were nearly constant. On call was hell week until a few of us insisted on fixing the issues. The "few" of us were contractors. The employees seemed more than willing to just let the pages continue.

Re: Incident with Actions and Pages

#205

Post by LPisGood »

sunrunner wrote: post #174
echelon wrote: post #134

Earlier quoted context omitted.

I worked at a large fintech moving billions of dollars in volume a day. I had a fairly long tenure, where I maintained multiple key services in critical online payments flow. Authentication, authorization, core business and risk data, as well as some cross-cutting control plane stuff, etc. You needed one or more of our services to take a payment, serve any request from the employee dashboard - pretty much everything…

> We paged for every single 500. Assuming the existence of some kind of network (with zero guarantee of 100% reliability), how does this work in practice? Is each 500 treated as an event that needs investigation, even if the result of that would end up as 'a router dropped something from an internal buffer but the transaction as a whole was re-tried by a parent so the service itself recovered'?

A reliability engineer from Jane Street gave a great talk about this, five nine’s of correctness in reporting, etc isn’t enough for the SEC.

https://youtu.be/zR9PpXWsKFQ

Re: Incident with Actions and Pages

#207

Post by rhyperior »

echelon wrote: post #83
jaapz wrote: post #62

Earlier quoted context omitted.

All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.

In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s. Maybe the Github Actions infrastructure isn't run like that. edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262

You only do this when you’re trying to use incident management as a hammer to make a point to somebody whom you have otherwise failed to convince to fix something through persuasive argument. Ie, it’s punitive.

Re: Incident with Actions and Pages

#208

Post by cryo32 »

bouk wrote: post #14

Insane, we have to come up with contingency plans now for long-duration GitHub outages because we can't safely do deployments. For a service we're paying thousands of $ per year for even though we host runners ourselves...

You should never entirely depend on a third party service for deployments.

Been burned too many times on that one.

Re: Incident with Actions and Pages

#210

Post by dncornholio »

hnlmorg wrote: post #151
jaapz wrote: post #62

Earlier quoted context omitted.

All these monitoring rules are of the format "when 500 errors > baseline for x minutes". Otherwise you'd have monitoring alerts every second. So it is normal for users to already see errors before github officially counts it as an outage.

You'd expect them to be monitoring more than just the HTTP response codes from user requests for precisely this reason. If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well. But effective monitoring is harder than people assume.

> If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well.

Isn't that what monitoring actually is? The issue seems to be in their testing, not monitoring.

Post reply on HN