Will more copilot usage fix this? We should try more copilot.
Incident with Actions and Pages
221–230 of 407 posts
Re: Incident with Actions and Pages
#222Insane, we have to come up with contingency plans now for long-duration GitHub outages because we can't safely do deployments. For a service we're paying thousands of $ per year for even though we host runners ourselves...
It's always best to be portable - always be able to do builds and releases locally (at least, once you get the keys - it shouldn't be possible by default), then add things like github actions on top as convenience.
Re: Incident with Actions and Pages
#223Earlier quoted context omitted.
You'd expect them to be monitoring more than just the HTTP response codes from user requests for precisely this reason. If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well. But effective monitoring is harder than people assume.
> If the first they hear of an outage is when user requests start to fail, then that's a failure in their monitoring as well. Isn't that what monitoring actually is? The issue seems to be in their testing, not monitoring.
No, monitoring for HTTP response code is a subset of observability and not one that generally gives you the best insights into which subsystems are misbehaving nor why.
There are synthetic tests, where you can generate API request calls or even simulate an entire user journey. These allow you to control the user agent, the payloads, and thus you know anything errors back are actual errors. These are triggered by the observability platform (think like running a cron-job) and thus you're not tied to user activity to see when problems arise.
There are other metrics outside of HTTP response codes too. Think like free RAM, CPU usage, disk space, etc. This is just naming some obvious ones because these types of metrics are generally bespoke to the type of application your monitoring. And with these types of monitors, you'd not just have an alert when things have failed, but ideally have alerts when an irregular trend is showing that things are likely to fail too. This latter type of monitors helps you get ahead of the problem before it become customer facing.
Then you have more traditional stuff like logs. This will also be bespoke to the application. But you'd expect errors in logs to get surfaced quickly. Assuming Github have good hygiene in what's being logged.
Tie that up with APMs, RUM, and other goodies like that and you'll have diagnostics to investigate issues when they appear.
(this is just a super high level view of observability too)
Re: Incident with Actions and Pages
#224Re: Incident with Actions and Pages
#225Earlier quoted context omitted.
More likely that 'update the Status site' lives a long way down their incident response plan, and they have alarms going off well before that
it should be automatic tho. Probably isn't so they can at least get the one nine on availability
Marketing definitely takes interest in status sites
Re: Incident with Actions and Pages
#226here all is ok, 3 actions without problem
Re: Incident with Actions and Pages
#227Earlier quoted context omitted.
I worked at a large fintech moving billions of dollars in volume a day. I had a fairly long tenure, where I maintained multiple key services in critical online payments flow. Authentication, authorization, core business and risk data, as well as some cross-cutting control plane stuff, etc. You needed one or more of our services to take a payment, serve any request from the employee dashboard - pretty much everything…
> We paged for every single 500. Assuming the existence of some kind of network (with zero guarantee of 100% reliability), how does this work in practice? Is each 500 treated as an event that needs investigation, even if the result of that would end up as 'a router dropped something from an internal buffer but the transaction as a whole was re-tried by a parent so the service itself recovered'?
Client network timeout shouldn't result in 500. With 408 and retry you should, dependent on the business criteria, get either an upsert (transaction is retried) or 422 (validation that given entry already exists).
Even if it's "DB in datacenter I tried to save to was hit by meteor" event, you can cater for this not to result in 500 (ie - DB unreachable, retry in a couple of minutes); the question is if you want to.
Re: Incident with Actions and Pages
#228Earlier quoted context omitted.
Re: "page for all 500s": there's a world of difference between "page me with a critical alert at 3am" and "notify me on Monday morning when my normal workday starts". At the extremes: If my DB health check endpoint is returning 500s for N consecutive checks over M minutes, yeah, please wake me up at 3am! If one user hit a weird edge case in form validation and got a one-off 500, please don't! We can fix that on Monda…
Shouldn't Github be large enough to not have anyone on-call, but just rotate the responsible team around the world?
At least when I worked at a Bigcorp a lot of that was being cut to save costs.
Re: Incident with Actions and Pages
#229Will more copilot usage fix this? We should try more copilot.
no maybe we should make copilot the pilot so the bad humans in the loop finally cannot break anything.
Re: Incident with Actions and Pages
#230Insane, we have to come up with contingency plans now for long-duration GitHub outages because we can't safely do deployments. For a service we're paying thousands of $ per year for even though we host runners ourselves...
You should never entirely depend on a third party service for deployments. Been burned too many times on that one.
Ok.
Move to EC2.
Darn AWS is down.
Alright, run it on a Mac Mini in your basement. Ahh dawn, your ISP is having issues. Good thing you have a backup 5G hotspot.
Ohh no, the power is out.
Eventually you have to trust someone else.
GitHub is a tragedy of the Commons. Too many people are using it, and Microsoft isn't willing to handle it correctly.
Feels like a very good business opportunity. Minimum 50k yearly contracts, GitHub with actual uptime. GitPro ?