61–70 of 125 posts
Earlier quoted context omitted.
Had the same issue this morning. The lagging status always causes the issue of "is it you, me or GitHub?" snaffoos. Really annoying to have these issues so consistently. Would switch to gitea or similar in a moment given the choice.
Just a friendly correction: `SNAFU:' Situation Normal: All Fucked Up
Foobar: for when you are too polite to say FUBAR (Fucked Up Beyond All Recognition).
GitHub actions downtime is becoming painful for us. Having been lured on there with 10,000 included minutes which they shortly thereafter dropped to 3,000 I feel aggrieved paying for overages incurred from actions regularly shitting the bed.
Also having outages at Azure DevOps Pipelines every other month or so it seems. And that's paying - for hours there's no mention on the status page and we are stuck there, not being able to merge PRs or release our app in the standard way.
It's gotten more reliable over time (especially selenium events being dropped on the floor causing tests to stall and fail), but I used to have to babysit it quite a bit and there were quite a number of times where IE instances just would not spool up (with a multiple minute timeout set). Sometimes it was a one-shot thing, other times it went on for hours.
During these incidents the average allocation times listed on their status page would double for Windows VMs (I don't recall the exact numbers but they were on the order of 10 seconds vs 5) but nothing would be red, and most of the time nothing ever did go red.
And that's what you get for using averages for things and divide infinity by n improperly.
I was seeing PRs failing to update & webhooks failing to trigger upon pushing code for 30 minutes before GH's status page acknowledged anything. I'm surprised they don't have monitoring in place that would catch webhooks failing within minutes of the failure beginning.
https://gitea.io/en-us/ https://git.zx2c4.com/cgit/ https://about.gitlab.com/install/?version=ce
Frankly I can't wait. Imagine being able to reference other users across instances with @username:instance or something to that extent, or projects and tickets.
GitHub actions downtime is becoming painful for us. Having been lured on there with 10,000 included minutes which they shortly thereafter dropped to 3,000 I feel aggrieved paying for overages incurred from actions regularly shitting the bed.
Github actions has been a huge let down for me. Between uptime issues and the lack of support for so many basic CI features is killing it for me (and has been for a year). The only reason we're using it is because it's free..
Earlier quoted context omitted.
Yeah it's very frustrating - especially if your customers are technical, they are seeing the errors and the status page says everything is fine. I've seen status pages and error counts tied to bonuses, which only caused a giant mess of bad incentive alignment and internal lies, customers are unhappy, developers are unhappy, management are lying to upper management, it's so much easier to focus efforts on real problem…
Do these companies not have live error reporting and tracing? Like surely github got alerts that things weren't working? Why don't they just hookup their status page and their alerts? Or is it a political/relationship thing, and they want to have a human give out the status page updates? This could have been caught with a cron job and some curl requests :\
Building infra I have to work around all sorts of 3rd party services going out or having blips throughout the day, docker registries, caches, bgp, etc., it's totally an expected part of infra design but not every team has the time or need to build in the resiliency. I see tons of outages that never get reported or IMO aren't reported adequately enough.
With that said, I'm no angel, I get all my service down notifications through slack, so when slacks down..
Earlier quoted context omitted.
Just a friendly correction: `SNAFU:' Situation Normal: All Fucked Up
This misspelling brought to you by foobar. Foobar: for when you are too polite to say FUBAR (Fucked Up Beyond All Recognition).
Named, of course, by the Army when they built the Alaska Highway.
Earlier quoted context omitted.
I read an article about it, look for "status page evolution" https://nimbleindustries.io/2020/06/04/has-github-been-down-...
From the first two graphs it looks like they are a lot less liberal about using "down" instead of "warn".
Severity went something like this (sometimes the numbers flip which always confuses at least 20% of the team about whether things are almost normal or people are hunting each other for sport).
1: data loss
2: some workflows blocked
3: some workflows unavailable w/ workarounds (ie other routes)
4: Everything else except
5: Irritations
Having a UI break but the underlying functionality is still working is not good but people can still do their jobs, if more slowly. It's important to classify these separate from S2 and S4. There is urgency but don't panic. Go eat lunch or have your planning meeting, then go fix it. If data is getting lost ain't nobody doing nothin' until we figure it out, and then some people can go back to work but don't interrupt the people still working on it.
I think the problem is that so many metrically dysfunctional people, to the point of cliché, have rationalized that an S2 means that only 20% of our customers can't do their jobs so we are degraded but still working normally, when really a yellow status should be at S3, while S2 should be at least orange although those affected will be upset that it's not red.
Over time that 20% will shift around to most of your customers. Eventually several times, and then you'll wonder why everyone is talking trash about you on HN. It's not like that many people were affected!
Don’t host yourself, it’s impossible to meet the reliability of the professionals
I don't regret it.