Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub availability report: October 2022

github.blog

Post reply on HN
Print view View original

41–42 of 42 posts

Re: GitHub availability report: October 2022

#41

Post by smcleod »

everfrustrated wrote: post #2

Of all the many SaaS vendors I use, GitHub has the worst availability by far. There isn't a month that goes by without our devs being impacted. GitHub - please just work on fixing this. Your product is great but your availability is your biggest problem. It's beyond a joke at this point.

I absolutely second this. Their APIs are incredibly flaky - especially with Actions, and it's incredibly annoying that they don't report on their _real_ availability - I can't remember the last time there was any form of related service degradation on their customer facing monitoring during an outage.

Re: GitHub availability report: October 2022

#42

Post by throwawaythekey »

kgeist wrote: post #26

>Attempting to retry these failed jobs tied up our worker and it was unable to process new incoming events, resulting in a severe backlog in our queues. Interesting - we have this kind of thing quite often. Basically, an event is stuck in the queue due to a logic error or a prior race condition, and it's endlessly retried blocking the rest of the events from being processed. We can't just automatically remove such an…

It sounds like you've identified the issue yourself. You are relying on ordering when processing events. You need to either loosen that requirement or do better testing to prevent head of line blocking.

I don't know much about your application but the fact that you can mitigate the problem by scaling the number of workers suggests that the order requirements might actually be fairly weak. As a worst case outcome you may be able to push all events interdependent to the one with an error to a DLQ using a temporary blacklisting mechanism, but by that stage I think I would just prefer better testing.

Post reply on HN