Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub availability report: October 2022

github.blog

Post reply on HN
Print view View original

21–30 of 42 posts

Re: GitHub availability report: October 2022

#21

Post by byroot »

pdimitar wrote: post #20
eddsh1994 wrote: post #17

Earlier quoted context omitted.

Why do you assume the outages are language related and not due to the complex product having bugs? How does Rust prevent bad schema changes or missing data in the DB?

Because I worked with Rails for 6.5 years. Outages beyond smaller scales were at best a weekly occurrence. Obviously I can't know for sure but it's not an uninformed assumption.

You know you can just click on the post title, that will open the posted link in which you can read the detailed cause of all the outages they had that month.

If you do this, you will realize that none are close to what you describe.

Also have you considered that if you had weekly outage when billion dollars companies continued to stick with Rails, maybe you were the problem?

Re: GitHub availability report: October 2022

#22

Post by pdimitar »

byroot wrote: post #21
pdimitar wrote: post #20

Earlier quoted context omitted.

Because I worked with Rails for 6.5 years. Outages beyond smaller scales were at best a weekly occurrence. Obviously I can't know for sure but it's not an uninformed assumption.

You know you can just click on the post title, that will open the posted link in which you can read the detailed cause of all the outages they had that month. If you do this, you will realize that none are close to what you describe. Also have you considered that if you had weekly outage when billion dollars companies continued to stick with Rails, maybe you were the problem?

I did read the article. One of the incidents was about their webhook worker(s) being swamped -- plus had errors due to deleted DB workloads that were necessary for the event to be processed. So I'd count that one as a slow endpoint attributed to Ruby on Rails (and it's famous for that).

And even if zero of their incidents alluded to performance problems with Rails I still worked a lot with it and I know for a fact that it's a factor.

Your snark doesn't change reality but you are free to pretend otherwise, fine with me.

> Also have you considered that if you had weekly outage when billion dollars companies continued to stick with Rails, maybe you were the problem?

Indeed, a programmer not having executive powers to influence change of deployment tech and server (was Puma at the time) is indeed me being a problem, surely. Especially after he made a study demonstrating the problems and calculated how much programmer time is wasted on these matters every week and he still got ignored. Perhaps I am the problem indeed!

Re: GitHub availability report: October 2022

#23

Post by eddsh1994 »

pdimitar wrote: post #22
byroot wrote: post #21

Earlier quoted context omitted.

You know you can just click on the post title, that will open the posted link in which you can read the detailed cause of all the outages they had that month. If you do this, you will realize that none are close to what you describe. Also have you considered that if you had weekly outage when billion dollars companies continued to stick with Rails, maybe you were the problem?

I did read the article. One of the incidents was about their webhook worker(s) being swamped -- plus had errors due to deleted DB workloads that were necessary for the event to be processed. So I'd count that one as a slow endpoint attributed to Ruby on Rails (and it's famous for that). And even if zero of their incidents alluded to performance problems with Rails I still worked a lot with it and I know for a fact th…

> However, many of these events caused exceptions in our webhook delivery worker because data needed to generate their webhook payloads had been deleted from the database. Attempting to retry these failed jobs tied up our worker and it was unable to process new incoming events, resulting in a severe backlog in our queues.

I bet you I could cause this bug on a Rust product if you let me near the code ;)

Re: GitHub availability report: October 2022

#24

Post by pizza234 »

brink wrote: post #3

I'm not sure about the technical details behind their outages since they're a little vague on that, but it's funny how every Rails developer champions Github as a Ruby on Rails shop as why Rails should continue living on when their availability is some of the worst in the tech scene. Lazy evaluation is great until it's not.

All the incidents seem to be platform-agnostic:

- improper database validation

- older component not tested against configuration change

- uncontrolled automation DOS

- incompletely distributed secrets

Re: GitHub availability report: October 2022

#25

Post by pdimitar »

eddsh1994 wrote: post #23
pdimitar wrote: post #22

Earlier quoted context omitted.

I did read the article. One of the incidents was about their webhook worker(s) being swamped -- plus had errors due to deleted DB workloads that were necessary for the event to be processed. So I'd count that one as a slow endpoint attributed to Ruby on Rails (and it's famous for that). And even if zero of their incidents alluded to performance problems with Rails I still worked a lot with it and I know for a fact th…

> However, many of these events caused exceptions in our webhook delivery worker because data needed to generate their webhook payloads had been deleted from the database. Attempting to retry these failed jobs tied up our worker and it was unable to process new incoming events, resulting in a severe backlog in our queues. I bet you I could cause this bug on a Rust product if you let me near the code ;)

Oh, absolutely. It can happen everywhere -- in theory.

In practice however, I found people working with certain languages and stacks to be more thorough. Still largely depends on the person in the important position though, that much is always true.

Re: GitHub availability report: October 2022

#26

Post by kgeist »

>Attempting to retry these failed jobs tied up our worker and it was unable to process new incoming events, resulting in a severe backlog in our queues.

Interesting - we have this kind of thing quite often. Basically, an event is stuck in the queue due to a logic error or a prior race condition, and it's endlessly retried blocking the rest of the events from being processed. We can't just automatically remove such an event from the queue because events must be processed in order or client data can get corrupted. It requires manual intervention (we have alerts in place), and every time it's a new event so we have to be creative and think quickly - how to unblock the queue without corrupting client data by skipping events. After an event is unstuck, there's a huge queue of unprocessed events which can take up to a few hours to be emptied in worst cases. Fortunately we have some sharding in place so there can be several independent workers processing the same global queue - with workers' shard affinity we can process shard data in order AND in parallel, so SRE can temporarily increase the number of workers when the queue gets too large, to speed it up. I still don't know how to solve this kind of problem once and for all (i.e. to have zero manual intervention). Is it even solvable?

Re: GitHub availability report: October 2022

#27

Post by lol768 »

everfrustrated wrote: post #2

Of all the many SaaS vendors I use, GitHub has the worst availability by far. There isn't a month that goes by without our devs being impacted. GitHub - please just work on fixing this. Your product is great but your availability is your biggest problem. It's beyond a joke at this point.

> Of all the many SaaS vendors I use, GitHub has the worst availability by far.

Have you ever tried using GitLab?

Re: GitHub availability report: October 2022

#28

Post by byroot »

pdimitar wrote: post #22
byroot wrote: post #21

Earlier quoted context omitted.

You know you can just click on the post title, that will open the posted link in which you can read the detailed cause of all the outages they had that month. If you do this, you will realize that none are close to what you describe. Also have you considered that if you had weekly outage when billion dollars companies continued to stick with Rails, maybe you were the problem?

I did read the article. One of the incidents was about their webhook worker(s) being swamped -- plus had errors due to deleted DB workloads that were necessary for the event to be processed. So I'd count that one as a slow endpoint attributed to Ruby on Rails (and it's famous for that). And even if zero of their incidents alluded to performance problems with Rails I still worked a lot with it and I know for a fact th…

> their webhook worker(s) being swamped

That's a capacity problem caused by a logic bug. Nothing stack specific. If you throw more work at a system than it is designed to handle, you'll hit a bottleneck.

> Your snark doesn't change reality

What reality? You are just barking your uneducated opinion. No one who ever worked on a service anywhere close to the scale of GitHub (regardless of the stack) would make such statements.

Re: GitHub availability report: October 2022

#29

Post by simonpantzare »

kgeist wrote: post #26

>Attempting to retry these failed jobs tied up our worker and it was unable to process new incoming events, resulting in a severe backlog in our queues. Interesting - we have this kind of thing quite often. Basically, an event is stuck in the queue due to a logic error or a prior race condition, and it's endlessly retried blocking the rest of the events from being processed. We can't just automatically remove such an…

Sounds like a DAG based task orchestrator could be a good fit. Where tasks state their dependencies and are allowed to run only when they have all completed.

Re: GitHub availability report: October 2022

#30

Post by [deleted] »

byroot wrote: post #28
pdimitar wrote: post #22

Earlier quoted context omitted.

I did read the article. One of the incidents was about their webhook worker(s) being swamped -- plus had errors due to deleted DB workloads that were necessary for the event to be processed. So I'd count that one as a slow endpoint attributed to Ruby on Rails (and it's famous for that). And even if zero of their incidents alluded to performance problems with Rails I still worked a lot with it and I know for a fact th…

> their webhook worker(s) being swamped That's a capacity problem caused by a logic bug. Nothing stack specific. If you throw more work at a system than it is designed to handle, you'll hit a bottleneck. > Your snark doesn't change reality What reality? You are just barking your uneducated opinion. No one who ever worked on a service anywhere close to the scale of GitHub (regardless of the stack) would make such stat…

[deleted]
Post reply on HN