Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Issues and Pull Requests

githubstatus.com

Post reply on HN
Print view View original

81–90 of 143 posts

Re: Incident with Issues and Pull Requests

#81

Post by qmacro »

edgyquant wrote: post #74
iepathos wrote: post #67

Earlier quoted context omitted.

They blamed the march and april outages on some database query that was changed due to an infrastructure change they rolled out. I'm guessing their infrastructure change caused some other race condition issue that they are only seeing after major production failure due to not load testing enough in their staging environment https://github.blog/2023-05-03-github-availability-report-ap...

As much as I’ve been frustrated by these outrages, we’ve all been there

Now that is a good typo

Re: Incident with Issues and Pull Requests

#82

Post by dugmartin »

I've had some actions queued for multiple days now on certain repos, but not others. I've cancelled them and restarted them during the green status intervals but they all go back to "Queued". I've also cancelled them and then made slight documentation tweaks to get new commit hashes on the branches and it still goes to queued.

Re: Incident with Issues and Pull Requests

#84

Post by tonyhb »

dbingham wrote: post #20

What is going on over there? Third day in a row is... kind of impressive.

From an SRE, one of their DB clusters failed. They use Vitess which is great, but it can be prone to hotspots and doesn't auto-shard. Heavy usage (esp. from large customers, rogue jobs) can take down the cluster. When it goes down, it's a PITA to resolve.

Re: Incident with Issues and Pull Requests

#85

Post by misnome »

capableweb wrote: post #56
atl4s wrote: post #17

Just lost a merge commit to dev/null. This is getting tiresome

How??? You do the merge, which either creates a new commit for the change, or appends the commits to your existing tree. Then you push that to the remote. If the push fails, you can just push again, it's not lost. And if the merge failed, you didn't have any merge commit to begin with.

There used to be a pretty consistent bug that if an on-site PR merge failed but you clicked "Retry", that it just did a basic non-squash full-merge discarding all your commit message work, often requiring a revert to tidy things up. It could be similar to that.

Re: Incident with Issues and Pull Requests

#88

Post by bluehatbrit »

greenie_beans wrote: post #12

this is the third day in a row this is a problem yet they're framing it as a new outage.

I'd say it's reasonable to list it as separate outages on the status page as it's really a representation of "is github available and working as expected". Even if it is the same issue, when they manage to mitigate it (or it goes away) I'd want to see that everything is now available from a user perspective.

That said, they're getting to the point where they really need to make some larger post about this. It seems reasonable to assume it is all from one root cause.

Re: Incident with Issues and Pull Requests

#90

Post by samlambert »

tonyhb wrote: post #84
dbingham wrote: post #20

What is going on over there? Third day in a row is... kind of impressive.

From an SRE, one of their DB clusters failed. They use Vitess which is great, but it can be prone to hotspots and doesn't auto-shard. Heavy usage (esp. from large customers, rogue jobs) can take down the cluster. When it goes down, it's a PITA to resolve.

This literally isn't true and looks awfully like the talking points of one of our competitors.
Post reply on HN