Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Disruption with Some GitHub Services – Resolved

githubstatus.com

Post reply on HN
Print view View original

21–30 of 176 posts

Re: Disruption with Some GitHub Services – Resolved

#21

Post by inigyou »

ferguess_k wrote: post #15
everfrustrated wrote: post #14

> We've identified an issue with a database primary and are failing over to a replica immediately This is why it's hard to take GitHub seriously. How can a single database cause an outage for everyone? This is amateur stuff. Have they no sharding or partitioning internally? Paying customers should not be impacted in the same way as free ones are.

I wonder what is this database, and why it is hard to fall-over automatically.

RDBMS replication and failover is way more difficult and manual than anyone would like. You can't just set up two postgres, tell them they're clustered and have it basically work; at a minimum you have to design the client to somehow know which one is currently the master, or use some sort of proxy (which becomes its own SPOF).

RDBMS integrity basically requires that one master server is responsible for the whole data set and other servers may replicate from it. And it usually doesn't wait for a quorum of replicas, just for one, because the design is to recover from a hardware failure, not a network partition, although that could be fixed at the cost of increased latency.

Re: Disruption with Some GitHub Services – Resolved

#27

Post by everfrustrated »

sdetheridge wrote: post #19
nr378 wrote: post #10

GitHub needs to completely bifurcate their enterprise/paid services from their free services at the infra level.

According to their status pages (e.g. https://eu.githubstatus.com/ , https://us.githubstatus.com/ ), their Enterprise Cloud uptime for Actions is significantly higher.

That is a different and later product with a confusingly similar name.

Re: Disruption with Some GitHub Services – Resolved

#28

Post by thecosmicfrog »

ad_fontes wrote: post #6

> Update - We've identified an issue with a database primary and are failing over to a replica immediately Seems like a weird thing to post on a status page. Shouldn't this have happened automatically and therefore precluded the need to inform users of it?

> Update - primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues

Re: Disruption with Some GitHub Services – Resolved

#29

Post by saxonww »

nr378 wrote: post #10

GitHub needs to completely bifurcate their enterprise/paid services from their free services at the infra level.

They have that-ish as an option: https://docs.github.com/en/enterprise-cloud@latest/admin/dat...

I'm told that GitHub has asserted to us that moving to this model means we would not be exposed to github.com outages. It's not at feature parity with github.com though.

Re: Disruption with Some GitHub Services – Resolved

#30

Post by gaigalas »

weli wrote: post #17
nr378 wrote: post #10

GitHub needs to completely bifurcate their enterprise/paid services from their free services at the infra level.

That's what I don't understand. They could mitigate their name so much if they just split free/paid/enterprise. It's already shown that enterprise is much more estable and is largely unaffected from service disruptions. Why don't they go one more layer? For sure it's worth the extra complexity.

Depending on the cause of the current issues, that move would likely cause more harm to paid services than good.

Their last postmortem made clear that their challenges are operational. Scale puts pressure on operation, but it's not what blocks them from keeping up.

Doubling the operation doubles the operational challenges.

Post reply on HN