Addressing GitHub’s recent availability issues
1–10 of 12 posts
Re: Addressing GitHub’s recent availability issues
#2While the ad-hoc blog post to address last week's outage is appreciated, I'm curious as to why GitHub would continue to roll these up in a monthly report as opposed to when an investigation concludes? Many companies that operate live services that have a global user base do this now.
Re: Addressing GitHub’s recent availability issues
#3It's curious that all of the incidents are database related.
Re: Addressing GitHub’s recent availability issues
#4aaaaand it's down again
Re: Addressing GitHub’s recent availability issues
#5It's curious that all of the incidents are database related.
No, well kinda. I think it's related to how github chunks data on the backend. That chunking methodology does depend on the database. They mention it worked elsewhere in the org, which makes me wonder if it was a fix to support their managed hosts.
Re: Addressing GitHub’s recent availability issues
#6>GitHub recently has been
Not just recently, not recently too. Within the last 3 months, maybe longer, it's been miserable.
I've had to add their status to an RSS feed so my dev team knows when they're down, now.
Re: Addressing GitHub’s recent availability issues
#7It's curious that all of the incidents are database related.
One thought: GitHub is basically a bunch of very complicated interconnected databases, no? I mean, to some extent all large web companies are, but GitHub much more than most. Any failure is going to be, in some sense, a database failure.
Another thought: this could speak to a critical weakness in GitHub’s database team. Maybe they hired a lot of bad developers to the database team at the same time, maybe a bad culture took root in the database wing’s management team.
Re: Addressing GitHub’s recent availability issues
#8>GitHub recently has been Not just recently, not recently too. Within the last 3 months, maybe longer, it's been miserable. I've had to add their status to an RSS feed so my dev team knows when they're down, now.
Have you considered a different host or at least mirroring elsewhere?
Re: Addressing GitHub’s recent availability issues
#9It's curious that all of the incidents are database related.
One thought: GitHub is basically a bunch of very complicated interconnected databases, no? I mean, to some extent all large web companies are, but GitHub much more than most. Any failure is going to be, in some sense, a database failure. Another thought: this could speak to a critical weakness in GitHub’s database team. Maybe they hired a lot of bad developers to the database team at the same time, maybe a bad cultur…
Looking at the issue details, I'll guess they have "poor ops practices":
- 9th may: during a deployment, rollback didn't work
- 10th may: a recently deployed function killed the database
- 11th may: failover didn't work
Re: Addressing GitHub’s recent availability issues
#10It's curious that all of the incidents are database related.
Databases are legitimately the worst part of every company.
All the new, and much better tech is so fucking expensive.
CockroachDB has the worst sales people that can’t even demonstrate their own benchmarks.
Spanner is just unaffordable for most.
Postgres/MySQL are great but making them scale for write heavy workloads is a pain. I also hate managing primary/failover systems. We have consensus algorithms now. Let’s just use them.