Live data from Hacker News

GitHub is down again

githubstatus.com

351–360 of 425 posts

Re: GitHub is down again

#351
post #311

Earlier quoted context omitted.

Here it is. It looks like they are down to a single 9 at this point across all services: https://mrshu.github.io/github-statuses/

Can you add a line graph with incidents per month? Would be useful to see if the number of incidents are going up or down over time.

I threw together https://mkantor.github.io/github-incident-timeline/>. It's by day rather than month, and only shows the last 50 incidents since that's all their API returns.

Re: GitHub is down again

#352
post #136

Earlier quoted context omitted.

Is there any reason why Github needs 99.99% uptime? You can continue working with your local repo.

Many teams work exclusively in GitHub (ticketing, boards, workflows, dev builds). People also have entire production build systems on GitHub. There's a lot more than git repo hosting.

It's especially painful for anyone who uses Github actions for CI/CD - maybe the release you just cut never actually got deployed to prod because their internal trigger didn't fire... you need to watch it like a hawk.

Re: GitHub is down again

#353
post #349

Earlier quoted context omitted.

> It looks like they are down to a single 9 at this point across all services That's not at all how you measure uptime. The per area measures are cool but the top bar measuring across all services is silly. I'm unsure what they are targeting, seems across the board it's mostly 99.5+ with the exception of Copilot. Just doing math, 3 (independent, which I'm aware they aren't fully) 99.5 services brings you down to an o…

It depends whether the outages are overlapped or not. If the outages are not overlapped then that is indeed how you do it since some of your services being unavailable means your service is not fully available.

They are overlapped. You can hover over the bars and some bars have multiple issues.

Re: GitHub is down again

#354
post #147
post #136

Earlier quoted context omitted.

Is there any reason why Github needs 99.99% uptime? You can continue working with your local repo.

Are you kidding? I need my code to pass CI, and get reviewed, so I can move on, otherwise the PRs just keep piling. You might as well say the lights could go out, you can do paperwork.

When in doubt - schedule a meeting about how you're unable to do work to keep doing work!

Re: GitHub is down again

#355
post #42

Earlier quoted context omitted.

You're absolutely right! Sorry I deleted your database.

I can help you restore from backups if you will tell me where you backed it up. You did back it up, right? Right before you ran me with `--allow-dangerously-skip-permissions` and gave me full access to your databases and S3 buckets?

I'm very sorry I deleted your `backups` bucket, despite being specifically instructed not to touch the `backups` bucket.

Re: GitHub is down again

#356

Earlier quoted context omitted.

It's 100% because the number of operations happening on Github has likely 100x'd since the introduction of coding agents. They built Github for one kind of scale, and the problem is that they've all of a sudden found themselves with a new kind of scale. That doesn't normally happen to platforms of this size.

A major platform lift and shift does not help. They are always incredibly difficult. There are probably tons of baked in URLs or platform assumptions that are very easy to break during their core migration to Azure.

> A major platform lift and shift does not help.

ISTR that the lift-n-shift started like ... 3 years ago? That much of it was already shifted to Azure ... 2 years ago?

The only thing that changed in the last 1 year (if my above two assertions are correct (which they may not be)) is a much-publicised switch to AI-assisted coding.

Re: GitHub is down again

#357

Earlier quoted context omitted.

I can help you restore from backups if you will tell me where you backed it up. You did back it up, right? Right before you ran me with `--allow-dangerously-skip-permissions` and gave me full access to your databases and S3 buckets?

You're right! Let's just quickly promote your only read replica to the new primar---oops!

Good news: I optimized your infrastructure costs to zero. Bad news: I did it by deleting everything. You're welcome.

Re: GitHub is down again

#358
post #42

Earlier quoted context omitted.

You're absolutely right! Sorry I deleted your database.

I can help you restore from backups if you will tell me where you backed it up. You did back it up, right? Right before you ran me with `--allow-dangerously-skip-permissions` and gave me full access to your databases and S3 buckets?

> I can help you restore from backups if you will tell me where you backed it up.

"Whoops, now that one is nuked too. You have any more backups I can practice my shell commands on?"

Post reply on HN