Live data from Hacker News

Incident with Github.com [resolved]

githubstatus.com

421–430 of 1001 posts

Re: Incident with Github.com [resolved]

#421
post #76

Earlier quoted context omitted.

That's fine, but rate limiting is a solved problem at this point. They need to rate-limit free accounts and be realistic about their limits instead of letting the system fail.

Free (and non-free) accounts are already rate limited. The likely problem is that individual accounts/repos rarely hit those limits, instead the explosive demand is due to a massive growth in the number of small, individual projects being creatd.

The person you were applying to said “rate-limiting” but I think they meant “load-shedding”.

Re: Incident with Github.com [resolved]

#422

We're moving to self hosted Gitea. It's never been easier to do devops with the help of LLMs. I cannot believe how flaky GitHub is. https://git.tinygrad.win/tinygrad/tinygrad

Funny because I spent a couple of years on self-hosted Gitea, including doing some work on Gitea itself to add features we wanted. Then I changed projects and thought "well that was a lot of work, let's just do what everyone does and use GitHub". Now twiddling my thumbs this morning...wondering if I need to resurrect the Gitea deployment.

Re: Incident with Github.com [resolved]

#423

Earlier quoted context omitted.

Luckily we already have a distributed version control system that can take over github: https://fossil-scm.org/home/doc/trunk/www/index.wiki I suppose people can use git too if they ever figure out how to send emails from a dedicated client.

There are plenty of alternatives for git hosting. The last thing which needs to be migrated before we get an exodus off this vibe coded monstrosity is reputation (i.e. github stars) which are inherently sticky.

I think as we see more AI participation in code, github stars are going to matter less.

Re: Incident with Github.com [resolved]

#424

> Update - We are seeing an approximate 20% error rate across numerous experiences including Pull Requests, Issues, and others. Investigations are currently under way and we will be posting updates as they become available brutal

The error rate is 100% on the service that allows you to create PRs in the web site. Actual remote git service is working just fine. I might try the API to make a PR, see if that's working.

Update: yes that worked. gh CLI using the API was able to make a PR while the web interface was inoperable.

Second update: defeat from the jaws of victory -- the page that displays open PRs is also down.

Third update: CI isn't working anyway, so the CLI PR creation workaround doesn't achieve much.

Re: Incident with Github.com [resolved]

#425

We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?" No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it. What's everyone else using? Self hosted Gitlab? Gitea?

blacksmith.sh just raised tens of millions on the core premise that self-hosting CI/CD is hard [0]. they've had multiple outages in the last week, so i guess they are proving themselves correct. in practice, giving ssh/sudo to AI (codex) and telling it to setup ephemeral self-hosted runners on your own host VM (qemu), takes about 30 minutes and works extremely well. it bangs out all the shell scripts. moved my own bu…

They run on top of GH, so if GH goes down, so does Blacksmith, every time

Re: Incident with Github.com [resolved]

#427
post #225

Earlier quoted context omitted.

Huh, they usually do? I'm not a full-time developer though so I'm surely biased towards doing the obvious quick fixes or already-discussed things, but discussing how to solve a problem before throwing something at the wall seems like the normal thing to do for me Or is their point about abandoned projects where the dev never clicks release after merging, or even never merges?

The point is you can't ship shit during an outage.

Yeah I saw the humor of that :) Just curious about the thing they're trying to even sell, it doesn't sound like something they can offer even if they had a good SLA

Re: Incident with Github.com [resolved]

#428

Earlier quoted context omitted.

Those are both hilarious to me. In my last FTE role we had the exact opposite beliefs - we wanted to leave GitLabs increases in pricing and move to annual sub, but the reliability of self-hosted runners and the flexibility (and clarity) of the CI system was fantastic.

> Those are both hilarious to me. Why is it hilarious to believe that the industry leader might be better at something than doing it yourself?

[deleted]

Re: Incident with Github.com [resolved]

#429
post #225

Funny enough, I received a marketing email from GitHub during the outage with the following subject: "What if every pull request you started actually shipped"

Huh, they usually do? I'm not a full-time developer though so I'm surely biased towards doing the obvious quick fixes or already-discussed things, but discussing how to solve a problem before throwing something at the wall seems like the normal thing to do for me Or is their point about abandoned projects where the dev never clicks release after merging, or even never merges?

The joke was that we have open pull requests that we cannot merge because of this downtime.

Re: Incident with Github.com [resolved]

#430
post #29

I had a lot of goodwill for GitHub but I think today is the tipping point. Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again. The hope is dead.

I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?

They've had an insane influx. It seems that they should more aggressively limit free tier activity or isolate paid users, this has become unbearable...
Post reply on HN