Live data from Hacker News

GitHub having issues [resolved]

githubstatus.com

111–120 of 150 posts

Re: GitHub having issues [resolved]

#112

Earlier quoted context omitted.

Except actually it was, that was what my git client was reporting trying to run a pull.

I'm going to trust the constant stream of updates from the company itself which shows exactly what went down and came back up rather than a random anecdote.

This seems intelligent, after all companies are incapable of making errors in reporting and also have absolutely no incentive to lie about stuff like that. Those 500 errors others have reported as experiencing must have just been the wind.

Re: GitHub having issues [resolved]

#113
I am getting really tired of github. outages happen that's a given. but on so much stuff they don't even care or try. Github is becoming the bottleneck in my agentic coding workflows. unless I make Claude do it intelligently, I hit rate limits checking on CI jobs (5000 api requests in an hour). Depot makes their CI so much better, but it is still tied to github in a couple of annoying places.

PRs are a defacto communication and coordination bus between different code review tools, its all a mess.

LLMs make it worse because I'm pushing more code to github than ever before, and it just isn't setup to deal with this type of workload when it is working well.

Re: GitHub having issues [resolved]

#114
post #97

A directory over SSH can be your git server. If your CI isn't too complex, a post-receive hook looping into Docker can be enough. I wrote up about self hosting git and builds a few weeks ago[1]. There are heavier solutions, but even setting something like this up as a backstop might be useful. If your blog is being hammered by ChatGPT traffic, spare a thought for Github. I can only imagine their traffic has ballooned…

Doesn't post-receive block the push operation and get cancelled when you cancel the push?

Re: GitHub having issues [resolved]

#115

Insert the standard comment about how git doesn't even need a hub. The whole point of it is that it's distributed and doesn't need to be "hosted" anywhere. You can push or pull from any repo on anyone's machine. Shouldn't everyone just treat GitHub as an online backup? Zero reason it being down should block development.

The problem is that any kind of automatic code change process like CI, PRs, code review, deployments, etc etc are based on having a central git server. Even security may be based on SSO roles synced to GH allowing access to certain repos.

A self-hosted git server is trivial. Making sure everything built on top of that is able to fallback to that is not. Especially when GH has so many integrations out of the box

Re: GitHub having issues [resolved]

#116
post #97

A directory over SSH can be your git server. If your CI isn't too complex, a post-receive hook looping into Docker can be enough. I wrote up about self hosting git and builds a few weeks ago[1]. There are heavier solutions, but even setting something like this up as a backstop might be useful. If your blog is being hammered by ChatGPT traffic, spare a thought for Github. I can only imagine their traffic has ballooned…

Doesn't post-receive block the push operation and get cancelled when you cancel the push?

It does, you're just running a command over ssh, so if you've a particularly long build then something more involved may make more sense.

Re: GitHub having issues [resolved]

#117
post #47

In moments like this, it's useful to have a "break glass" mode in your CI tooling: a way to run a production CI pipeline from scratch, when your production CI infrastructure is down. Otherwise, if your CI downtime coincides with other production downtime, you might find yourself with a "bricked" platform. I've seen it happen and it is not fun. It can be a pain to setup a break-glass, especially if you have a lot of l…

It’s a hard sell. I always get blank looks when I suggest it, and often have to work off book to get us there.

I generally recommend that the break glass solution always be pair programmed.

Re: GitHub having issues [resolved]

#118
post #47

In moments like this, it's useful to have a "break glass" mode in your CI tooling: a way to run a production CI pipeline from scratch, when your production CI infrastructure is down. Otherwise, if your CI downtime coincides with other production downtime, you might find yourself with a "bricked" platform. I've seen it happen and it is not fun. It can be a pain to setup a break-glass, especially if you have a lot of l…

100%. We used to design the pipeline a way that is easily reproducible locally, e.g. doesn’t rely on plugins of the CI runtime. Think build.sh shell script, normally invoked by CI runner but just as easy to run locally.

My automation is always an escalation of a run book that has gotten very precise and handles corner cases.

Even if I get the idea of an automation before there’s a run book for it.

Re: GitHub having issues [resolved]

#119

Insert the standard comment about how git doesn't even need a hub. The whole point of it is that it's distributed and doesn't need to be "hosted" anywhere. You can push or pull from any repo on anyone's machine. Shouldn't everyone just treat GitHub as an online backup? Zero reason it being down should block development.

The problem is that any kind of automatic code change process like CI, PRs, code review, deployments, etc etc are based on having a central git server. Even security may be based on SSO roles synced to GH allowing access to certain repos. A self-hosted git server is trivial. Making sure everything built on top of that is able to fallback to that is not. Especially when GH has so many integrations out of the box

Forgejo has all of the features you mentioned and is completely open source!

Re: GitHub having issues [resolved]

#120
post #116

Earlier quoted context omitted.

Doesn't post-receive block the push operation and get cancelled when you cancel the push?

It does, you're just running a command over ssh, so if you've a particularly long build then something more involved may make more sense.

Most builds take a long time, at least in C++ and Rust (the two languages I work in). And from what I have seen of people working in Python, the builds aren't fast there either (far faster of course, but still easily a minute or two).

Also, how would PRs and code review be handled?

Your suggestion really only makes sense for a small single developer hobby project in an interpreted language. Which, if that is what you intended, fair enough. But there really wasn't enough context to ascertain that.

Post reply on HN