Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub having issues [resolved]

githubstatus.com

Post reply on HN
Print view View original

121–130 of 150 posts

Re: GitHub having issues [resolved]

#121

Post by qudat »

arianvanp wrote: post #114
duggan wrote: post #97

A directory over SSH can be your git server. If your CI isn't too complex, a post-receive hook looping into Docker can be enough. I wrote up about self hosting git and builds a few weeks ago[1]. There are heavier solutions, but even setting something like this up as a backstop might be useful. If your blog is being hammered by ChatGPT traffic, spare a thought for Github. I can only imagine their traffic has ballooned…

Doesn't post-receive block the push operation and get cancelled when you cancel the push?

I use https://pipe.pico.sh for this use case. It’s a pubsub over ssh. It’s multicast so you can have multiple listeners on the same topic, and you can have it block or not block the event.

Re: GitHub having issues [resolved]

#123

Post by VorpalWay »

shykes wrote: post #47

In moments like this, it's useful to have a "break glass" mode in your CI tooling: a way to run a production CI pipeline from scratch, when your production CI infrastructure is down. Otherwise, if your CI downtime coincides with other production downtime, you might find yourself with a "bricked" platform. I've seen it happen and it is not fun. It can be a pain to setup a break-glass, especially if you have a lot of l…

At times like this is when I'm so happy I don't work with deploying to a production environment, but rather we release software that (after extensive qualification), customers can install in their environment on their airgapped networks. Using a USB stick to cross the air gap. If we miss a release by a day or thrre, there is enough slack in the process before it goes to the customer that no one will be any the wiser.

Crazy in 2026, but installable software has some pros still, for both the developer and for the customer. And I would personally love if I could do things that way for more things.

Re: GitHub having issues [resolved]

#124

Post by duggan »

VorpalWay wrote: post #120
duggan wrote: post #116

Earlier quoted context omitted.

It does, you're just running a command over ssh, so if you've a particularly long build then something more involved may make more sense.

Most builds take a long time, at least in C++ and Rust (the two languages I work in). And from what I have seen of people working in Python, the builds aren't fast there either (far faster of course, but still easily a minute or two). Also, how would PRs and code review be handled? Your suggestion really only makes sense for a small single developer hobby project in an interpreted language. Which, if that is what you…

I did give additional context in the blog post I linked, but yes, to be clear, this is something that will really work best for small projects with reasonably fast build cycles.

If you're already at the point where you're fielding pull requests, lots of long running tests, etc., you'll probably already know you need more than git over ssh.

Re: GitHub having issues [resolved]

#125

Post by lelanthran »

rezonant wrote: post #50
risyachka wrote: post #43

Earlier quoted context omitted.

and we all know why

Because they're moving it to Azure and doing it far too quickly, not taking care to avoid availability issues

Could be.

Or could be that the recent 12 months of 100x increase in code and activity is more than they had planned for when they last did capacity planning.

Vibe-coders, many of them here, often boast about the insane amount of KLoC/hour they can generate and merge.

Re: GitHub having issues [resolved]

#126

Post by lelanthran »

paddy_m wrote: post #113

I am getting really tired of github. outages happen that's a given. but on so much stuff they don't even care or try. Github is becoming the bottleneck in my agentic coding workflows. unless I make Claude do it intelligently, I hit rate limits checking on CI jobs (5000 api requests in an hour). Depot makes their CI so much better, but it is still tied to github in a couple of annoying places. PRs are a defacto commun…

> I am getting really tired of github. outages happen that's a given. but on so much stuff they don't even care or try. Github is becoming the bottleneck in my agentic coding workflows. unless I make Claude do it intelligently, I hit rate limits checking on CI jobs (5000 api requests in an hour). Depot makes their CI so much better, but it is still tied to github in a couple of annoying places.

Have you ever considered that this is the problem? GH never planned for this sort of pointless and unpaid activity before. Now they have a large increase (I've seen figures of 100x) in activity and they can't keep up.

It doesn't help that almost none of the added activity is actually useful; it's just thousands and thousands of clones of some other pointless product.

Re: GitHub having issues [resolved]

#127

Post by dzonga »

I remember the days when it was mostly Gitlab having issues.

Github was super stable - then it got shitty once they switched to React on the frontend instead of the server rendered pages, then Co-pilot stuff

lately I haven't heard them bragging about the Rails Monolith

Re: GitHub having issues [resolved]

#128

Post by bubblewand »

rezonant wrote: post #50
risyachka wrote: post #43

Earlier quoted context omitted.

and we all know why

Because they're moving it to Azure and doing it far too quickly, not taking care to avoid availability issues

There may be other problems but as someone who's somehow ended up integrating Git into a service twice in my career without even trying that hard to find a reason (it turns out it's weirdly handy in quite a few situations, god I wish it were implemented as a library and not a pile of Perl and shit, and yes I know about libgit2) and has looked into some of Git's and Gitlab's posts about their architectures over the years though the lens of having fought a few of the same beasts, an Azure migration was very obviously going to make things worse.

Re: GitHub having issues [resolved]

#129

Post by noplacelikehome »

nadirollo wrote: post #91
shykes wrote: post #47

In moments like this, it's useful to have a "break glass" mode in your CI tooling: a way to run a production CI pipeline from scratch, when your production CI infrastructure is down. Otherwise, if your CI downtime coincides with other production downtime, you might find yourself with a "bricked" platform. I've seen it happen and it is not fun. It can be a pain to setup a break-glass, especially if you have a lot of l…

This is a must when your systems deal with critical workloads. At Fastly, we process a good chunk of the internet's traffic and can't afford to be "down" while waiting for the CI system to recover in the event of a production outage. We built a CI platform using dagger.io on top of GH Actions, and the "break glass" pattern was not an afterthought; it was a requirement (and one of the main reasons we chose dagger as t…

I would really love to hear more about this, but my cursory search didn't find a write up about it.

I did a PoC of Dagger for an integration and delivery workload and loved the local development experience. Being able to define complex pipelines as a series of composable actions in a language which can be type checked was a great experience, and assembling these into unix-style pipelines felt very natural.

I struggled to go beyond this and into an integration environment, though. Dagger's current caching implementation is very much built around there being a single long-lived node and doesn't scale out well, at least without the undocumented experimental OCI caching implementation. Are you able to share any details on how Fastly operates Dagger?

Re: GitHub having issues [resolved]

#130

Post by TacticalCoder »

duggan wrote: post #97

A directory over SSH can be your git server. If your CI isn't too complex, a post-receive hook looping into Docker can be enough. I wrote up about self hosting git and builds a few weeks ago[1]. There are heavier solutions, but even setting something like this up as a backstop might be useful. If your blog is being hammered by ChatGPT traffic, spare a thought for Github. I can only imagine their traffic has ballooned…

> The origin of a git repo is more or less just the contents of the .git directory in a remote location. That's it. You don't even need to run a git server if you're happy enough using ssh for transport.

Yeah. You probably do want to make sure you turn your .git/ into a "bare" git repository but that's basically it.

And it's what I do too: an OCI container that gives me access to all my private Git repos (it sets up SSH with U2F so I get to use my Yubikey to push/pull from various machines to those Git repos).

Post reply on HN