Does anyone know what could be the cause of this?
LLM code or just terrible management?
Incident with Github.com [resolved]
731–740 of 1001 posts
Re: Incident with Github.com [resolved]
#732Re: Incident with Github.com [resolved]
#733Earlier quoted context omitted.
I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?
If they did, they would’ve paused new feature development a long time ago and redirected that resource towards scaling the platform. And yet they’ve been releasing tonedeaf features more than ever.
Feature work is frankly much easier than scaling and bullet-proofing.
Re: Incident with Github.com [resolved]
#734Earlier quoted context omitted.
That's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.
It is interesting, how much money is being lost during this outage? My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced…
A story as old as time.
Re: Incident with Github.com [resolved]
#735Earlier quoted context omitted.
That's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.
It is interesting, how much money is being lost during this outage? My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced…
Name names.
Re: Incident with Github.com [resolved]
#736Earlier quoted context omitted.
Doubly so given this is hardly a surprise. We've been on this trajectory for at least a couple of years now. They don't get to shrug, point at 10x volume, and act like they've been blindsided.
They had large increase in volume, and trying to move to Azure at the same time. I don't envy them for either work they need to do. But also don't feel pithy because it's Microslop, at the end of the day.
[dead]
Re: Incident with Github.com [resolved]
#737Earlier quoted context omitted.
The thing is Github’s core is git’s core plus extras, and thet core is fundamentally text wrangling. Youtube and Insta and Facebook and many other social media platforms handle very resource intensive media for billions of people plus bots. Increased influx from AI is not plausible when viewed from that lens.
It is though. Maybe you think they should be doing a better job but 12-24xing scale when you were already handling billions of commits is definitely understandable to me. It just does feel like they should have some way of ensuring customers, paying ones, don’t have to suffer so much.
Yes. It seems that at the very least repositories owned by paying customers should be processed on priority threads. And perhaps throttle commits/pushes to repos where there are a lot of pushes from the same person all the time. This can be effectively throttled at the push level.
Re: Incident with Github.com [resolved]
#738Re: Incident with Github.com [resolved]
#739I'm self-hosting Gitea and Woodpecker CI for least 6 years, alongside my own docker and pypi registries. Adding x86 and ARM runners, it costs me circa $30/month to run the whole thing, and I am reasonably sure that I could make the whole thing scale at a lower cost per user than what Github charges. I am seriously, 100%, contemplating the idea of offering consulting for migrating companies and projects out of Github.…
I work at a startup with ~20-30 devs. We're talking about a project like this. I'm not sure weekends would work for an actual company, though maybe it depends. I think there would be an appetite for this. From my POV - the main areas of interest are how you can roll this out to a team gracefully without breaking everyone's workflows for multiple days. Good luck
This is exactly the migration problem we're trying to make boring with Fjord. It shouldn't require a flag day, bring up Forgejo alongside GitHub, mirror while the team moves over, then migrate CI incrementally.
For a 20–30 dev team I'd be happy to help work through what that rollout would actually look like
Re: Incident with Github.com [resolved]
#740heaven forbid I want to commit some code this morning