Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Github.com [resolved]

githubstatus.com

Post reply on HN
Print view View original

691–700 of 1001 posts

Re: Incident with Github.com [resolved]

#691

Post by cvburgess »

cebert wrote: post #688

GitHub can’t focus on the basics like a stable platform. However, when they release stack commits a few weeks ago somebody had time to make the hamburger menu an unprofessional stack of pancakes…cute.

While i get the sentiment and the frustration, I highly doubt an SRE worked on the hamburger menu.

In my experience some teams move a lot faster than others and that leads to asymmetric quality. If all devs could pick up a new language and skillset overnigh then companies would have a lot more flexibility in their roadmaps, but humans don't scale that way - with or without AI.

Re: Incident with Github.com [resolved]

#692

Post by mhitza »

unknownfuture wrote: post #653
pera wrote: post #573

Earlier quoted context omitted.

Save that for your local mom-and-pop store: Microsoft is a multi-billion dollar corporation with enough resources to, at the very least, provide a reliable service for enterprise clients. The "AI is using a lot of resources" excuse was maybe acceptable last year but not in Q3 2026.

Doubly so given this is hardly a surprise. We've been on this trajectory for at least a couple of years now. They don't get to shrug, point at 10x volume, and act like they've been blindsided.

They had large increase in volume, and trying to move to Azure at the same time. I don't envy them for either work they need to do. But also don't feel pithy because it's Microslop, at the end of the day.

Re: Incident with Github.com [resolved]

#693

Post by memonkey »

ChoGGi wrote: post #673
jubilanti wrote: post #637

Almost 3 hours in and still "We are still working to identify the root cause" Can't even view diffs in the web interface! I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issue…

No idea about some of that stuff, as I have more basic needs; nearlyfreespeech.net is cheap and stays out of my way.

wow totally forgot about nearlyfreespeech. so glad they're still around. wonderful host.

Re: Incident with Github.com [resolved]

#694

Post by Groxx »

krs_ wrote: post #650
mrighele wrote: post #409

Earlier quoted context omitted.

> scaling problems due to the influx of LLM assisted coding You mean people pushing LLM assisted coding, or github's own coders pushing LLM assisted code to production ? :-). I am more inclined to think that it is the second, but if the reason is the first, what is particular hard to scale ? It's not like everybody is vide-coding into a single repository, so I would expect that most of the load would naturally be sca…

> You mean people pushing LLM assisted coding, or github's own coders pushing LLM assisted code to production ? :-). Why not both?

All three: their own quite aggressive pushing of LLM coding as a whole.

Re: Incident with Github.com [resolved]

#695

Post by roastedfunction »

nullify88 wrote: post #600
946789987649 wrote: post #576

Earlier quoted context omitted.

Are you sure? We have enterprise and I couldn't even access our repo briefly.

We have data residency in the EU and didn't notice anything.

GHE.com has only been around since November 2024. Most GHEC customers I imagine are still stuck on the shared global infrastructure on github.com.

Re: Incident with Github.com [resolved]

#696

Post by akkad33 »

unknownfuture wrote: post #685
tom1337 wrote: post #677

Earlier quoted context omitted.

Update 16:36 UTC - We identified the problematic component and have taken corrective actions. There are strong signs of recovery but we are still working to completely restore service, with error rates still remaining slightly elevated. We will post further updates as recovery continues. just shy of 3 hours to find the issue

Weird, I've been told you can just throw AI at all your logs and metrics and it can magically solve all the things...

Maybe they are dedicated hand coders!

Re: Incident with Github.com [resolved]

#697

Post by adamddev1 »

jubilanti wrote: post #637

Almost 3 hours in and still "We are still working to identify the root cause" Can't even view diffs in the web interface! I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issue…

I was in a very similar situation and this is what I did:

- put Forgejo on a little $5 Vultr VPS (I share the VPS with other things, Forgejo doesn't need much resources. I also added some block storage to host my Docker containers in the Forgejo Docker registry)

- got a 8GB ram cheaper Hetzner VPS (you have to wait for their availability) for around 8 EUR and used that to install the CI runner

- Use Netlify for hosting static sites and PWA

It takes a bit to set up but everything works snappy, clean, and flawlessly. I have unlimited CI minutes. As a bonus I get free Docker image and registry hosting because it's included with Foregejo. I am so, so happy with it all.

Also with Forgejo you just feel like you're using something clean and solid. GitHub started to feel more and more bloated, slow, and unstable.

Re: Incident with Github.com [resolved]

#698

Post by rurban »

tom1337 wrote: post #677
jubilanti wrote: post #637

Almost 3 hours in and still "We are still working to identify the root cause" Can't even view diffs in the web interface! I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issue…

Update 16:36 UTC - We identified the problematic component and have taken corrective actions. There are strong signs of recovery but we are still working to completely restore service, with error rates still remaining slightly elevated. We will post further updates as recovery continues. just shy of 3 hours to find the issue

So just 3 more hours to fix and restart and process the queues

Re: Incident with Github.com [resolved]

#699

Post by adamddev1 »

tom1337 wrote: post #677
jubilanti wrote: post #637

Almost 3 hours in and still "We are still working to identify the root cause" Can't even view diffs in the web interface! I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issue…

Update 16:36 UTC - We identified the problematic component and have taken corrective actions. There are strong signs of recovery but we are still working to completely restore service, with error rates still remaining slightly elevated. We will post further updates as recovery continues. just shy of 3 hours to find the issue

Furiously prompting Claude, "please, please fix this!!!!"

Re: Incident with Github.com [resolved]

#700

Post by rurban »

unknownfuture wrote: post #685
tom1337 wrote: post #677

Earlier quoted context omitted.

Update 16:36 UTC - We identified the problematic component and have taken corrective actions. There are strong signs of recovery but we are still working to completely restore service, with error rates still remaining slightly elevated. We will post further updates as recovery continues. just shy of 3 hours to find the issue

Weird, I've been told you can just throw AI at all your logs and metrics and it can magically solve all the things...

My blackest Monday today. Deepseek is off/too expensive, Claude tokens out for 3 days, kimi refuses, and Github down for the whole afternoon. Maybe I'll take the week off
Post reply on HN