Live data from Hacker News

Incident with Github.com [resolved]

githubstatus.com

711–720 of 1001 posts

Re: Incident with Github.com [resolved]

#711
The most shocking thing to me about the entire GitHub fiasco is: No matter how bad the uptime is, no matter how bad the UI is. We will keep using the software / service we depend on.

Why do we as software engineers have set fault tolerance and high availability as the primary goal of any software / infrastructure architecture? This shows it's completely irrelevant.

Re: Incident with Github.com [resolved]

#712

Earlier quoted context omitted.

If you're seriously looking at Gitea/Forgejo, I run Fjord, dedicated managed Forgejo, so you don't have to operate it yourself. We can help with migration and runners too. We also run a free public Forgejo instance if you just want to kick the tires first.

Thank you for your work on Fjord! We're currently evaluating it vs. self-hosting, the comparison at https://fjord.sh/compare has been very useful

That's really good to hear, thank you. If you run into anything that's unclear while comparing the two, I'd genuinely like to hear it. Happy to answer questions here too.

Re: Incident with Github.com [resolved]

#713
post #129

Earlier quoted context omitted.

I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?

I have little sympathy that they're buckling over a problem they helped create.

Yeah let's not forget that in those brief few days when Sam Altman was fired, Satya Nadella swooped in and hired him. Despite the failure of that move, Microsoft has invested $13 billion directly into OpenAI, and has spent $100 billion on the partnership. No sympathy from me watching their software crumble as a result.

Re: Incident with Github.com [resolved]

#714

Earlier quoted context omitted.

Update 16:36 UTC - We identified the problematic component and have taken corrective actions. There are strong signs of recovery but we are still working to completely restore service, with error rates still remaining slightly elevated. We will post further updates as recovery continues. just shy of 3 hours to find the issue

Weird, I've been told you can just throw AI at all your logs and metrics and it can magically solve all the things...

Well, it would be a mistake on your part to assume that everything in an organization, especially a large one like github, is a reflection of the capabilities of the state of the art in anything (AI, architecture, programming language, organizational management, operations, etc.).

It's clear that updating the status page as quickly as possible isn't even something github cares about.

Re: Incident with Github.com [resolved]

#715

Earlier quoted context omitted.

Personally I never engaged in the community aspect of GitHub. In my eyes it would be a ginormous gain to our colective mental health to not have the expectation to contribute to open source projects for free, or to have "green squares" every day on a public GitHub profile.

This seems to be such a non issue to me, I always ignored this and its strange to hear others feel compelled.

That's the point of it though. It's designed to be gamified and addictive to coerce users into spending more time Engaging with the site.

Great that it doesn't work on you, but it works really well on a depressingly large number of people.

Re: Incident with Github.com [resolved]

#716
post #558

Earlier quoted context omitted.

GitHub being ruined after the AI hype is a shallow take imo. The root problem isn't AI, Microsoft, or Azure; It's why people keep _trusting_ centralized infra. GitLab and Codeberg can absolutely become trash, each in their own way. The solution isn't just "migrating off of GitHub"; it's "migrating to a platform that is unownable by design, not only hope". And there are no good enough decentralized alternatives to Git…

What are the serious holes?

The main hole is that ATProto's data model is a bit awkward for a code forge.

Issues/PRs/comments are records owned by their authors, not by the repo. So an issue can span N users' PDSes. That's a neat model for social media, but a forge wants the repo to be the durable owner/context of its issues and history because that's where the comment/issue/etc. logically belongs.

The solution? tangled.org acts as a super advanced cache/aggregator for that. What can possibly go wrong?

Re: Incident with Github.com [resolved]

#717
post #527

Earlier quoted context omitted.

> Github is hard to displace and it is similar to the era of the Twitter Fail Whales Hard but not impossible. Being unreliable might nudge a few people. Also, Twitter had their fail whale in their early phase (mostly), not twenty years or so into their existence. The issue with replacing Github is that it takes more than version control and issue tracking to replace the network that is Github. It's effectively the la…

> throw out the baby with the bathwater by focusing on self hosting without attempt at networking with other instances May I ask what networking with other instances brings? Git is decentralised, so it works with self-hosted instances, right?

Git is decentralized in the sense that you can clone your repository wherever. But there is no real networked community on top. The first versions actually piggy backed on the social network that is email. Linux still is managed via email. It's social network was mailing lists. By decentralizing, they could avoid the pesky topic of having to give each other read write access to each other's git servers.

But if you want more than what email can offer, social networking is a thing with Github and it's a big reason for its success. It added a friendly UI and a way for users to engage with each other via issue trackers, pull requests, and all the rest. It has tens of millions of users that have accounts that each can each create issues, pull requests, fork each other's projects, etc. These people collaborate with each other on many millions of projects. The network effect of all these people being in one place where they can collaborate is vital to a lot of open source projects.

It's effectively the largest developer centric social network around. If the open source community fragments over a gazillion self hosted servers that each require a signup that don't really federate any signins and that don't allow aggregating notifications from all the projects you contributed to, etc. it would be a big loss for the open source world.

Tangled by making use of the atproto replicates much of this functionality. While also allowing people to run their own nodes and while allowing users and their projects to be moved between those nodes. That makes it the best of both worlds. You can self host it but without completely disconnecting from the community.

Re: Incident with Github.com [resolved]

#718
post #679
post #211

Earlier quoted context omitted.

Because any price at all will immediately cause users to shift to another platform, and GitHub's value is that it is _the_ place to put your code on the internet.

The network effects isn’t that much, and like what we see with Steam, you want to be where the audience is. More realistically, MAU or something is a metric/OKR. It’s easy to imagine why a business wouldn’t want to cap that.

Someone did a study that suggested the network effect is logarithmic in nature, but if that’s true it means something with 95% market share is 3 times “better” for having everyone there. Not 20 or two orders of magnitude better but not nothing.

Re: Incident with Github.com [resolved]

#719

Almost 3 hours in and still "We are still working to identify the root cause" Can't even view diffs in the web interface! I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issue…

I was in a very similar situation and this is what I did: - put Forgejo on a little $5 Vultr VPS (I share the VPS with other things, Forgejo doesn't need much resources. I also added some block storage to host my Docker containers in the Forgejo Docker registry) - got a 8GB ram cheaper Hetzner VPS (you have to wait for their availability) for around 8 EUR and used that to install the CI runner - Use Netlify for hosti…

I've been very happy with forgejo for a personal, private server.

I don't care much about the social aspects of GitHub, but I do care a ton about hosting code for long periods of time, long after I've completely changed my personal homelab.

GitHub's most essential service for me is it being the equivalent of the Internet Archive, but for code.

Re: Incident with Github.com [resolved]

#720
post #211

I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.

Because any price at all will immediately cause users to shift to another platform, and GitHub's value is that it is _the_ place to put your code on the internet.

And they can keep training their own models off of the code, and on private repos too.

AI makes investors/shareholders go brrr.

Post reply on HN