Live data from Hacker News

Incident with Github.com [resolved]

githubstatus.com

521–530 of 1001 posts

Re: Incident with Github.com [resolved]

#521
post #466

I wonder if you could replace the whole github issues feature with just a directory within each git repo named "Issues/". Within that, have another couple directories, "open/", and "closed/". Within "open", have a directory per issue, e.g. "such-and-such-does-not-work/". Within each issue directory, have "issue.txt", or "issue.md", which takes the place of the page for each issue on github. You could store other thin…

Check out Haxxy, it has a lot of fresh ideas related to issues/pr management https://youtu.be/iBDUTaKWVdU

I don't think a YouTube link is the best way to evangelize a new tool.

Re: Incident with Github.com [resolved]

#522
post #413

I'm self-hosting Gitea and Woodpecker CI for least 6 years, alongside my own docker and pypi registries. Adding x86 and ARM runners, it costs me circa $30/month to run the whole thing, and I am reasonably sure that I could make the whole thing scale at a lower cost per user than what Github charges. I am seriously, 100%, contemplating the idea of offering consulting for migrating companies and projects out of Github.…

I work at a startup with ~20-30 devs. We're talking about a project like this. I'm not sure weekends would work for an actual company, though maybe it depends. I think there would be an appetite for this. From my POV - the main areas of interest are how you can roll this out to a team gracefully without breaking everyone's workflows for multiple days. Good luck

The most natural way to roll this out would be to have a "blue/green" system, where both systems are running and the weekends would be the time to take the steps towards the migration. First set up the servers, integrate Auth/SSO (depending on the current company set up), etc. That could be done probably in a single weekend.

Once this is done, the team would have two possible git remotes to work with. Given that gitea also allows mirroring with Github, the team would be able to even switch to Gitea as the "main" git remote and for PR reviews even if the CI part is still on Github.

The harder part would be migrating the CI jobs. But this also can be done in parallel and no one needs to cut out their Github actions immediately. Start with the simplest of the stages, e.g, the automated test suite, and once you get the whole pipeline from Woodpecker on an staging environment, you'd could take another weekend to test the whole workflow. Once the first project is successfully migrated, it becomes a matter of repeating the process.

Woodpecker is still a bit finicky and debugging it can be a bit annoying, but given that it's all based on docker it is a lot easier to debug a pipeline locally.

Re: Incident with Github.com [resolved]

#523
The exponential growth they are having right now is absolutely insane. I would caution against assuming its an easy problem, or that anyone else has demonstrated an ability to handle this level of scale/growth any better.

In May they had to solve for 30x growth overnight, and I would guess its closer to 100x by now. The reality is a centralized solution like this may no longer be a viable option with the scale of code generation.

Re: Incident with Github.com [resolved]

#525

Earlier quoted context omitted.

It is interesting, how much money is being lost during this outage? My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced…

Any engineered system where an individual can accidentally cause a 7 figure outage is poorly designed. And engineering leadership that decider to terminate an individual due to such failure (as long as there was no malicious actions) is completely clueless.

Captain obvious over here

Re: Incident with Github.com [resolved]

#527

It is not working at all, I can not load any repo pages. Github.com's promise is that it can be the central broker of open source code because it is reliable. That promise hasn't been kept recently. That said, Github is hard to displace and it is similar to the era of the Twitter Fail Whales. It was a sign of growth that couldn't be properly managed but there was not viable alternative.

> Github is hard to displace and it is similar to the era of the Twitter Fail Whales Hard but not impossible. Being unreliable might nudge a few people. Also, Twitter had their fail whale in their early phase (mostly), not twenty years or so into their existence. The issue with replacing Github is that it takes more than version control and issue tracking to replace the network that is Github. It's effectively the la…

> throw out the baby with the bathwater by focusing on self hosting without attempt at networking with other instances

May I ask what networking with other instances brings? Git is decentralised, so it works with self-hosted instances, right?

Re: Incident with Github.com [resolved]

#528

Cannot even merge code, this is the limit for me. We are prioritising migration to a different platform and this time going to decouple CI - right now we have too many points of failure on one vendor.

Apologies if you already tried this, but I have been able to merge and get some things done via command line.

e.g.: gh pr merge 5062 -R --squash --delete-branch

Re: Incident with Github.com [resolved]

#529

June 17, 2023: Dohmke says Copilot will write 80% of code: https://www.freethink.com/robots-ai/github-copilot Slop code is written and pushed ... Aug 11, 2025: Dohmke leaves the sinking ship: https://www.entrepreneur.com/business-news/github-ceo-thomas... This is a true AI victory.

Do y'all have any evidence that github's recent outages are caused by slopcode rather than the significant increase in load, which is exactly what Github & MS have been claiming as the root cause?

I'm tired of these BS stories being pushed by green accounts on HN like there's some damn agenda by someone who thinks their job will be saved "if only the CEOs realized that Slopcode Bad".

Re: Incident with Github.com [resolved]

#530
post #362

Help encourage alternatives by trying Gitlab.com or Codeberg.org or Tangled.org or others, and please star some projects there to help motivate more options. Self-promo links below if you would like to star a few of mine good for developers. https://gitlab.com/architecture-decision-record https://gitlab.com/ways-of-working https://gitlab.com/coordinated-vulnerability-disclosure

Codeberg isn't a viable infrastructure alternative. Its an opinionated community not a neutral host as they will delete repos based off subjective opinions on LLM usage and resource consumption. If continued hosting is dependent on moderators deciding if a project fits whatever the current philosophy is, and the whims of the moderation team, then I cannot consider it a serious choice for infrastructure. Infrastructur…

[flagged]
Post reply on HN