Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was having issues

githubstatus.com

Post reply on HN
Print view View original

111–120 of 231 posts

Re: GitHub was having issues

#111

Post by jewel »

I used to work at a place that had a second copy of all the git repositories on a server, available over ssh. We could push there and then, at deploy time, instruct the servers to pull from that repo instead of bitbucket.

If I were to set up the same thing again today, I'd add some automation to automatically keep it in sync with github as well as automate the servers so that they'd attempt to pull from both on every deploy.

I share this as an idea for those who need an emergency or backup deploy mechanism.

Re: GitHub was having issues

#112

Post by lucb1e »

petcat wrote: post #74
taude wrote: post #68

git was designed as a distributed vcs for high-latency connected developers with plenty of ability to work offline. I don't think I've really been impacted by any of the outages. Maybe I wait an extra hour to merge a feature or something, in which case I actually get to eat lunch and browse HN, doesn't feel quite as catastrophic for me, as some of you.

The problem is that people design their entire development and release lifecyle to be dependent on Github. A lot of times they can't even push code hotfixes to production without it. It's a terrible SPOF for a lot of engineering orgs.

We also started having customers since a few years that declare GitHub fully trusted, as in, it is simply not worth considering what the impact would be if that vendor gets compromised. I can't name names, but this includes a vendor that aims to prevent supply chain attacks (technically language-agnostic; in practice aiming to be the solution chosen by one of the biggest programming language's package manager)

> can't even push code hotfixes to production without it. It's a terrible SPOF

GitHub's availability impact is the least of my concerns these days. It'll be a really tough year for society worldwide if we need to rebuild loads of infrastructure after some threat actor got into github and managed to change key pieces of code without being detected a couple of years. Having seen how hospitals handle updates, they might get lucky and be old enough to not be affected yet, or have a really tough time recovering due to understaffed IT

No clue how to even begin solving this since our OSes are likely all pulling dependencies from GitHub without verification of the developer's PGP key, if the project even has that and applies it correctly. I guess I can only recommend being aware of the problem and doing what you can in your own organization to reduce the impact

Re: GitHub was having issues

#113

Post by Balinares »

chaz6 wrote: post #80
sashank_1509 wrote: post #76

I’m at Google, we have a million + file codebase. Every piece of code is snapshotted (no need to commit, automatic snapshot whenever a change is made). Every line of code has its own unique URL (this is for your branch too, not just overall). Background tests running nonstop hourly. Never seen any downtime. The infra at Google is insane

Without manual commits, is there any way to record the intention of each change?

There are manual commits, they're just called changelists.

Re: GitHub was having issues

#114

Post by zamalek »

jeltz wrote: post #97
homebrewer wrote: post #77

Earlier quoted context omitted.

Depends on your needs. Last time I checked, Gitlab wanted money for e.g. assigning multiple PR reviewers, which is available in gitea/forgejo. The real issue with gitea/forgejo compared to Gitlab is their terrible CI, which is (to some approximation) a clone of GitHub Actions, also a dumpster fire for those of us proficient with/preferring the UNIX command line. You'll probably need a separate CI runner, like Woodpec…

Gitlab CI is so much better than Github Actions. With some additional improvements Gitlab CI could become amazing.

CI is one area that it's "lacking." I quote that because, honestly, all the bells and whistles in these CI yamls are starting to hurt. Woodpecker[1] (what forgejo uses) is strikingly simple.

[1]: https://woodpecker-ci.org/

Re: GitHub was having issues

#115

Post by tracker1 »

gazook89 wrote: post #87
bob1029 wrote: post #75

I've been working on a self-hosted alternative to GitHub and I am curious what HN finds to be the most important features. I think Code, Issues and PRs are the critical aspects, but I don't know what typical workflows look like for others these days. It seems like there are some teams that have figured out a way to turn GH into a labyrinth of CI/CD actions that allegedly produces a ton of business value, but I don't…

Any time I see this topic brought up, two things are always mentioned: the "hub" part, meaning the discoverability and social aspect, and the "network effect" of having everyone use a single service (so everyone already has an account and they don't have to create additional account for every self-hosted project.

Agreed, it's definitely the network cohesion that keeps GH together. Especially for FLOSS. For advanced features, there are some niceties that say Azure DevOps offers that GH Enterprise still lacks, though it feels like there's some convergence on the backend.

I like GH Actions myself, though sometimes it can get a little cumbersome with self-hosted workers on private/enterprise projects. I'm a big fan of automation and like to keep related bits closer together. As opposed to alternatives that have completely detached workflows/deployments.

Re: GitHub was having issues

#116

Post by colechristensen »

cddotdotslash wrote: post #91
everfrustrated wrote: post #42

Enterprise customers, remember to email your sales rep and ask for them to report on their contracted uptime with you that you are allowed to do as per contract. They wont do this unless you ask hoping you don't notice the outages. It creates lots of internal pain - they have no automation internally for reporting on this. This is the only way anything will ever change. GitHub is _easily_ the most unreliable SaaS pro…

Companies should automate this. Write their own outage monitoring, feed the results, plus the cumbersome format you have to send to the provider, into an LLM, have it spit out an email requesting SLA credits or whatever the contract specifies. Probably not worth it for low cost services, but if you’re paying GitHub $x millions per year, maybe it is.

They intentionally underreport outages. Everybody does. When your performance metrics for your customers, managers, and individual contributors all include uptime, what you get isn't better uptime but lies about uptime.

Re: GitHub was having issues

#117

Post by kevingadd »

awestroke wrote: post #29
evilmonkey19 wrote: post #26

I'm a fan of github but lately i'm seeing a lot of issues like these... Also they don't have yet support for IPv6 (surprising).

The refusal to support IPv6 is embarrassing at this point

Had to buy an IPv4 address for a VPS the other day in order to clone some git repositories. Couldn't believe it. Costing their customers money when they should be able to support v6 by now.

Re: GitHub was having issues

#118

Post by colechristensen »

CoderJoshDK wrote: post #84

With continued reliability issues and the CEO stepping down, now feels like such an opportune time for a competitor to start taking market share. I sure am rooting for it! For the longest time, I thought that there was absolutely no way for some of these corner stone companies (slash tech) to be toppled. And I’m very impressed with their ability to destroy consumer trust!

Sadly gitlab lacked focus especially on software quality.

Re: GitHub was having issues

#119

Post by neilv »

iN7h33nD wrote: post #83
bob1029 wrote: post #75

I've been working on a self-hosted alternative to GitHub and I am curious what HN finds to be the most important features. I think Code, Issues and PRs are the critical aspects, but I don't know what typical workflows look like for others these days. It seems like there are some teams that have figured out a way to turn GH into a labyrinth of CI/CD actions that allegedly produces a ton of business value, but I don't…

Have you seen: https://forgejo.org/

I like what I've used of Forgejo (Git, Issues+Board, Wiki), and have hosted it on servers and localhost easily. I haven't tried its CI features yet.

Codeberg is a cloud site for open source projects that runs Forgejo.

Forgejo is a fork of Gitea, which is another option, especially if you want commercial support, but I haven't tried it yet.

I also kinda like GitLab, both the cloud one and the enterprise on-prem version. And their issue label features work more easily with the board than Forgejo's (automatically moving issues between columns based on scoped labels). Though their pricing tiers have been unfortunate at times (I don't know latest).

Post reply on HN