Live data from Hacker News

GitHub Packages Is Down

githubstatus.com

51–60 of 83 posts

Re: GitHub Packages Is Down

#51
post #4

FYI: this breaks homebrew

Nix too I presume

Nope.

This is actually the second (maybe third; I didn't even know about GitHub's outage a week and a half ago so idk how Nix was or was not affected) time in three months or less that a partial GitHub outage or GitHub change has taken down Homebrew while leaving Nix unaffected.

Re: GitHub Packages Is Down

#52

Earlier quoted context omitted.

So this is why hb started asking for access to my keychain all the time? (macos) Does it need to log in to gh to install open source software now?

you don't have to, but hb will ask you to if you have a saved credential because your quota on GitHub for downloading packages is much higher if you are logged in. anonymous stuff on GitHub is usually limited to 60 requests per hour per ip address. if you're authenticated, it's several hundred if not several thousand.

That sounds... problematic for people on big NATs? (e.g. universities?)

Re: GitHub Packages Is Down

#53
post #35

What I find funny and unexplainable is that this class of problem was solved decades ago with distribution mirrors. It's not really clear to me why, within the last decade or so, we collectively decided to centralize hosting on one specific cloud service whose downtime now affects builds across nearly every company. What's perhaps even more surprising to me is that, after a repeated track history of severe and freque…

Mostly agreed, but I'd hazard a guess that the scale of github is far larger than distribution mirrors of old.

There are many distribution mirrors that are financed by universities that are on the the Internet backbone in the US

Heck, even in Asia I did not have trouble with finding a good mirror.

Re: GitHub Packages Is Down

#54

What I find funny and unexplainable is that this class of problem was solved decades ago with distribution mirrors. It's not really clear to me why, within the last decade or so, we collectively decided to centralize hosting on one specific cloud service whose downtime now affects builds across nearly every company. What's perhaps even more surprising to me is that, after a repeated track history of severe and freque…

IMO we all realized that it doesn't actually matter that much, most of the time. Here we are, indeed, after three years of severe and frequent outages! But everything is... basically fine? Life is full of tradeoffs.

People who really care make their own dependency/build caches, eg, we had Docker containers we could fall back to. If you really needed to patch, build on top of an existing artifact image — and then rebuild when vendor service comes back. In practice, I just waited a few hours.

Problem solved(ish).

Re: GitHub Packages Is Down

#55

I'm disappointed that this is an issue for some package management systems. 20 years ago I helped run a mirroring service, it's still running today. Distributions such as Debian have hundreds of mirrors. This is a solved problems, but we just decided to but everything in the hands of one for-profit company.

just yesterday I was stuck for an hour because a debian package mirror went down. took a long time to talk the user through changing their sources.list so that another mirror was chosen, and the mirror chosen out of that pool was down also. finally I had to manually check for a good mirror and give them the URLs. the user's take was "why don't they use GitHub packages?" "still running today" doesn't mean 100.0% uptim…

This is not how this works anymore. The system that is behaving this way must be relatively old at this point since almost all modern Debian based distros use the "mirror://" URI syntax now that automatically falls back to another mirror if one fails.

Re: GitHub Packages Is Down

#56
post #14

Earlier quoted context omitted.

I did not know package managers relied on Github, this is the most unwise thing to do from a package manager perspective. Anyone could just change username/organization and break thousands/millions of build.

Hello and welcome to PEP-508! In Python, we don't say "we don't host packages on a proprietary platform", we say "we have absolutely no clue where they are hosted and nobody audits them anyways, and we don't enforce package signing, and we'll just build from source with no build isolation what so ever, unless you remember to specify an obscure command-line option when installing... and have a nice day!"

it's amazing how we never learn. Things that Perl and CPAN and Linux distros figured out decades ago are constant issues today. It wasn't that long ago that NPM didn't even have checksums. CPAN runs unit tests on install. I can't imagine how slow that would be with npm.

Package signing is, well... I suppose that's another lesson from the '90s people will learn about soon enough. With a web of trust as broad as python or npm you'll just have everyone running around with signing keys and "trusting" any key they come across because none of it is built on personal relationships. When Archlinux asks me to confirm adding package keys, what am I going to do? Say no? I don't know these people, but I want my shit to work.

Re: GitHub Packages Is Down

#57

What I find funny and unexplainable is that this class of problem was solved decades ago with distribution mirrors. It's not really clear to me why, within the last decade or so, we collectively decided to centralize hosting on one specific cloud service whose downtime now affects builds across nearly every company. What's perhaps even more surprising to me is that, after a repeated track history of severe and freque…

It's not even close to the same thing. Universities hosting mirrors piggybacked off academic networks; not just the computer kind but the social kind, where professors would regularly meet professors from other institutions at academic conferences. It was in the collective interests of the universities to set up mirrors to solve the pre-eminent issue of slow WAN networks.

Today most companies need private package registries. Legacy networks are a resource drain. Nobody else uses your private packages nor do you want anybody else to host a mirror and authentication is required anyway.

Plus the idea that GitHub is hosting everything in a single datacenter is laughable on its face.

Re: GitHub Packages Is Down

#58
post #4

FYI: this breaks homebrew

Man, this disappoints me. I was a tech lead on the packages project about 3 years ago, specifically on the redesign for OCI container support and making anonymous downloads of public packages as reliable as possible was a top priority.

If that flow was broken there’s only a handful of things it could be; specifically azure blob store or azure MySQL, but both of those should have layers of redundancy. Public anonymous download bypasses most everything else; no auth services, rails monolith, or metered billing. It does emit some events to the message bus for metrics, but that’d effect much more than packages if there was an issue with it.

As far as I’m aware this is the first time anonymous public package download broke since I left a few years back.

Re: GitHub Packages Is Down

#59

I'm disappointed that this is an issue for some package management systems. 20 years ago I helped run a mirroring service, it's still running today. Distributions such as Debian have hundreds of mirrors. This is a solved problems, but we just decided to but everything in the hands of one for-profit company.

If only there were some way to make git distributed! /s

You mean something like a git annex enabled branch tracking mirror locations of each release artifact like HTTP URLs, (webseeded) torrents, maybe even something content addressed like IPFS? sigh

Re: GitHub Packages Is Down

#60
post #52

Earlier quoted context omitted.

you don't have to, but hb will ask you to if you have a saved credential because your quota on GitHub for downloading packages is much higher if you are logged in. anonymous stuff on GitHub is usually limited to 60 requests per hour per ip address. if you're authenticated, it's several hundred if not several thousand.

That sounds... problematic for people on big NATs? (e.g. universities?)

NATs are problematic already.

Every office I've ever worked at has that one guy who is really good at tickling Google with scripts until it puts you all behind a CAPTCHA.

Post reply on HN