Live data from Hacker News

CocoaPods downloads max out five GitHub server CPUs

github.com

141–150 of 320 posts

Re: CocoaPods downloads max out five GitHub server CPUs

#141
post #115

Go's package manager, `go get`, also downloads from GitHub. I don't know the details of how `go get` and CocoaPods work, but I would be interested in learning why one is unscalable and the other seems to work.

Because cloning the dependency is not the issue here. CocoaPods is keeping its index in a git repo that is updated by the user as a way to get the latest index. This is the repo that incur a lot of requests from everyone.

Go get on the other hand doesn't keep any index. It just uses the url to download the dependency because of the mapping "url==project name" that exists with go projects.

Re: CocoaPods downloads max out five GitHub server CPUs

#142

Earlier quoted context omitted.

Arch and Debian contributors have tried a good approach for package management.. 0: p2pacman - Bittorrent powered pacman wrapper 1: pacman & torrent, feasible? 2: DebTorrent (0) https://bbs.archlinux.org/viewtopic.php?id=163362 (1) https://bbs.archlinux.org/viewtopic.php?id=115731 (2) https://wiki.debian.org/DebTorrent I believe scaling this could happen with either: 1) lightweight filesystem\directory versioning sup…

> Arch and Debian contributors have tried a good approach for package management.. > 0: p2pacman - Bittorrent powered pacman wrapper > 1: pacman & torrent, feasible? > 2: DebTorrent That's about distributing packages via p2p. The problematic repository doesn't store any package data, it stores package metadata (it's the cocoapods index if you will).

>~ package metadata, not package data

metadata != data ??

Re: CocoaPods downloads max out five GitHub server CPUs

#143
post #14

This reply: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm... "Not having to develop a system that somehow syncs required data at all means we get to spend more time on the work that matters more to us, in this case. (i.e. funding of dev hours)" In other words, using github as a free unlimited CDN lets them be as inefficient as they like. Such as having 16k entries in a directory ( https://github.com/Co…

Debian's sync may be nicer, but their client-side solution leaves a bit to be desired.

Trivial apt operations (e.g. trying to install a package which is already installed) on an NSLU2 (an ancient 266MHz ARM machine) take several minutes, whereas the same operation takes several seconds on a modern laptop.

It turns out this is due to the fact that Debian "main" (Packages.gz) has ballooned to 32MB of plain text when uncompressed, comprising more than 41,000 packages, and it has to be parsed and assembled into a dependency tree for every apt operation. This problem screams for SQLite.

A side project I've started looking into is to make a transparent apt proxy which provides a trimmed down Packages.gz (e.g., removing anything which uses X11), which would be a lot easier that rewriting apt to use a SQLite backend.

Re: CocoaPods downloads max out five GitHub server CPUs

#144
post #54

I love GitHub's response, but I would urge the project more strongly to use modern CDN solutions. CDNs are dirt cheap and incredibly powerful nowadays, for the data sizes that we're talking about here.

I believe the problem was not the bandwith usage (which would be saved by a CDN) but the way CocoaPods is using git that generates a lot of CPU expensive operations.

Things that GitHub suggested help with that: faster check for updates, breaking up big directories so diffs are computed faster.

Re: CocoaPods downloads max out five GitHub server CPUs

#145

I help run Mozilla's version control infrastructure and the problems described by the GitHub engineer have been known to me for years. Concerns over scaling Git servers are one of the reasons I am extremely reluctant to see Mozilla support a high volume Git server to support Firefox development. Fortunately for us, Firefox is canonically hosted in Mercurial. So, I implemented support in Mercurial for transparently cl…

GitHub claim in this thread that they pay about 1s for a full clone without a caching CDN, due to their bitmap indexing patches.

Re: CocoaPods downloads max out five GitHub server CPUs

#146
post #26
post #11

From CocoaPods.org: > CocoaPods is a dependency manager for Swift and Objective-C Cocoa projects. It has over ten thousand libraries and can help you scale your projects elegantly. The developer response: > [As CocoaPods developers] Scaling and operating this repo is actually quite simple for us as CocoaPods developers whom do not want to take on the burden of having to maintain a cloud service around the clock (user…

It's very "sharing economy": someone else has a resource you can use for free, so why not take it? (Edit: )

I take your use of "sharing economy " as post-uber/airbnb/etc, instead of, as I first heard it, from couchsurfing, potlucks, bittorrent, etc well before that.

Also while nit-picking, I would clear up your use of "for free " as "as a freebie", again post-[insert: x̄ȳz, inc].

Re: CocoaPods downloads max out five GitHub server CPUs

#147
post #2

Note how perfect that response from mhagger is. A clear, honest sounding assurance of what Github wants to deliver. A perfectly comprehensible description of what is the problem, and where it is coming from. And then suggestion how to fix it the project actually can work on, plus mentioning changes to git itself that Github is trying to make that would help. It not only shows great work going on behind the scenes (an…

I was astonished at how selfish/myopic/whatever alloy's response was. To be blunt, you're abusing the shit out of SOMEONE ELSE'S product that you're not even paying for. Your first question shouldn't be to see what Github can do for you to make it so you don't have to make changes. You should be falling over yourself investigating all available avenues for reducing load. It's an incredibly entitled way to think about…

It probably wasn't intentionally selfish. Just really short-sighted, which isn't uncommon. Sometimes it takes a while for things to penetrate for anybody. Alloy's later comments seem much more productive. Whether that's because of backlash, he reread the earlier comment and realized how it seemed, or just had some time to think about the imposition on Github, at least it looks like CocoaPods is going to reconsider how they handle their distribution.

Re: CocoaPods downloads max out five GitHub server CPUs

#148
The criticism against CocoaPods here seems awfully harsh.

Think about it from their perspective. GitHub advertises a free service, and encourages using it. Partly it's free because it's a loss leader for their paid offerings, and partly it's free because free usage is effectively advertising GitHub. CocoaPods builds builds their project on this free service, and everything is fine for years.

Then one day things start failing mysteriously. It looks like GitHub is down, except GitHub isn't reporting any problems, and other repositories aren't affected.

After lots of headscratching, GitHub gets in touch and says: you're using a ton of resources, we're rate limiting you, you're using git wrong, and you shouldn't even be using git.

That's going to be a bit of a shock! Everything seemed fine, then suddenly it turns out you've been a major problem for a while, but nobody bothered to tell you. And now you're in hair-on-fire mode because it's reached the point where the rate-limiting is making things fail, and nobody told you about any of these problems before they reached a crisis point.

It strikes me as extremely unreasonable to expect a group to avoid abusing a free service when nobody tells them that it's abuse, and as far as they know they're using it in a way that's accepted and encouraged. If somebody is doing something you don't like and you want them to stop, you have to tell them, or nothing will happen!

I'm not blaming GitHub here either. I'm sure they didn't make this a surprise on purpose, and they have a ton of other stuff going on. This looks like one of those things where nobody's really to blame, it's just an unfortunate thing that happened.

(And just to be clear, I don't have much of a dog in this fight on either side. My only real exposure to CocoaPods is having people occasionally bug me to tag my open source repositories to make them easier to incorporate into CocoaPods. I use GitHub for various things like I imagine most of us do, but am not particularly attached to them.)

Re: CocoaPods downloads max out five GitHub server CPUs

#149
post #2

Note how perfect that response from mhagger is. A clear, honest sounding assurance of what Github wants to deliver. A perfectly comprehensible description of what is the problem, and where it is coming from. And then suggestion how to fix it the project actually can work on, plus mentioning changes to git itself that Github is trying to make that would help. It not only shows great work going on behind the scenes (an…

I was astonished at how selfish/myopic/whatever alloy's response was. To be blunt, you're abusing the shit out of SOMEONE ELSE'S product that you're not even paying for. Your first question shouldn't be to see what Github can do for you to make it so you don't have to make changes. You should be falling over yourself investigating all available avenues for reducing load. It's an incredibly entitled way to think about…

I had the same impression from alloy's response. I've basically read it as "hi ho we will not change anything".

And it had this passive agressive ring to it, with the hand clapping and the hurray in the beginning and the stone walling in effect.

Re: CocoaPods downloads max out five GitHub server CPUs

#150

Earlier quoted context omitted.

> Arch and Debian contributors have tried a good approach for package management.. > 0: p2pacman - Bittorrent powered pacman wrapper > 1: pacman & torrent, feasible? > 2: DebTorrent That's about distributing packages via p2p. The problematic repository doesn't store any package data, it stores package metadata (it's the cocoapods index if you will).

>~ package metadata, not package data metadata != data ??

You got it.
Post reply on HN