Live data from Hacker News

CocoaPods downloads max out five GitHub server CPUs

github.com

101–110 of 320 posts

Re: CocoaPods downloads max out five GitHub server CPUs

#101
post #86

I help run Mozilla's version control infrastructure and the problems described by the GitHub engineer have been known to me for years. Concerns over scaling Git servers are one of the reasons I am extremely reluctant to see Mozilla support a high volume Git server to support Firefox development. Fortunately for us, Firefox is canonically hosted in Mercurial. So, I implemented support in Mercurial for transparently cl…

Wow, this is pretty cool. Reminds me of the performance optimizations Facebook has done with Mercurial: https://code.facebook.com/posts/218678814984400/scaling-merc... Mercurial was designed to be easy to extend, and it shows.

Git was created and designed to support Linus' workflow when developing the Linux kernel.

Hg was designed to be a DVCS system.

Re: CocoaPods downloads max out five GitHub server CPUs

#102
post #45
post #11

From CocoaPods.org: > CocoaPods is a dependency manager for Swift and Objective-C Cocoa projects. It has over ten thousand libraries and can help you scale your projects elegantly. The developer response: > [As CocoaPods developers] Scaling and operating this repo is actually quite simple for us as CocoaPods developers whom do not want to take on the burden of having to maintain a cloud service around the clock (user…

It seems particularly galling that their response to GitHub was to essentially throw their hands up and say "We don't want to change anything, fix it for us". I think GitHub had a near perfect response to this, they analyzed the problem, came up with a set of changes that could be made to help fix it (both short and long term), and pointed to steps they've taken to help out. CocoaPods on the other hand (or at least o…

A later comment from @alloy was a bit more gracious about this https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm..., but I agree, it wasn't a good look.

Re: CocoaPods downloads max out five GitHub server CPUs

#103
post #14

This reply: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm... "Not having to develop a system that somehow syncs required data at all means we get to spend more time on the work that matters more to us, in this case. (i.e. funding of dev hours)" In other words, using github as a free unlimited CDN lets them be as inefficient as they like. Such as having 16k entries in a directory ( https://github.com/Co…

Arch and Debian contributors have tried a good approach for package management..

0: p2pacman - Bittorrent powered pacman wrapper

1: pacman & torrent, feasible?

2: DebTorrent

(0) https://bbs.archlinux.org/viewtopic.php?id=163362

(1) https://bbs.archlinux.org/viewtopic.php?id=115731

(2) https://wiki.debian.org/DebTorrent

I believe scaling this could happen with either: 1) lightweight filesystem\directory versioning support, like how btrfs allows you to mount snapshots. This way, peers could update whichever version of a torrent they have. Or 2) very reliable means to update to the latest torrent release (as reliable as syncing with peers), which afaict means smarter bittorrent clients that can perform DHT-based "crawling". Those recent defcon(?) "hacks" to query peers for similar torrents based on user pools and connection histories (or something like that) would make sense here.

A cool side-note: In one of my few experiences diving into `.git`, I diff'd it before and after making changes to its tracked sources, like adding files and modifying them. It looked like a torrent that included version control data would make out just fine if an updated torrent expected similar data in the same location. Again, a smarter bittorrent client would need to sort some of this out. See also 0': Updating Torrents Via Feed URL. Anyway, most users would probably leave that part out, in favor of only which parts they need.

(0') http://www.bittorrent.org/beps/bep_0039.html

Another cool side-note: This would also allow for easily adding repos from multiple sources... Look at how many ( non-automated :-( ) merge requests com.github/CocoPods/Specs's caregivers have reviewed: 13,331 as of now (0'').

(0'') https://github.com/CocoaPods/Specs/pulls?q=is%3Apr+is%3Aclos...

Re: CocoaPods downloads max out five GitHub server CPUs

#105

Earlier quoted context omitted.

How would you define "dirt cheap"? One of the most popular CDN's out there (Akamai) charges $3,500/month for 10TB/month. Who's going to foot that bill? :)

10TB Amazon AWS is around $85-100, depending on how international your users are.

I believe that's off by an order of magnitude - that's actually the price per 1 TB. Also, you can turn off distribution into remote PoPs that charge more than $85/TB.

Re: CocoaPods downloads max out five GitHub server CPUs

#106
post #11

From CocoaPods.org: > CocoaPods is a dependency manager for Swift and Objective-C Cocoa projects. It has over ten thousand libraries and can help you scale your projects elegantly. The developer response: > [As CocoaPods developers] Scaling and operating this repo is actually quite simple for us as CocoaPods developers whom do not want to take on the burden of having to maintain a cloud service around the clock (user…

Well, the flip side is that the CocoaPods developers are all volunteers (right?). They aren't really deriving any benefit out of the work they do on CocoaPods, and if you ask them to take on financial or ongoing maintenance obligations for a volunteer project, they probably just won't do it. The major benefits of CocoaPods existence go to iOS developers, but there's a tragedy of the commons effect here, where no indi…

> They aren't really deriving any benefit out of the work they do on CocoaPods

No direct financial benefit, but they are deriving a benefit out of their work.

Re: CocoaPods downloads max out five GitHub server CPUs

#107
post #71

Earlier quoted context omitted.

Perhaps because Cocoapods is not an OS package manager or anything close to it. It installs libraries within the context of an XCode project, regardless of the host system or what is installed for other projects.

It is a package manager, though? The fundamental idea of downloading a list of available options of which the user picks some, and the system pulls in dependencies, is almost exactly how dpkg and yum work. The location to which the packages are installed is a detail.

CocoaPods is much more akin to RubyGems or PyPI or CPAN, all of which are established as useful tools outside of OS level package managers. There's a need for a iOS/Cocoa package manager (that understands Xcode!) and CocoaPods has so far been the most successful.

Re: CocoaPods downloads max out five GitHub server CPUs

#108
post #3

Wow, really impressive response from GitHub. The right amount of technical detail coupled with balanced tone--halfway between "we support you" and "you make us crazy." One correction to the post title: it's not maxing five nodes, but five CPUs.

Ok, we replaced "nodes" with "server CPUs" in the title.

Re: CocoaPods downloads max out five GitHub server CPUs

#110
post #74
post #11

From CocoaPods.org: > CocoaPods is a dependency manager for Swift and Objective-C Cocoa projects. It has over ten thousand libraries and can help you scale your projects elegantly. The developer response: > [As CocoaPods developers] Scaling and operating this repo is actually quite simple for us as CocoaPods developers whom do not want to take on the burden of having to maintain a cloud service around the clock (user…

1) I don't think they mean "scale your projects elegantly" in a "distribute your project to millions of customers" sense, but rather in a "add lots of libraries and not have it be a hassle" sense. 2) It makes perfect sense to let GitHub handle the performance hit until issues arise. Premature optimization is the devil, right? But once there start to be issues, it's definitely unfair to turn around and say "well you o…

Yea pretty sure they definitely mean "Add More Libraries" when they say "scaling", it wouldn't make sense otherwise given what CocoaPods does...
Post reply on HN