CocoaPods (and Homebrew) mainly exist because of a lack of tooling in the typical Apple ecosystem. So I would blame Apple for not supporting the community with money or tooling. Letting GitHub with its limited amount of funding pay the bill isn't a nice move. Apple dev relations should throw some money at GitHub so they can provide some dedicated resources or offer to pay the cost of other solutions (like a 3rd party CDN/AWS/Google Cloud/…).
CocoaPods downloads max out five GitHub server CPUs
81–90 of 320 posts
Re: CocoaPods downloads max out five GitHub server CPUs
#82Earlier quoted context omitted.
RubyGems.org has their own web server and web services for publishing library versions and to allow the clients to fetch libraries and query the universal registry.
Same goes for almost any language-specific package manager you could name: Rust/Cargo, Clojure/Clojars, Node/npm, Elixir/hex, Python/PyPi. No matter which way you slice it, what CocoaPods is doing is a bit daft, especially at their scale.
Small note: Cargo _does_ use an index (not the source code), in git, on GitHub. However, we're already doing the directory layout that they recommend in-thread, so we shouldn't have this specific problem.
Re: CocoaPods downloads max out five GitHub server CPUs
#83I've always found Github's business model interesting. What if a massive open-source organization (e.g. Fedora, Apache) decided to use it for all of their development, integrating it with continuous builds and all the associated pulls. Of course this isn't likely to happen for a number of reasons, but there are large open source projects that could put a significant load on their infrastructure if they chose to use G…
> What if a massive open-source organization (e.g. Fedora, Apache) decided to use it
This one is pretty big. https://github.com/torvalds/linux
Re: CocoaPods downloads max out five GitHub server CPUs
#84Earlier quoted context omitted.
They are using GitHub for their intended purpose, hosting their code. That is perfectly fine. What is not perfectly fine is using GitHub as your package host, CocoaPods/Specs is the equivalent of Debians APT using one big GitHub repo to host all their packages. It has 92567 commits and 6872 contributors.
OTOH Homebrew also uses github as its package host and has a respectable 62000 commits and 5600 contributors and github seems to be just fine with it. The big differences seems to be in the way they do their thing: I'm reasonably sure homebrew just git clones then updates the local repository normally[0], it has "only" 2500 files in Library/Formula, and because of its different subject it is way less write-active, Co…
The fact that Github added an API specifically to reduce the server load from Homebrew suggests that it wasn't "just fine". One of the Homebrew maintainers works for GH so they just had a much more direct route to solving the problem than with CocoaPods.
Re: CocoaPods downloads max out five GitHub server CPUs
#85When he says approaches similar to 'other packaging systems', which ones is he referring to? I can see why this is a bad approach but am unfamiliar with what would be considered a better practice (outside of just hosting a .tar on CloudFront).
Probably homebrew, which is also heavily based on github, although not to the same extent as CocoaPods apparently is. It's mentioned further downthread.
The main difference is that homebrew actually updates the git tree to provide updated versions of package specs. CocoaPods adds a new directory and some files for each package version, causing the repo to balloon.
Re: CocoaPods downloads max out five GitHub server CPUs
#86I help run Mozilla's version control infrastructure and the problems described by the GitHub engineer have been known to me for years. Concerns over scaling Git servers are one of the reasons I am extremely reluctant to see Mozilla support a high volume Git server to support Firefox development. Fortunately for us, Firefox is canonically hosted in Mercurial. So, I implemented support in Mercurial for transparently cl…
Wow, this is pretty cool. Reminds me of the performance optimizations Facebook has done with Mercurial: https://code.facebook.com/posts/218678814984400/scaling-merc...
Mercurial was designed to be easy to extend, and it shows.
Re: CocoaPods downloads max out five GitHub server CPUs
#87I love GitHub's response, but I would urge the project more strongly to use modern CDN solutions. CDNs are dirt cheap and incredibly powerful nowadays, for the data sizes that we're talking about here.
How would you define "dirt cheap"? One of the most popular CDN's out there (Akamai) charges $3,500/month for 10TB/month. Who's going to foot that bill? :)
[deleted]
Re: CocoaPods downloads max out five GitHub server CPUs
#88This reply: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm... "Not having to develop a system that somehow syncs required data at all means we get to spend more time on the work that matters more to us, in this case. (i.e. funding of dev hours)" In other words, using github as a free unlimited CDN lets them be as inefficient as they like. Such as having 16k entries in a directory ( https://github.com/Co…
Care to substantiate that last paragraph? Are you really suggesting OS X users use yum?
Re: CocoaPods downloads max out five GitHub server CPUs
#89Just last night, all my pod installs were timing out after ~30ish minutes. That explains it.
Should we infer from your behavior that such a timeframe is fairly normal for cocopod? If I ran an "npm i" (or "pip install", etc.) that didn't respond for a minute, I would suspect a problem and kill it. How can any development process that takes longer than that be practical?
Indeed, I was killing and restarting the process for a while, trying to get it to be more verbose, etc. Eventually I decided to try letting it run, and that didn't work either.
I don't know what's normal; last night was one of my first iOS projects.
Re: CocoaPods downloads max out five GitHub server CPUs
#90Earlier quoted context omitted.
Because abusing that resource makes sure you (or anyone else) won't be able to take it in the future. I thought that's pretty obvious?
pjc50's question was rhetorical, and was offering a critique of the "sharing economy" mindset. That it should be pretty obvious is exactly the point.
OT (from primary discussion): This isn't the sharing economy. This is someone abusing a resource.