Live data from Hacker News

CocoaPods downloads max out five GitHub server CPUs

github.com

121–130 of 320 posts

Re: CocoaPods downloads max out five GitHub server CPUs

#121
post #42
post #26

Earlier quoted context omitted.

It's very "sharing economy": someone else has a resource you can use for free, so why not take it? (Edit: )

Because abusing that resource makes sure you (or anyone else) won't be able to take it in the future. I thought that's pretty obvious?

You're assuming that people are willing to defer gratification today in order to ensure long-term sustainability. To illustrate why this might be a problematic assumption, I would point to all of human history.

Re: CocoaPods downloads max out five GitHub server CPUs

#122
post #46

I keep coming back to point #4 - who ever thought that 16k objects in a single directory would be a good idea? Ever since FAT that's been a bad idea, and while modern FSes will handle it without completely melting down it's still going to cause long access operations on anything to do with it. Even Finder or `ls` will have trouble with that, and anything with * is almost certainly going to fail. Is the use-case for t…

I do think that your last sentence is the answer: if you're using a package manager instead of working with the directory heavily, this isn't a visible problem which is going to motivate people to work on it.

The other side to consider: “one directory per package” is a very simple policy and it feels right in many ways to people (e.g. Homebrew has a similar structure because it's a natural fit for the domain). If the filesystem and basic tools like ls work just fine (which is certainly the case on OS X, where even "ls -l" or the Finder take less than a second on a directory of that size), isn't there a valid argument that the answer should be some combination of fixing tools which don't handle that well or encouraging people to learn about things like `find` instead of using wildcards which match huge numbers of files?

Re: CocoaPods downloads max out five GitHub server CPUs

#123
post #50

I find it amusing how GitHub's contact[1] form has (probably a recent addition): > GitHub Support is unable to help with issues specific to CocoaPods/CocoaPods. --- [1]: https://github.com/contact

They seem to be doing this based on the last repository visited (for some popular once). Try visiting another repository before going to the contacts page and the message will change or disappear.

Re: CocoaPods downloads max out five GitHub server CPUs

#124
post #10

I've always found Github's business model interesting. What if a massive open-source organization (e.g. Fedora, Apache) decided to use it for all of their development, integrating it with continuous builds and all the associated pulls. Of course this isn't likely to happen for a number of reasons, but there are large open source projects that could put a significant load on their infrastructure if they chose to use G…

> What if a massive open-source organization (e.g. Fedora, Apache) decided to use it This one is pretty big. https://github.com/torvalds/linux

Sure but it's just the kernel, and that's just a mirror. Linus does not use Github to manage kernel development. In fact he's been vitriolic in the past about how Github does pull requests.

I wonder how much traffic the Github Linux repo gets. Seems to me that people who want to use Linux, will go get a distro instead. And people who want to develop the kernel, will follow the kernel development process (which doesn't rely on Github).

Re: CocoaPods downloads max out five GitHub server CPUs

#125

I found the solution humorous. Ironically shallow clones are causing the problems, so fetch the max :) $ git fetch --depth=2147483647

The problem is that the initial clone is depth=1, then subsequent fetches are depth=MAX. If you did clone depth=MAX in the first place it is faster to serve.

Shallow (depth=1) can be converted into a full clone with the above.

Re: CocoaPods downloads max out five GitHub server CPUs

#126
post #9

Earlier quoted context omitted.

Yes, it's a hugely positive advertisement for the essential role that Github plays in the open source community. Also impressive is the followup message from a Github API developer (and Homebrew maintainer) offering access to a beta API that Homebrew is working with to reduce load. https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm...

taking a look at homebrew's implementation of this new API feature, i fail to see how it would dramatically reduce fetches for their (homebrew's) use case. from what i understand, it will only be called when the user manually invokes `brew update`. how often are users calling this command over and over? that being said, i do believe it could help cocoapod's use case since the fetches are done automatically (as i unde…

Hello! I'm the Homebrew maintainer and GitHub employee who wrote this. The main thing this API does for Homebrew is make no-ops really fast for `brew update`. As you point out this results in no speedup (in fact a tiny slowdown) for the case where you only run it where you know you have changes. Where it becomes useful is if you are using multiple taps (Homebrew's 3rd-party repositories) which update infrequently or if you want to run `brew update` automatically in e.g. a project bootstrap script that's run frequently.

In the medium to long-term I'd like to consider Homebrew running `brew update` automatically before you `brew install` (https://github.com/Homebrew/homebrew/issues/38818). For us to ever be able to do that `brew update`'s no-op case needs to be extremely fast.

Re: CocoaPods downloads max out five GitHub server CPUs

#128
post #15

I've never really understood CocoaPods. Dragging a framework into Xcode was never much trouble, and the amount of 3rd party libraries in a OS X / iOS project ought to be fairly small, so the gains are trivial. The potential downsides seem much more annoying. Do you really want to have your dependencies on an overloaded central server somewhere?

Until recently iOS only supported statically linked libraries which could lead to issues if you needed to use multiple components that had a shared dependency that you needed to upgrade for one reason or another. You couldn't touch the embedded version. Additionally there was no native package manager which made sharing libraries a clumsy affair. Cocoapods makes both cases easier.

Dynamic libraries don't help shared dependencies, do they? In both cases, you want the shared dependency to be in its own library. There is a difference in that duplicated dependencies in shared libraries will produce a subtle runtime error rather than an obvious build-time error, but that's not really better.

Re: CocoaPods downloads max out five GitHub server CPUs

#129

Earlier quoted context omitted.

OTOH Homebrew also uses github as its package host and has a respectable 62000 commits and 5600 contributors and github seems to be just fine with it. The big differences seems to be in the way they do their thing: I'm reasonably sure homebrew just git clones then updates the local repository normally[0], it has "only" 2500 files in Library/Formula, and because of its different subject it is way less write-active, Co…

The fact that Github added an API specifically to reduce the server load from Homebrew suggests that it wasn't "just fine". One of the Homebrew maintainers works for GH so they just had a much more direct route to solving the problem than with CocoaPods.

In this case it's actually more that there's a Homebrew maintainer who works for GitHub (me) who has been working on a bunch of improvements to Homebrew's update system (in my spare time). The desire for the API came from Homebrew's side rather than GitHub's and, as it reduces load for GitHub, it was a net win for both parties.

Re: CocoaPods downloads max out five GitHub server CPUs

#130

It's totally reasonable to host your code on github and to build a package manager that loads the content of a package from it's github repo. What seems insane is to use a single github repo as the universal directory of packages and their versions driving your package manager. There's a reason rubygems has their own servers and web services to support this use case for the central library registry, even if the sourc…

I assume they modelled it after Homebrew, which has been working fine doing exactly that for the last 7 years. That only has 3,000 packages vs 15,000 for CocoaPods or 115,000 for RubyGems.

In case somebody is interested in such figures (I certainly am) - NPM has 249,838 as of today [0].

[0]: https://www.npmjs.com

Post reply on HN