Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CocoaPods downloads max out five GitHub server CPUs

github.com

Post reply on HN
Print view View original

201–210 of 320 posts

Re: CocoaPods downloads max out five GitHub server CPUs

#201

Post by pkaler »

mikeash wrote: post #148

The criticism against CocoaPods here seems awfully harsh. Think about it from their perspective. GitHub advertises a free service, and encourages using it. Partly it's free because it's a loss leader for their paid offerings, and partly it's free because free usage is effectively advertising GitHub. CocoaPods builds builds their project on this free service, and everything is fine for years. Then one day things start…

I think Github's response was about as good as it could be. In hindsight, they probably should have contacted CocoaPods when they pegged one CPU. And they could have given the same general solution to Homebrew and others.

With respect to CocoaPods, I would hope someone on the team had thought through performance characteristics of their architecture.

It's like they brought a shopping cart onto a city bus and were then surprised that it inconvenienced the bus driver and the other passengers.

Re: CocoaPods downloads max out five GitHub server CPUs

#202

Post by jwarren »

manyxcxi wrote: post #116
onli wrote: post #2

Note how perfect that response from mhagger is. A clear, honest sounding assurance of what Github wants to deliver. A perfectly comprehensible description of what is the problem, and where it is coming from. And then suggestion how to fix it the project actually can work on, plus mentioning changes to git itself that Github is trying to make that would help. It not only shows great work going on behind the scenes (an…

I was astonished at how selfish/myopic/whatever alloy's response was. To be blunt, you're abusing the shit out of SOMEONE ELSE'S product that you're not even paying for. Your first question shouldn't be to see what Github can do for you to make it so you don't have to make changes. You should be falling over yourself investigating all available avenues for reducing load. It's an incredibly entitled way to think about…

I think there's definitely a tone problem in his first response, but if you continue reading the thread, you'll see that everything else is very positive. I guess it was just a bit of the classic internet text expression problem. Massive high five for the Github guys for looking past it and being extreme pros.

Re: CocoaPods downloads max out five GitHub server CPUs

#203

Post by vacri »

nrser wrote: post #169
tptacek wrote: post #139

Earlier quoted context omitted.

I don't know, it sounded to me like he just didn't totally understand what Github was saying. By the end of the thread, it seemed like everyone was agreeing. I wouldn't be comfortable using words like "selfish" to describe any of what I read. I certainly don't think the barb about your willingness to employ people who write things on Github issues threads that you disagree with is helping anyone understand any part o…

yeah, my read was that he didn't totally get the context of the package dist / CDN suggestion immediately. i think the github peeps probably understood that they were using this approach because the developer workflow was simple and strait-forward, but as you scale up simple and strait-forward approaches often break down, and CocoaPods seems to be hitting this tipping point with using Git and Github as a package dist…

> my read was that he didn't totally get the context of the package dist / CDN suggestion immediately.

This is mine as well, but it's also troubling to me, given that the repo in question is meant to be a package management system; it means there are fundamental holes in the user's understanding of packaging systems.

My mentor has a lot of contempt for the bevy of packaging solutions that people come up with - invariable people look at the old ones, think they're too complex and wrong, write up new code that is Slick(tm) and Fast(tm) and Cool(tm) and they are... until they hit scale. Whether that scale is number of users, or serving multiple environments, or serving a great many packages of different versions... the lack of domain knowledge in the design stage will cause huge amounts of issues.

Re: CocoaPods downloads max out five GitHub server CPUs

#204

Post by zodiac »

web007 wrote: post #159
acdha wrote: post #122

Earlier quoted context omitted.

I do think that your last sentence is the answer: if you're using a package manager instead of working with the directory heavily, this isn't a visible problem which is going to motivate people to work on it. The other side to consider: “one directory per package” is a very simple policy and it feels right in many ways to people (e.g. Homebrew has a similar structure because it's a natural fit for the domain). If the…

One directory per package is completely sensible, just not all in one bunch. It's even fine if the mapping is to a flat namespace at something like the HTTP level - I can mod_rewrite /abcdefg to /a/b/c/abcdefg no problem. My only objection is to file- or directory-level structures that are this flat. I might be mentally deficient, but I can't even process anything that's structured this way. As loathe as I am to admi…

My (perhaps naive) thoughts on this are - suppose a 16k-packages-in-one-directory solution were just as fast as a 16k-packages-sharded-by-prefix (the CPAN solution), then the former is conceptually simpler and so should be preferred. And the fact that you can mechanically transform one structure to the other means that the filesystem (or git) should be able to transparently do it for you (eg use the sharded approach as a hidden implementation, while the end user sees a flat directory). This seems to be similar to what ext4 does (https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#Hash...).

Re: CocoaPods downloads max out five GitHub server CPUs

#205

Post by [deleted] »

Gorbzel wrote: post #134
rmoriz wrote: post #81

CocoaPods (and Homebrew) mainly exist because of a lack of tooling in the typical Apple ecosystem. So I would blame Apple for not supporting the community with money or tooling. Letting GitHub with its limited amount of funding pay the bill isn't a nice move. Apple dev relations should throw some money at GitHub so they can provide some dedicated resources or offer to pay the cost of other solutions (like a 3rd party…

CocoaPods exists because developers want to learn how to "build apps" but lack the resources to intelligently include and link to 3rd party code in their projects. CocoaPods doesn't enable anything not otherwise configurable via git submodules and Xcode project hierarchies / build settings. Therefore, it's not Apple's problem. In fact, I've talked to a non-trivial amount of engineers (both in Cupertino and long time…

[deleted]

Re: CocoaPods downloads max out five GitHub server CPUs

#206

Post by cellularmitosis »

debacle wrote: post #174
cellularmitosis wrote: post #143

Earlier quoted context omitted.

Debian's sync may be nicer, but their client-side solution leaves a bit to be desired. Trivial apt operations (e.g. trying to install a package which is already installed) on an NSLU2 (an ancient 266MHz ARM machine) take several minutes, whereas the same operation takes several seconds on a modern laptop. It turns out this is due to the fact that Debian "main" (Packages.gz) has ballooned to 32MB of plain text when un…

> A side project I've started looking into... Wouldn't cleaning up the package search interface be a similar effort with much greater payoff?

It isn't just package search which is the problem, it is everything which has to parse the packages file, which is basically every apt command. So if you can trim "main" down to 10,000 packages, suddenly every part of apt is faster, and no one has to install any custom apt replacements.

Re: CocoaPods downloads max out five GitHub server CPUs

#207

Post by snuxoll »

cellularmitosis wrote: post #143
pjc50 wrote: post #14

This reply: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm... "Not having to develop a system that somehow syncs required data at all means we get to spend more time on the work that matters more to us, in this case. (i.e. funding of dev hours)" In other words, using github as a free unlimited CDN lets them be as inefficient as they like. Such as having 16k entries in a directory ( https://github.com/Co…

Debian's sync may be nicer, but their client-side solution leaves a bit to be desired. Trivial apt operations (e.g. trying to install a package which is already installed) on an NSLU2 (an ancient 266MHz ARM machine) take several minutes, whereas the same operation takes several seconds on a modern laptop. It turns out this is due to the fact that Debian "main" (Packages.gz) has ballooned to 32MB of plain text when un…

> This problem screams for SQLite.

This is precisely why yum/dnf has been switching from XML for repodata to SQLite. In fact, the only thing that is still XML-only is the comps file which just lists package groups, is updated rarely and "only" weighs in at half a MB.

Re: CocoaPods downloads max out five GitHub server CPUs

#208

Post by cellularmitosis »

mschuster91 wrote: post #199
cellularmitosis wrote: post #143

Earlier quoted context omitted.

Debian's sync may be nicer, but their client-side solution leaves a bit to be desired. Trivial apt operations (e.g. trying to install a package which is already installed) on an NSLU2 (an ancient 266MHz ARM machine) take several minutes, whereas the same operation takes several seconds on a modern laptop. It turns out this is due to the fact that Debian "main" (Packages.gz) has ballooned to 32MB of plain text when un…

> Trivial apt operations (e.g. trying to install a package which is already installed) on an NSLU2 (an ancient 266MHz ARM machine) take several minutes Actually, I'm surprised one can actually run a modern Linux on the NSLU2 given its shameful lack of RAM and slooow USB port. But it was a nice gadget when it came out and it was fun to experiment with it. > It turns out this is due to the fact that Debian "main" (Pack…

Actually the part which parses the packages file is a bunch of c++

Re: CocoaPods downloads max out five GitHub server CPUs

#209

Post by hatsix »

ragall wrote: post #179
mikemcquaid wrote: post #126

Earlier quoted context omitted.

Hello! I'm the Homebrew maintainer and GitHub employee who wrote this. The main thing this API does for Homebrew is make no-ops really fast for `brew update`. As you point out this results in no speedup (in fact a tiny slowdown) for the case where you only run it where you know you have changes. Where it becomes useful is if you are using multiple taps (Homebrew's 3rd-party repositories) which update infrequently or…

If you plan to rehaul Homebrew, I suggest having by default 3 operations, "update", "fetch" and "install" because some people might find themselves in the situation of having bad connectivity(especially low bandwidth) and being able to fetch the sources, to compile later is very important. Especially having "install" issue synchronous "update" operations is bad if you're on a network with high packet loss, like tethe…

I don't get it... If you've never used homebrew, and are only really a user of other package managers, why are you attempting to offer advice on what they've been doing for years.

Is this classic "I have an opinion and all of my opinions are great, so I must pontificate!"?

Maybe you should just trust that the developers of the only successful package manager for OS X have some idea of what their users want and need... and that as someone who has NEVER used their software, and not a seasoned veteran of any sort of similar projects, your opinion counts WAY less than any of their actual users.

Re: CocoaPods downloads max out five GitHub server CPUs

#210

Post by vacri »

masklinn wrote: post #140
pjc50 wrote: post #71

Earlier quoted context omitted.

It is a package manager, though? The fundamental idea of downloading a list of available options of which the user picks some, and the system pulls in dependencies, is almost exactly how dpkg and yum work. The location to which the packages are installed is a detail.

A language package manager must be able to "install" the same packages over and over again (and possibly "install" multiple versions of the same package in the same environment), and the ability to push packages is generally considered part of their duty, not so for OS package managers, you don't use dpkg to send a package to debian's repositories. > The fundamental idea of downloading a list of available options of…

The fundamentals for package management also move the package into it's final resting place(s), where it's going to do it's work. Curl doesn't do that, it just gives you a single file somewhere.

I'd also consider removing a package to be a fundamental part of a manager. The two items you describe would be a 'package grabber'.

Post reply on HN