Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CocoaPods downloads max out five GitHub server CPUs

github.com

Post reply on HN
Print view View original

261–270 of 320 posts

Re: CocoaPods downloads max out five GitHub server CPUs

#261

Post by niij »

jethro_tell wrote: post #254
zyxley wrote: post #244

Earlier quoted context omitted.

Homebrew uses Github in roughly the same way.

In this case the difference between 'abuse' and fair use may be a homebrew dev that works for github. I would expect that to be a perk of most places that I work. I've run tor exit nodes and repo hosting when I worked for ISPs and Datacenters while at the same time shuting customers down who do the same. The difference being that I had that conversation with my boss and said, 'this will violate our normal terms of se…

Hey, I run 2 middle relays (~80-90mb/s bandwidth total) on cheap VPS's, but was thinking of upping bandwidth and hosting an exi on some dedicated hardwaret. Do you have any tips for running an exit node (legally and technically)?

If you want, you can email me (link in profile). Thanks!

Re: CocoaPods downloads max out five GitHub server CPUs

#262

Post by je42 »

vacri wrote: post #203
nrser wrote: post #169

Earlier quoted context omitted.

yeah, my read was that he didn't totally get the context of the package dist / CDN suggestion immediately. i think the github peeps probably understood that they were using this approach because the developer workflow was simple and strait-forward, but as you scale up simple and strait-forward approaches often break down, and CocoaPods seems to be hitting this tipping point with using Git and Github as a package dist…

> my read was that he didn't totally get the context of the package dist / CDN suggestion immediately. This is mine as well, but it's also troubling to me, given that the repo in question is meant to be a package management system; it means there are fundamental holes in the user's understanding of packaging systems. My mentor has a lot of contempt for the bevy of packaging solutions that people come up with - invari…

There is not an old "xcode" compatible package system. The issue is here, that the C++/Objective-C/C eco system doesn't have a standardize module system. Cocoa pods fills good part of the gap.

Re: CocoaPods downloads max out five GitHub server CPUs

#263

Post by braythwayt »

manyxcxi wrote: post #116
onli wrote: post #2

Note how perfect that response from mhagger is. A clear, honest sounding assurance of what Github wants to deliver. A perfectly comprehensible description of what is the problem, and where it is coming from. And then suggestion how to fix it the project actually can work on, plus mentioning changes to git itself that Github is trying to make that would help. It not only shows great work going on behind the scenes (an…

I was astonished at how selfish/myopic/whatever alloy's response was. To be blunt, you're abusing the shit out of SOMEONE ELSE'S product that you're not even paying for. Your first question shouldn't be to see what Github can do for you to make it so you don't have to make changes. You should be falling over yourself investigating all available avenues for reducing load. It's an incredibly entitled way to think about…

  > To be blunt, you're abusing the shit out of SOMEONE ELSE’S
  > product that you're not even paying for.
I AFAIC, that makes every free user a customer. They may not be a paying customer, but it's GitHub’s choice to be in the free hosting business.

Re: CocoaPods downloads max out five GitHub server CPUs

#264

Post by cyphar »

bobwaycott wrote: post #117
orclev wrote: post #45

Earlier quoted context omitted.

It seems particularly galling that their response to GitHub was to essentially throw their hands up and say "We don't want to change anything, fix it for us". I think GitHub had a near perfect response to this, they analyzed the problem, came up with a set of changes that could be made to help fix it (both short and long term), and pointed to steps they've taken to help out. CocoaPods on the other hand (or at least o…

I think that's pretty unfair. It's really obvious that the initial reply didn't really understand what was going on, and what was being explained. A couple followup additional explanations later, the same dev grokked the problem, CocoaPods' responsibility for the problem, and outlined a list of how they're going to solve it. Seemed to me to be a pretty nice example of professional and helpful candor between GH and an…

I don't understand how you can read "our use of git as our package manager is better than some other package managers we won't name :wink:" is an example of not understanding the problem. It looks more like someone who doesn't want to accept that the problem is on their end.

Re: CocoaPods downloads max out five GitHub server CPUs

#265

Post by cyphar »

justinclift wrote: post #73
pjc50 wrote: post #14

This reply: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm... "Not having to develop a system that somehow syncs required data at all means we get to spend more time on the work that matters more to us, in this case. (i.e. funding of dev hours)" In other words, using github as a free unlimited CDN lets them be as inefficient as they like. Such as having 16k entries in a directory ( https://github.com/Co…

As a data point, "dnf" is the successor to yum. Started using it recently with Fedora 23... and it's pretty decent. (It may not have been earlier on, I really don't know. ;>) Something nifty about the new dnf is several of the older yum commands (eg builddep, yum-downloader) are now integrated directly so don't need extra utils installed. Seems like refinement is still happening. If only my fingers didn't keep typing…

There's also zypper from SUSE/openSUSE.

Re: CocoaPods downloads max out five GitHub server CPUs

#266

Post by cyphar »

zodiac wrote: post #204
web007 wrote: post #159

Earlier quoted context omitted.

One directory per package is completely sensible, just not all in one bunch. It's even fine if the mapping is to a flat namespace at something like the HTTP level - I can mod_rewrite /abcdefg to /a/b/c/abcdefg no problem. My only objection is to file- or directory-level structures that are this flat. I might be mentally deficient, but I can't even process anything that's structured this way. As loathe as I am to admi…

My (perhaps naive) thoughts on this are - suppose a 16k-packages-in-one-directory solution were just as fast as a 16k-packages-sharded-by-prefix (the CPAN solution), then the former is conceptually simpler and so should be preferred. And the fact that you can mechanically transform one structure to the other means that the filesystem (or git) should be able to transparently do it for you (eg use the sharded approach…

The obvious question is how would you implement that. You might argue (as you should) that git has closer semantics to a filesystem than version control. But actually implementing this sharding would require git be a kernel module. Hardlinks and softlinks won't save you because they are both still dentries and thus have the same performance pathology. Maybe you could do it with fuse, but what have you gained by making your version control system even more annoying to use?

Re: CocoaPods downloads max out five GitHub server CPUs

#267

Post by cyphar »

mikeash wrote: post #245
pkaler wrote: post #201

Earlier quoted context omitted.

I think Github's response was about as good as it could be. In hindsight, they probably should have contacted CocoaPods when they pegged one CPU. And they could have given the same general solution to Homebrew and others. With respect to CocoaPods, I would hope someone on the team had thought through performance characteristics of their architecture. It's like they brought a shopping cart onto a city bus and were the…

It's more like bringing a shopping cart onto a city bus, when the bus company said "bring all your stuff! we love it!" doing this for years with no problem, the bus driver says nothing, and then one day the bus driver says "hey, you've been causing a ton of problems with that shopping cart, you need to stop." Surprise seems entirely warranted.

I can't seem to find any posting by GitHub saying "yes! please use our free service as your git-based package manager's backend!" Advertising "host your code and assets with us" doesn't suddenly mean that it's justified to say "fuck it, GitHub can be our CDN".

Re: CocoaPods downloads max out five GitHub server CPUs

#268

Post by couchand »

Nexxxeh wrote: post #239
BogusIKnow wrote: post #149

Earlier quoted context omitted.

I had the same impression from alloy's response. I've basically read it as "hi ho we will not change anything". And it had this passive agressive ring to it, with the hand clapping and the hurray in the beginning and the stone walling in effect.

I took it to be more defensive than passive-aggressive. I took that emojis to be the exact opposite of the almost sarcastic tone I think you're interpreting it to have. I guess even with emojis, it's hard not to make tone ambiguous.

I guess even with emojis, it's hard not to make tone ambiguous.

...probably especially with emoji.

Re: CocoaPods downloads max out five GitHub server CPUs

#269

Post by couchand »

braythwayt wrote: post #263
manyxcxi wrote: post #116

Earlier quoted context omitted.

I was astonished at how selfish/myopic/whatever alloy's response was. To be blunt, you're abusing the shit out of SOMEONE ELSE'S product that you're not even paying for. Your first question shouldn't be to see what Github can do for you to make it so you don't have to make changes. You should be falling over yourself investigating all available avenues for reducing load. It's an incredibly entitled way to think about…

> To be blunt, you're abusing the shit out of SOMEONE ELSE’S > product that you're not even paying for. I AFAIC, that makes every free user a customer. They may not be a paying customer, but it's GitHub’s choice to be in the free hosting business.

Just to clarify, GitHub hosts a very specific type of content: open source software development projects. They never offered to be a general purpose hosting provider.

From @mhagger's measured and thoughtful reply: "We understand that part of the CocoaPods workflow is that its end users (i.e., not just the people contributing to CocoaPods/Specs) fetch regularly from GitHub..."

Re: CocoaPods downloads max out five GitHub server CPUs

#270

Post by cyphar »

Const-me wrote: post #246

I don’t think GitHub acts wisely here. Short term sure, they’re doing the right thing, implementing a nice way to manage the free rider problem without hurting them too much. But long term it’s different. Financially, one average programmer = $80k/year, one average cloud server = $4k/year. And, GitHub has hundreds of millions of venture capital. More than enough to provision a few more servers, even if they will be i…

The way CocoaPods is using git is bad, so they'd probably run into some problems no matter how they host it. If you know anything about filesystems, they start to have pathologies when you have many entries in the same directory (which is why git has it's object store in the format prefix/object). In addition, it looks like git has it's own pathologies with such large numbers of dentries. Having 16000 entries in your "specs" directory is not a good idea. No matter how you store it.

Not to mention that "just buy another server for this one project" sounds like something CocoaPods should pay for.

Post reply on HN