Live data from Hacker News

CocoaPods downloads max out five GitHub server CPUs

github.com

311–320 of 320 posts

Re: CocoaPods downloads max out five GitHub server CPUs

#311

Earlier quoted context omitted.

I was astonished at how selfish/myopic/whatever alloy's response was. To be blunt, you're abusing the shit out of SOMEONE ELSE'S product that you're not even paying for. Your first question shouldn't be to see what Github can do for you to make it so you don't have to make changes. You should be falling over yourself investigating all available avenues for reducing load. It's an incredibly entitled way to think about…

> To be blunt, you're abusing the shit out of SOMEONE ELSE’S > product that you're not even paying for. I AFAIC, that makes every free user a customer. They may not be a paying customer, but it's GitHub’s choice to be in the free hosting business.

This is getting granular, but any GitHub user - whether free or paid - is bound by the Terms of Service. Yes, that makes them a customer, and they accordingly have to adhere to the conditions of using the product.

The one that CP's usage most likely confronts is G.12 - found here: (https://help.github.com/articles/github-terms-of-service/)

> If your bandwidth usage significantly exceeds the average bandwidth usage (as determined solely by GitHub) of other GitHub customers, we reserve the right to immediately disable your account or throttle your file hosting until you can reduce your bandwidth consumption.

Re: CocoaPods downloads max out five GitHub server CPUs

#312

Earlier quoted context omitted.

I would be interested if any of the top five open source projects consuming the most resources are paying Github anything.

How much cash do the top five open source projects bring in? That's the other side of it. Funding a side project, let alone a large open source project, is hard

The top five open source projects on Github can't bring in $200/month each?

Re: CocoaPods downloads max out five GitHub server CPUs

#313
post #271

Reading the issue, it seems that one of the problems is a single directory with lots and lots of files in it, which is something of a pathological case for Git. Now, this could be "fixed" by splitting the files in that directory into subdirectories, but the one giant directory will still exist in all the past commits. So would this actually fix anything, or just keep it from getting worse?

Filesystems also have the same pathology, which is why git's object store is of the form prefix/object.

In a filesystem, you can address the problem by reorganizing the directory structure. And you can do the same for future Git commits. But unless I'm mistaken, that colossal directory is stuck in the git history unless you actually rewrite that history, which would require everyone who cloned the repo to either rebase or re-clone the repo. Maybe it's not a problem because the cost is only paid when one checks out one of those old commits, which would happen rarely?

Re: CocoaPods downloads max out five GitHub server CPUs

#314
post #223

Earlier quoted context omitted.

Ugh, I'm skeptical of giving Apple ownership of any kind of developer tool. We all saw how badly they screwed up TestFlight, and now you want to give them the only OSS package manager?

If you're upset about Apple TestFlight, why not just use HockeyApp?

Haven't tried HockeyApp. I use Crashlytics Beta now and it's amazing. Literally "press of a button" deployment: no dealing with provisioning profiles, device UUIDs, or any of that garbage. Just build and deploy, and all your testers get the update instantly!

Re: CocoaPods downloads max out five GitHub server CPUs

#315

Earlier quoted context omitted.

Just to clarify, GitHub hosts a very specific type of content: open source software development projects. They never offered to be a general purpose hosting provider. From @mhagger's measured and thoughtful reply: "We understand that part of the CocoaPods workflow is that its end users (i.e., not just the people contributing to CocoaPods/Specs) fetch regularly from GitHub..."

I know where you’re going with this, but GitHub deliberately blurs that line too. For example, I host both of my blogs on GitHub. I guess I’m saying that I don’t see CocoaPods as being a “bad actor” so much as the extreme tail of a distribution.

Are you hosting your blogs as plain git repos or are you publishing static pages to GitHub Pages, the static hosting option?

Re: CocoaPods downloads max out five GitHub server CPUs

#316
post #260

Earlier quoted context omitted.

> It's really obvious that the initial reply didn't really understand what was going on, and what was being explained. If you are in such a position, then it seems like the best course action would be to ask questions rather than list off reasons that you don't want to deal with it.

Sure. If you know that you don't know what's going on. I walked away with the impression that the guy didn't actually know that.

If he can't understand that his project's resources are consuming 5 whole nodes and terabytes of throughput on Github's infrastructure, then I question his skills as a developer. Even if all of the other technical details are completely obtuse to him, he should at the very least be able to understand the sheer scope of the resources their project is consuming on Github's infrastructure.

Re: CocoaPods downloads max out five GitHub server CPUs

#317
post #270

I don’t think GitHub acts wisely here. Short term sure, they’re doing the right thing, implementing a nice way to manage the free rider problem without hurting them too much. But long term it’s different. Financially, one average programmer = $80k/year, one average cloud server = $4k/year. And, GitHub has hundreds of millions of venture capital. More than enough to provision a few more servers, even if they will be i…

The way CocoaPods is using git is bad, so they'd probably run into some problems no matter how they host it. If you know anything about filesystems, they start to have pathologies when you have many entries in the same directory (which is why git has it's object store in the format prefix/object). In addition, it looks like git has it's own pathologies with such large numbers of dentries. Having 16000 entries in your…

>The way CocoaPods is using git is bad

Very likely true, but I don’t see how’s that related.

>they start to have pathologies when you have many entries in the same directory

Only true for inefficient filesystems like FAT.

For NTFS, 16k entries is nothing, the performance fill start to degrade (due to directory fragmentation) at around 100k entries: http://stackoverflow.com/a/291292/126995

>"just buy another server for this one project" sounds like something CocoaPods should pay for.

I don’t think that’s how 21 century economy works in this case.

Github’s value is likely between $0.75B and $2B.

Bad PR caused by this story will exceed 10 years TCO of that extra server.

Re: CocoaPods downloads max out five GitHub server CPUs

#318
post #310

Earlier quoted context omitted.

"> Told about the issues, the response was basically Alloy et al wanted to invest no time or money into a better situation." "That's actually not true." "taking the route they took, the benefits for them... Yes, that mentions the time and money benefit, but so what – it's honest, and it is valid to limit expectations on what is possible now." So, it was true. Then the dialog continued from there with Alloy paying att…

Saw how you went from "no time and money" to "no significant work"? My point is that he signaled explicitly that he was willing to invest time and work. He literally writes so: > I.e. I’d like us to continue this discussion, at first, from the notion of us maintaining the existing architecture. Where things are absolutely impossible, it would be great if you can include more links to docs/source that explain why thin…

"Saw how you went from "no time and money" to "no significant work"? "

You're right that text is not easy to interpret. For instance, there were two interpretations of my text: a literal and precise one that focuses on how much effort I say he would commit; an interpretation that realizes I was speaking figuratively with hyperbole. It was the latter. The message was a counterpoint supporting that he was selfish rather than a precise statement of how selfish he was. You would be 100% right if we were talking literally about him such as in a court filing or HR report.

"Notice the "at first", notice also the following proposed action of working with a snapshot. You made clear you were expecting another reaction. And I see why. Still, I think you miss how much good faith was contained in this response."

This is possible. Let me re-read his post first. Alright, done. Here's a re-review.

His first response starts with thanks and statements that show either (a) an incredibly joyous and friendly personality or (b) brown-nosing of a salesman before a pitch. Horizontal line. Unclear on some things. Asks for more information. We then get to the reasons:

1. Did no work on syncing data to reduce funded development hours.

2. Don't want to operate a repo due to reduced effort or funding.

3. Easier for their users and adoption.

These are all self-centered. Honest as you said but already support my claim of selfishness. Let's keep looking. Upon a suggestion of other packaging systems, vaguely claims they are using a "smarter" method then reiterates HR and funding justifications above. Ignores alternatives in next sentence to reiterate their existing, strange, and broken solution with a dismissal about having to build a cathedral rather than just using existing solutions.

So, Alloy already laid a foundation of total selfishness in terms of time, funding, and design inflexibility. At this point, Alloy is interested in solutions that totally maintain their existing design and lack of commitment to anything else. Offers to make a few simple changes that "would still use your resources." Asks for information that basically leads to those in recommendations that they begin to apply.

So, re-reading his post, it comes off as incredibly selfish using text that's not hard to interpret. He clearly believes their design works, won't be changed unless forced, changes must take little effort from them, they must not use their funding, and must specifically use GitHub's resources. My claim of selfish and externalizing is fully supported at this point. I think the other commenter's claim of being "myopic" about what he's doing in the project is accurate, too.

Re: CocoaPods downloads max out five GitHub server CPUs

#319
post #54

I love GitHub's response, but I would urge the project more strongly to use modern CDN solutions. CDNs are dirt cheap and incredibly powerful nowadays, for the data sizes that we're talking about here.

How would you define "dirt cheap"? One of the most popular CDN's out there (Akamai) charges $3,500/month for 10TB/month. Who's going to foot that bill? :)

Not surprising, Akamai is always at the top end of the pricing spectrum for CDN services.

That price point works out to $0.35/Gbyte. More typical list pricing for US/EU is in the $0.10-0.15/Gbyte ballpark. Prices decline rapidly as your utilization approaches 1PB per month.

Re: CocoaPods downloads max out five GitHub server CPUs

#320
post #195

Only the Apple development community would think it's OK to have 16,000 subdirectories in one place and abuse GitHub as a free CDN instead of putting some actual effort in and develop their own repository infrastructure - you know, like almost every other package manager in existence.

It should be "Apple or Ruby development community", homebrew is using similar tech stack distributing the `Formula`.

I guess it's no coincidence that those two hispter communities are closely related.
Post reply on HN