Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub’s Large File Storage is no panacea for Open Source

medium.com

Post reply on HN
Print view View original

21–30 of 66 posts

Re: GitHub’s Large File Storage is no panacea for Open Source

#21

Post by pbiggar »

SwellJoe wrote: post #15
dantiberian wrote: post #7

There's a lot of assumptions here about GitHub being greedy. I've got no idea how much money it costs GitHub to support Open Source projects, but it must easily be in the millions. I think that by this point GitHub deserves the benefit of the doubt before launching into vicious accusations.

While I don't think github is deserving of "vicious accusations", I do believe it is foolish to assume that the github we know today will be the github of tomorrow. SourceForge.net was once an excellent and trustworthy steward of Open Source software projects. It was predicted by some folks in the free software community that it would not always be the case, and alternatives like Savannah were maintained in order to…

Really? These aren't vicious?

"My guess is that some high-level greedy marketing dickwad, completely unaware of the asinine implications of his brilliant idea, signed off on this dumb-as-a-bag-of-rocks pricing model."

"All the marketing material pimping GitHub’s LFS support [...]. I do not believe this is unintentional."

"This is completely batshit. The side effect of this pernicious, greedy pricing model is to [...]"

"I honestly couldn’t believe that GitHub would be willing to do something that shortsighted, visibly motivated by greed from the cash they thought they could extract from some of their users"

Re: GitHub’s Large File Storage is no panacea for Open Source

#22

Post by joesmo »

dantiberian wrote: post #7

There's a lot of assumptions here about GitHub being greedy. I've got no idea how much money it costs GitHub to support Open Source projects, but it must easily be in the millions. I think that by this point GitHub deserves the benefit of the doubt before launching into vicious accusations.

Considering what they get in return for hosting all those free OSS projects, I'd say they're more than breaking even, so let's not pretend like they're doing anyone a favor here. Github would simply not be nearly as big if it wasn't the de facto storage option for most OSS projects. All these OSS projects are their marketing. My point is that they're not hosting OSS projects out of the goodness of their heart or out of service to some community. That's the core of their business. Their greediness is, of course, debatable.

Re: GitHub’s Large File Storage is no panacea for Open Source

#23

Post by forrestthewoods »

I wonder if Perforce Cloud will be able to fill this role at all. Probably not. Open Source isn't their target audience. But it could be a consideration.

Has anyone tried the new Perforce/Git stuff? Is it any good? We're still on an older pre-Helix version.

Re: GitHub’s Large File Storage is no panacea for Open Source

#24

Post by saurik »

lemevi wrote: post #19

Edit: I was wrong, however I learned from the conversation so I am leaving it here! Thanks to those who corrected me. > On the other hand, source files being mostly text, they are more intelligently handled and typically only differences between revisions are stored in the commits. This is completely incorrect, git stores whole blobs from one commit to the other. svn stored patches, but git does not. Every version of…

This is only true for recent commits: as you accumulate commits, garbage collections are performed of the loose blobs and the remaining generation is stored into a pack file, which has been carefully ordered by similarity and stored using a delta-encoding. For more information, this chapter from one of the popular online books about git might suffice.

https://git-scm.com/book/en/v2/Git-Internals-Packfiles

(edit: After I started responding to your comment, you edited your comment to link to the same book! I recommend you continue reading the later chapters: "you'll never believe how it works" ;P.)

Re: GitHub’s Large File Storage is no panacea for Open Source

#25

Post by Vendan »

lemevi wrote: post #19

Edit: I was wrong, however I learned from the conversation so I am leaving it here! Thanks to those who corrected me. > On the other hand, source files being mostly text, they are more intelligently handled and typically only differences between revisions are stored in the commits. This is completely incorrect, git stores whole blobs from one commit to the other. svn stored patches, but git does not. Every version of…

False, git can do both. Run a git gc and check those files again. Chances are, many of your loose object files are missing, but everything still works

Re: GitHub’s Large File Storage is no panacea for Open Source

#26

Post by hyperpape »

eshamow wrote: post #13

I'm not sure I understand why artifacts can't be stored in a different service - even an S3 bucket, if not a real repository service - and fetched dynamically via a build process. Is there a reason why binary blobs need to be stored directly next to code in order to be versioned?

Aside from a second point of failure, how does this integrate with anything? When you push, what piece of software pushes what where? And who pays?

Re: GitHub’s Large File Storage is no panacea for Open Source

#27

Post by jedbrown »

In the interest of not propagating this common misconception:

"The main problem with Git is that binary files are stored “as is” in the history of the project, so that every single revision of a new binary file (even if just a single byte has changed) is stored in full. [...] On the other hand, source files being mostly text, they are more intelligently handled and typically only differences between revisions are stored in the commits."

This is false. Git stores the full version of each file in "loose" format and uses compressed incremental diffs (originally based on xdiff) in packfiles (after "git gc") without distinguishing text vs binary in either case. The issue is that binary files are often compressed themselves (so a one-byte semantic change has nonlocal effect) or have positional references (like jump targets in an executable, causing small changes to cascade).

These factors explain the inefficient handling of binary files, but improving efficiency requires changing the semantics. LFS follows in the path of a few other tools (based on smudge/clean filters) that try to hide the semantic difference from the casual user, though that difference seems to bite people more frequently than we'd like.

Re: GitHub’s Large File Storage is no panacea for Open Source

#28

Post by vezzy-fnord »

pbiggar wrote: post #21
SwellJoe wrote: post #15

Earlier quoted context omitted.

While I don't think github is deserving of "vicious accusations", I do believe it is foolish to assume that the github we know today will be the github of tomorrow. SourceForge.net was once an excellent and trustworthy steward of Open Source software projects. It was predicted by some folks in the free software community that it would not always be the case, and alternatives like Savannah were maintained in order to…

Really? These aren't vicious? "My guess is that some high-level greedy marketing dickwad, completely unaware of the asinine implications of his brilliant idea, signed off on this dumb-as-a-bag-of-rocks pricing model." "All the marketing material pimping GitHub’s LFS support [...]. I do not believe this is unintentional." "This is completely batshit. The side effect of this pernicious, greedy pricing model is to [...]…

Nothing you wouldn't see in your average jwz rant, or really any rant.

Re: GitHub’s Large File Storage is no panacea for Open Source

#30

Post by general_failure »

lemevi wrote: post #19

Edit: I was wrong, however I learned from the conversation so I am leaving it here! Thanks to those who corrected me. > On the other hand, source files being mostly text, they are more intelligently handled and typically only differences between revisions are stored in the commits. This is completely incorrect, git stores whole blobs from one commit to the other. svn stored patches, but git does not. Every version of…

While not stored as 'text patches', when the objects are packed (as in pack files), they are stored as binary diffs.
Post reply on HN