Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoid Git LFS if possible

gregoryszorc.com

Post reply on HN
Print view View original

81–90 of 142 posts

Re: Avoid Git LFS if possible

#81

Post by 0xbadcafebee »

Dylan16807 wrote: post #59
tux1968 wrote: post #13

Earlier quoted context omitted.

$60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me. What price would you expect?

> $60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me. What you describe sounds fair to me. The problem is that 50GB is not a decent fraction of a hard disk.

I don't know if you remember back when service providers would put customers' entire dataset on a single set of spinning platters. The drive would of course die and the customer would get super pissed off when the provider would say "You were supposed to keep a backup..." So now providers like GitHub and GitLab are basically super-redundant storage, network, and application providers, who also happen to run Git.

If you store 50GB in AWS S3 (US-East-2), download 1000GB, do 100 PUT operations and 1000 GET operations, the cost is $89.68 per month.

Considering that GitHub isn't just providing you with storage, but a complete Git LFS solution plus storage, plus traffic that you can just use and not think about, I think it's worth the expense. But then again I probably wouldn't store binary blobs in Git.

Re: Avoid Git LFS if possible

#82

Post by theamk »

dwohnitmok wrote: post #26

git-annex is an interesting alternative the HTTP-first nature of Git LFS and the one-way door bother you. You can remove it after the fact if you don't like it, it supports a ton of protocols, and it's distributed just like git is (you can share the files managed by git-annex among different repos or even among different non-git backends such as S3). The main issue that git-annex does not solve is that, like Git LFS,…

Last time we used git-annex was a few years ago, and it was too decentralized: the "sync" command that we used to download the remote content would also upload the information about current state.

This means there are were no read-only operations: you just want some files.. and that throwaway clone and CI machine would get recorded into the global repo state. If you are not careful, and will be propagated forever and would appear in the various reports.

Re: Avoid Git LFS if possible

#83

Post by theamk »

CobrastanJorji wrote: post #3
breck wrote: post #2

My practice for storing large files with Git is to include the metadata for the large file in a tiny file(s): 1. Type information. Enough to synthesize a fake example. 2. A simple preview. This can be a thumb or video snippet, for example. 3. Checksum and URL of the big file. This way your code can work at compile/test time using the snippet or synthesized data, and you can fetch the actual big data at ship time. You…

Is this just a manual equivalent of git LFS, or is there some advantage here?

This is pretty superior to git LFS in many aspects:

- You have file type and preview that you can use without getting the full thing

- You have a custom metadata for each file enforced by your scripts -- for example for archives, you may store the list of files inside. This will allow your CI tests to validate the references into the files without having to download the whole huge thing.

- You fully control remote fetch logic. Multiple servers? Migration rules for old revisions? That weird auth scheme that your IT insists on? It is all supported with a bit of code.

- You fully control local storage. Do you want a computer-wide shared CAS cache between multiple users? What if you have NAS that most users mount? Or maybe s3fs is your thing? Adding support is easy.

The main downside is that you get to do all the tooling and documentation, so I would not recommend this for the smaller teams. Nor would I recommend this for open-source projects.

But if your infra team is big enough to support this, you'll definitely have the better experience than generic Git LFS.

Re: Avoid Git LFS if possible

#85

Post by cerved »

Aeolun wrote: post #48

Did he really just try to make the argument that we shouldn’t use LFS because Git will have large file support at some unspecified point in the future? LFS has existed for several years, and as far as I know Git still doesn’t have support for large files. At this point I’m not holding out much hope.

git supports large files, it just can't track changes in binary files efficiently and if they're large you check in a new blob every modification.

If they're just sitting around it's fine, but then why would you have them in VC

Re: Avoid Git LFS if possible

#86

Post by cerved »

madjam002 wrote: post #15

As much as Git LFS is a bit of a pain, on recent projects I've resorted to committing my node_modules with Yarn 2 to Git using LFS and it works really well. Note that with Yarn 2 you're committing .tar.gz's of packages rather than the JS files themselves, so it lends itself quite well to LFS as there are a smaller number of large files. https://yarnpkg.com/features/zero-installs#how-do-you-reach-... https://yarnpkg.c…

why are you committing packages?

Re: Avoid Git LFS if possible

#87

Post by cerved »

ecnahc515 wrote: post #63

You don't need to rewrite history unless you weren't using LFS or accidently committed large files to the repository. Nothing about LFS "requires" rewriting history. Not to mention, many users are paying for a service that provides LFS, and hosting an LFS service isn't crazy hard. It's a file server with a custom API, it's mostly doable using S3 as a backend. It's not like this is crazy complicated stuff.

other stuff might require require rewriting history

Re: Avoid Git LFS if possible

#88

Post by cerved »

KETpXDDzR wrote: post #49

This opinion only lists issues, not solutions. Sure, they advertise mercurial, but migrating from git to mercurial is unrealistic for many cases. I'd title it: "Why Mercurial is better than git+LFS"

The author is detailing the problems wrt git-lfs, why they are problems and how those problems are overcome in a similar technical solution in a similar VCS. I think the original title is fine

Re: Avoid Git LFS if possible

#89

Post by rakoo »

theamk wrote: post #82
dwohnitmok wrote: post #26

git-annex is an interesting alternative the HTTP-first nature of Git LFS and the one-way door bother you. You can remove it after the fact if you don't like it, it supports a ton of protocols, and it's distributed just like git is (you can share the files managed by git-annex among different repos or even among different non-git backends such as S3). The main issue that git-annex does not solve is that, like Git LFS,…

Last time we used git-annex was a few years ago, and it was too decentralized: the "sync" command that we used to download the remote content would also upload the information about current state. This means there are were no read-only operations: you just want some files.. and that throwaway clone and CI machine would get recorded into the global repo state. If you are not careful, and will be propagated forever and…

That's the whole premise of git-annex: not distributing content but distributing what machine has the content. If you just want to get the content you have to hack git-annex, probably by reading the manifest, to get the url and download content in a third party process

Re: Avoid Git LFS if possible

#90

Post by pooya13 »

Maybe I am missing the point. What is the alternative this article proposes then?... Also, Git is not central so how can you ever integrate large file support without a separate server?
Post reply on HN