Good points, but it seems optimistic to assume that git will have good, native, large file support anytime soon. I‘ve been waiting quite a while for git submodules to improve..
Avoid Git LFS if possible
111–120 of 142 posts
Re: Avoid Git LFS if possible
#112Earlier quoted context omitted.
It tracks the changes fine. It’s just that it doesn’t make sense to track changes in a binary.
So I decided to check this out. Used dd if=/dev/random to create a 100mb file, checked that in, used dd again to modify 10mb of that file, checked that in and the result were two 98mb objects. Tracking changes of binaries makes a lot of sense if you use that to only store incremental changes to the file. Git stores each modification of a binary file as a separate blob since it doesn't know how to track its changes. T…
Git LFS has the advantage of not pulling all versions of a large file, too. Instead, it only pulls the version it's checking out.
In our project it helped dramatically as you only pull X MB instead of X * Y MB when a CI or developer clone the (already big) repo.
Re: Avoid Git LFS if possible
#113Earlier quoted context omitted.
> $60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me. What you describe sounds fair to me. The problem is that 50GB is not a decent fraction of a hard disk.
I don't know if you remember back when service providers would put customers' entire dataset on a single set of spinning platters. The drive would of course die and the customer would get super pissed off when the provider would say "You were supposed to keep a backup..." So now providers like GitHub and GitLab are basically super-redundant storage, network, and application providers, who also happen to run Git. If y…
The number you're citing is basically entirely bandwidth, which has two main problems.
One is that amazon has an enormous markup on bandwidth, compared to their other products.
The other is that GitHub does not actually let you download each file 20x in a month and "not think about" it. 50GB of space for a month only gets you 50GB of bandwidth.
If Amazon didn't explicitly ban people from using Lightsail bandwidth with other services, you could put together an all-AWS package that has 150GB of high quality S3 storage and enough bandwidth to download it 2-3x per $5 (minimum order quantity 2). For a service like B2 you could store 250GB twice (each copy having its own cross-server RAID) and download it once for $5. At digitalocean $5 will get you 250GB of probably-redundant data with 1TB of bandwidth, though it eventually tapers off toward 167GB/$5.
Re: Avoid Git LFS if possible
#114Earlier quoted context omitted.
why are you committing packages?
I would assume to prevent situations like the left-pad incident.
PMs are made for managing and hosting packages, VCS are made for versioning source-code. If you're checking in packages into VC, you're going against the designs of both your PM and VCS. It's a bad idea. Don't.
If you for some reason require redundancy of a package repo, then host your own.
Re: Avoid Git LFS if possible
#115Earlier quoted context omitted.
So I decided to check this out. Used dd if=/dev/random to create a 100mb file, checked that in, used dd again to modify 10mb of that file, checked that in and the result were two 98mb objects. Tracking changes of binaries makes a lot of sense if you use that to only store incremental changes to the file. Git stores each modification of a binary file as a separate blob since it doesn't know how to track its changes. T…
Git LFS has the advantage of not pulling all versions of a large file, too. Instead, it only pulls the version it's checking out. In our project it helped dramatically as you only pull X MB instead of X * Y MB when a CI or developer clone the (already big) repo.
I prefer keeping large files out of source control and thus far I've not encountered a problem where their introduction has been required.
Re: Avoid Git LFS if possible
#116I keep hearing the mantra that "svn is better for large files than git" but never really understood why. To me a large file is a large file; if you make changes, worst case scenario you add the entire new file to the commit, best case you add some sort of binary diff. Does git do the former and svn the latter by any chance?
An svn working copy has one version stored locally. A git clone has all versions stored locally. All versions of a large file takes up lots of space.
I see. So the idea is not that svn's handling of large files at the repo-level is somehow better than that of a git repo per se, but that it's fine for the (possibly remote) svn repo to take the 'large file' hit, since the (presumably local) wc is disjoint from it, and thus unaffected in terms of local storage. Ok, that makes sense...
I've been using a decentralised svn workflow at work for so long, I didn't even think of this :)
Re: Avoid Git LFS if possible
#117If you have a hundred images in git, and one cannot be downloaded for any reason, git smudge will not be able to run, and you won't be able to git pull at all.
We had an image on AWS go bad, still not sure how. Our devs lost the ability to pull. Disabling LFS could not be done (because of rewriting history). "disable smudge" is not an official option, and none of the hacks work reliably. We finally excluded all images from smudge, and downloaded them with SFTP. Git status shows all the images as having changed, and we are downright unhappy...
It would be happy to hear that I just don't know how to use LFS - but even if so, that means the docs are woefully not useful.
I want to: 1) Tell LFS to get whatever files it could, and just throw a warning on issues. 2) If image is restored not using LFS, git should still know the file has not been modified (by comparing the checksum or whatever smudge would do).
Re: Avoid Git LFS if possible
#118Re: Avoid Git LFS if possible
#119Earlier quoted context omitted.
An svn working copy has one version stored locally. A git clone has all versions stored locally. All versions of a large file takes up lots of space.
I see. So the idea is not that svn's handling of large files at the repo-level is somehow better than that of a git repo per se, but that it's fine for the (possibly remote) svn repo to take the 'large file' hit, since the (presumably local) wc is disjoint from it, and thus unaffected in terms of local storage. Ok, that makes sense... I've been using a decentralised svn workflow at work for so long, I didn't even thi…
I suppose it depends how much disk you have :P
Re: Avoid Git LFS if possible
#120Earlier quoted context omitted.
Git LFS has the advantage of not pulling all versions of a large file, too. Instead, it only pulls the version it's checking out. In our project it helped dramatically as you only pull X MB instead of X * Y MB when a CI or developer clone the (already big) repo.
I prefer keeping large files out of source control and thus far I've not encountered a problem where their introduction has been required.
While I share the sentiment of keeping large files out of source control, one use-case I believe warrants having large files in source control is game development.