Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Avoid Git LFS if possible

gregoryszorc.com

Post reply on HN
Print view View original

131–140 of 142 posts

Re: Avoid Git LFS if possible

#131

Post by remram »

Izkata wrote: post #73
remram wrote: post #56

Earlier quoted context omitted.

Is git-annex still alive? Last time I tried to use it, it was very rough, and the official wiki (that serves as doc + bug tracker) gives database errors trying to create an account. Details: I wanted to have a remote I can push to but anonymous users can only pull from, couldn't piece it together.

I've always liked the simplicity of git-fat [0]: * Initial setup includes git filter rules so that "git add" automatically uses get-fat for matching files (no need to remember to invoke git-fat when adding/changing files). * It works by rsync'ing to/from the remote. The setup for this is in a single ".gitfat" file, separate from the filter rules. * You do need to run "git fat push" and "git fat pull"; this can probab…

Thanks I didn't know about this one! It seems to only support rsync though, so using it for public repositories would be difficult.

Re: Avoid Git LFS if possible

#132

Post by IshKebab »

cerved wrote: post #129
IshKebab wrote: post #121

Earlier quoted context omitted.

Because it's something that lots of people want to use Git for, and because it would work better if it were a core part of Git.

okay, but why? Lots of people want to use document databases as if they were relational, lots of people want to use their RDBMS as a file server and lots of people use spreadsheets for just about everything. Lots of people wanting to use a product in certain way doesn't mean it's a good idea, nor that someone else should make that work for them that way

Ok but those are unreasonable things to do. Versioning large binary files is clearly not an unreasonable thing to want to do.

Re: Avoid Git LFS if possible

#133

Post by IshKebab »

swiley wrote: post #126
IshKebab wrote: post #121

Earlier quoted context omitted.

Because it's something that lots of people want to use Git for, and because it would work better if it were a core part of Git.

Maybe I want to use vim as an interactive WYSIWYG modeling program but that doesn't mean it's in the scope of the project. LFS belongs in your build scripts, the model the git extension use doesn't even match the git VCS model.

Version control of large files is clearly a reasonable thing for a version control system to do.

I think if you introspect you'll see you are defending a flaw in something you like with spurious technical objections because you don't want to admit it isn't perfect. That's understandable. Happens a lot.

Re: Avoid Git LFS if possible

#134

Post by swiley »

IshKebab wrote: post #133
swiley wrote: post #126

Earlier quoted context omitted.

Maybe I want to use vim as an interactive WYSIWYG modeling program but that doesn't mean it's in the scope of the project. LFS belongs in your build scripts, the model the git extension use doesn't even match the git VCS model.

Version control of large files is clearly a reasonable thing for a version control system to do. I think if you introspect you'll see you are defending a flaw in something you like with spurious technical objections because you don't want to admit it isn't perfect. That's understandable. Happens a lot.

I didn't say versioning large files isn't a reasonable thing to do, I said it doesn't make sense in git.

How is it supposed to work? The LFS way where you're storing a pointer to an external http resource? Just put a script in your repo to fetch that then.

Or maybe stick the data in git's merkle tree and have a really slow repo? Why bother with LFS then?

Re: Avoid Git LFS if possible

#135

Post by IshKebab »

swiley wrote: post #134
IshKebab wrote: post #133

Earlier quoted context omitted.

Version control of large files is clearly a reasonable thing for a version control system to do. I think if you introspect you'll see you are defending a flaw in something you like with spurious technical objections because you don't want to admit it isn't perfect. That's understandable. Happens a lot.

I didn't say versioning large files isn't a reasonable thing to do, I said it doesn't make sense in git. How is it supposed to work? The LFS way where you're storing a pointer to an external http resource? Just put a script in your repo to fetch that then. Or maybe stick the data in git's merkle tree and have a really slow repo? Why bother with LFS then?

> How is it supposed to work?

One obvious improvement would be for Git to use the hash of the object rather than the pointer file when calculating tree hashes. That makes the storage method for the actual files independent of the commit hashes.

People have mentioned several other VCSs in this thread that do it already.

Re: Avoid Git LFS if possible

#136

Post by Danieru »

acdha wrote: post #67
Danieru wrote: post #64

Earlier quoted context omitted.

Just be thankful you are not on Gitlab. We pay $60/year for 10GB. I need the data for my team, so we pay, but I use it as an excuse to force the team to clean up data every so often. As a game company we need large repos. It was either Gitlab or Azure DevOps.

$60/year is $5/month. You said “game” and “company” — how is that not a tiny fraction of your total costs?

Can I afford it? Yes. Do I need it? Yes.

An I happy paying 6 dollars a year for a single gigabyte: I'd rather pay less.

Edit: I pay attention to costs, and repo size is the sort of thing which has a. Habit of growing. All else being equal I'd prefer my team receive the money. A dollar I can give to my team/employees feels good, a dollar paying for over priced storage feels bad.

Re: Avoid Git LFS if possible

#137

Post by acdha »

Danieru wrote: post #136
acdha wrote: post #67

Earlier quoted context omitted.

$60/year is $5/month. You said “game” and “company” — how is that not a tiny fraction of your total costs?

Can I afford it? Yes. Do I need it? Yes. An I happy paying 6 dollars a year for a single gigabyte: I'd rather pay less. Edit: I pay attention to costs, and repo size is the sort of thing which has a. Habit of growing. All else being equal I'd prefer my team receive the money. A dollar I can give to my team/employees feels good, a dollar paying for over priced storage feels bad.

Doesn't that also apply to, say, GitLab paying their team? The pricing for a highly-available service isn't as cheap as it could be but it doesn't seem terribly far off of S3's pricing.

I know that a small game studio must have a tight budget — and yours looks like a really interesting project (just subscribed) — but it seemed like an awfully strong objection to what I would have assumed would be a small fraction of your total expenses.

Re: Avoid Git LFS if possible

#138

Post by blablabla123 »

tardyp wrote: post #109
wokwokwok wrote: post #52

I despise LFS. I’m sure that if you know how to use it... maybe... you can figure it out. That said; here’s my battle story: Estimate the time it’ll take to move all our repositories from a to b they said. Us: with all branches? Them: just main and develop. Us: you just clone and push to the new origin, it’s not zero but it’s trivial. Weeks later... Yeah. LFS is now banned. LFS is not a distributed version control sy…

You can easily mirror LFS objects by just using git lfs fetch --all every modern git hosting servers now have support for LFS directly inside the git server (gitlab, github, gerrit to my knowledge) This solve the authentication issue nicely so make it easy for developers. git lfs starts to be adopted by vendors and becomes usable. It solves real problem when you are tired of having to double your git server CPUs ever…

> It solves real problem when you are tired of having to double your git server CPUs every 6 months

Putting blobs into an SCM was always a bad idea, but in git it's particularly bad because the whole tree is always checked out at once. (I think last year a major change was added, that makes blob handling slightly more efficient though)

Still, I think there is still no independently developed git lfs server. I know, most people don't run git servers themselves but in the LFS use case it actually makes sense. There is also git annex which is completely open and free but adoption is very poor and the handling is even more obscure.

Re: Avoid Git LFS if possible

#139

Post by Dylan16807 »

acdha wrote: post #137
Danieru wrote: post #136

Earlier quoted context omitted.

Can I afford it? Yes. Do I need it? Yes. An I happy paying 6 dollars a year for a single gigabyte: I'd rather pay less. Edit: I pay attention to costs, and repo size is the sort of thing which has a. Habit of growing. All else being equal I'd prefer my team receive the money. A dollar I can give to my team/employees feels good, a dollar paying for over priced storage feels bad.

Doesn't that also apply to, say, GitLab paying their team? The pricing for a highly-available service isn't as cheap as it could be but it doesn't seem terribly far off of S3's pricing. I know that a small game studio must have a tight budget — and yours looks like a really interesting project (just subscribed) — but it seemed like an awfully strong objection to what I would have assumed would be a small fraction of…

It's good for GitLab to pay their teams, but that stops being a justification to pay them more once the profit margin for a service gets high enough.

> it doesn't seem terribly far off of S3's pricing

GitHub's offering is close to S3 but only because AWS charges so much for bandwidth. The storage portion is less than a quarter of the equivalent bill.

And then GitLab is charging 5x as much as GitHub.

Re: Avoid Git LFS if possible

#140

Post by cerved »

IshKebab wrote: post #132
cerved wrote: post #129

Earlier quoted context omitted.

okay, but why? Lots of people want to use document databases as if they were relational, lots of people want to use their RDBMS as a file server and lots of people use spreadsheets for just about everything. Lots of people wanting to use a product in certain way doesn't mean it's a good idea, nor that someone else should make that work for them that way

Ok but those are unreasonable things to do. Versioning large binary files is clearly not an unreasonable thing to want to do.

No but that doesn't mean a decentralized VCS is the best tool for the job. A file server may be sufficient and more efficient
Post reply on HN