Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git-Annex

git-annex.branchable.com

Post reply on HN
Print view View original

21–30 of 64 posts

Re: Git-Annex

#21

Post by kajika91 »

I'm using my self-hosted forgejo. I don't see any benefit of git-annex over LFS so far, I'm not even sure I could setup annex as easily.

Digging a little bit I found that git-annex is coded in haskell (not a fan) and seems to be 50% slower (expected from haskell but also only 1 source so far so not really reliable).

I don't see appeal of the complexity of the commands, they probably serve a purpose. Once you opened a .gitattributes from git-LFS you pretty much know all you need and you barely need any commands anymore.

Also I like how setting up a .gitattribute makes everything transparent the same way .gitignore works. I don't see any equivalent with git-annex.

Lastly any "tutorial" or guide about git-annex that won't show me an equivalent of 'git lfs ls-files' will definitely not appeal to me. I'm a big user of 'git status' and 'git lfs ls-files' to check/re-check everything.

Re: Git-Annex

#22

Post by _Algernon_ »

nolist_policy wrote: post #10

I use git-annex to manage all my data on all my drives. It automatically keeps track of which files are on which drives, it ensures that there are enough copies and it checksums everything. It works perfectly with offline drives. git-annex can be a bit hard to grasp, so I suggest to create a throw-away repository, following the walkthrough[1] and try things out. See also workflows[2]. [1] https://git-annex.branchable…

I have thought about doing this in the past but ran into issues (one of them being the friction in permanently deleting files once added). I'd be curious how you use it if you have time to share.

Re: Git-Annex

#23

Post by stv0g »

kajika91 wrote: post #21

I'm using my self-hosted forgejo. I don't see any benefit of git-annex over LFS so far, I'm not even sure I could setup annex as easily. Digging a little bit I found that git-annex is coded in haskell (not a fan) and seems to be 50% slower (expected from haskell but also only 1 source so far so not really reliable). I don't see appeal of the complexity of the commands, they probably serve a purpose. Once you opened a…

There is a soft-fork of Forgejo which adds support for git-annex:

https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo

Re: Git-Annex

#24

Post by warp »

albertzeyer wrote: post #17
nolist_policy wrote: post #10

I use git-annex to manage all my data on all my drives. It automatically keeps track of which files are on which drives, it ensures that there are enough copies and it checksums everything. It works perfectly with offline drives. git-annex can be a bit hard to grasp, so I suggest to create a throw-away repository, following the walkthrough[1] and try things out. See also workflows[2]. [1] https://git-annex.branchable…

How much data do you have? I'm using git-annex on my photos, and that are around 100k-1M files, several TB of data, on a ZFS. In the beginning, everything was fine, but it starts to become increasingly slow, such that every operation takes several minutes (5-30 mins or so). I wonder a bit whether that is ZFS, or git-annex, or maybe my disk, or sth else.

My experience is the same, git-annex just doesn't work well with lots of small files. With annexes on slow USB disks, connected to a Raspberry Pi 3 or 4, I'm already annoyed when working with my largest annex (in file count) of 25000 files.

However, I mostly use annex as a way to archive stuff and make sure I have enough copies in distinct physical locations. So for photos I now just tar them up with one .tar file per family member per year. This works fine for for me for any data I want to keep safe but don't need to access directly very often.

Re: Git-Annex

#25

Post by avar »

ttiurani wrote: post #14

Relevant discussion 9 days ago about the new native git large object promisers in "The future of large files in Git is Git": https://news.ycombinator.com/item?id=44916783

Thanks, also not-so-relevant, for the reasons I noted in a comment in that thread: https://news.ycombinator.com/item?id=44922405

I.e. annex is really in a different problem space than "big files in git", despite the obvious overlap.

A good way to think about it is that git-annex is sort of a git-native and distributed solution to the storage problem at the "other side" ("server side") of something like LFS, and to reason about it from there.

Re: Git-Annex

#26

Post by avar »

kajika91 wrote: post #21

I'm using my self-hosted forgejo. I don't see any benefit of git-annex over LFS so far, I'm not even sure I could setup annex as easily. Digging a little bit I found that git-annex is coded in haskell (not a fan) and seems to be 50% slower (expected from haskell but also only 1 source so far so not really reliable). I don't see appeal of the complexity of the commands, they probably serve a purpose. Once you opened a…

Annex isn't slow because it's written in Haskell, it tends to be slow because of I/O and paranoia that's warranted as the default behavior in a distributed backup tool.

E.g. if you drop something it'll by default check the remotes it has access to for that content in real time, it can be many orders of magnitude faster to use --fast etc., to (somewhat unsafely) skip all that and trust whatever metadata you have a local copy of.

Re: Git-Annex

#27

Post by pinoy420 »

albertzeyer wrote: post #17
nolist_policy wrote: post #10

I use git-annex to manage all my data on all my drives. It automatically keeps track of which files are on which drives, it ensures that there are enough copies and it checksums everything. It works perfectly with offline drives. git-annex can be a bit hard to grasp, so I suggest to create a throw-away repository, following the walkthrough[1] and try things out. See also workflows[2]. [1] https://git-annex.branchable…

How much data do you have? I'm using git-annex on my photos, and that are around 100k-1M files, several TB of data, on a ZFS. In the beginning, everything was fine, but it starts to become increasingly slow, such that every operation takes several minutes (5-30 mins or so). I wonder a bit whether that is ZFS, or git-annex, or maybe my disk, or sth else.

[dead]

Re: Git-Annex

#28

Post by seanparsons »

kajika91 wrote: post #21

I'm using my self-hosted forgejo. I don't see any benefit of git-annex over LFS so far, I'm not even sure I could setup annex as easily. Digging a little bit I found that git-annex is coded in haskell (not a fan) and seems to be 50% slower (expected from haskell but also only 1 source so far so not really reliable). I don't see appeal of the complexity of the commands, they probably serve a purpose. Once you opened a…

LFS and git-annex have subtly different use cases in my experience. LFS is for users developing something with git that has large files in the repo like the classic game development example. git-annex is something you'd use to keep some important stuff backed up which happens to involve large files, like a home folder with music or whatever in it. In my case I do the latter.

Re: Git-Annex

#29

Post by matrss »

albertzeyer wrote: post #17
nolist_policy wrote: post #10

I use git-annex to manage all my data on all my drives. It automatically keeps track of which files are on which drives, it ensures that there are enough copies and it checksums everything. It works perfectly with offline drives. git-annex can be a bit hard to grasp, so I suggest to create a throw-away repository, following the walkthrough[1] and try things out. See also workflows[2]. [1] https://git-annex.branchable…

How much data do you have? I'm using git-annex on my photos, and that are around 100k-1M files, several TB of data, on a ZFS. In the beginning, everything was fine, but it starts to become increasingly slow, such that every operation takes several minutes (5-30 mins or so). I wonder a bit whether that is ZFS, or git-annex, or maybe my disk, or sth else.

I had tested a git-annex repository with about 1.5M files and it got pretty slow as well. The plain git repo size grew to multiple GiB and plain git operations were super slow, so I think this was mostly a git limitation. DataLad's approach of nested subdatasets (in practice git submodules where each submodule is a git-annex repository) can help, if it fits the data and workflows.

Re: Git-Annex

#30

Post by egwor »

albertzeyer wrote: post #17
nolist_policy wrote: post #10

I use git-annex to manage all my data on all my drives. It automatically keeps track of which files are on which drives, it ensures that there are enough copies and it checksums everything. It works perfectly with offline drives. git-annex can be a bit hard to grasp, so I suggest to create a throw-away repository, following the walkthrough[1] and try things out. See also workflows[2]. [1] https://git-annex.branchable…

How much data do you have? I'm using git-annex on my photos, and that are around 100k-1M files, several TB of data, on a ZFS. In the beginning, everything was fine, but it starts to become increasingly slow, such that every operation takes several minutes (5-30 mins or so). I wonder a bit whether that is ZFS, or git-annex, or maybe my disk, or sth else.

One thing to check is whether any security/monitoring software might be causing issues. Since there are so many files in git repos, it can put a lot of load on that type of software.
Post reply on HN