Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git-Annex

git-annex.branchable.com

Post reply on HN
Print view View original

31–40 of 64 posts

Re: Git-Annex

#31

Post by goku12 »

My only problem with git-annex is Haskell. I don't hate the language itself, but the sheer number of dependencies it has to install is staggering. Many of those dependencies are not used by anything else, or may be incompatible versions when more than one application uses it. The pain is when you install them using the system package manager. Just two Haskell applications - annex and pandoc - are enough to fill your daily updates with may be a dozen little Haskell packages. God forbid you're on a distro that installs from source!

It's quite safe to just statically link most, if not all of them directly into the application, even when some of them are shared by other applications. I have seen this complaint repeated a few times. The reply from the Haskelliers seem to be that this is for the fine grained modularity of the library ecosystem. But why do they treat it like everything starts and ends with Haskell? Sometimes, there are other priorities like system administration. None of the other compiled languages have this problem - Rust, Go, Zig, ... Even plain old C and C++ aren't this frustrating with dependencies.

I need to clarify that I'm not hostile towards the Haskell language, its ecosystem and its users. It's something I plan to learn myself. But why does this problem exist? And is there a solution?

Re: Git-Annex

#32

Post by aragilar »

goku12 wrote: post #31

My only problem with git-annex is Haskell. I don't hate the language itself, but the sheer number of dependencies it has to install is staggering. Many of those dependencies are not used by anything else, or may be incompatible versions when more than one application uses it. The pain is when you install them using the system package manager. Just two Haskell applications - annex and pandoc - are enough to fill your…

Which package manager are you using? I've not seen any issues with apt-based systems with Haskell?

Re: Git-Annex

#33

Post by IsTom »

goku12 wrote: post #31

My only problem with git-annex is Haskell. I don't hate the language itself, but the sheer number of dependencies it has to install is staggering. Many of those dependencies are not used by anything else, or may be incompatible versions when more than one application uses it. The pain is when you install them using the system package manager. Just two Haskell applications - annex and pandoc - are enough to fill your…

> It's quite safe to just statically link most, if not all of them directly into the application

If you're talking about distro's repos, isn't this a matter of distro and package manager policy?

Re: Git-Annex

#34

Post by aragilar »

kajika91 wrote: post #21

I'm using my self-hosted forgejo. I don't see any benefit of git-annex over LFS so far, I'm not even sure I could setup annex as easily. Digging a little bit I found that git-annex is coded in haskell (not a fan) and seems to be 50% slower (expected from haskell but also only 1 source so far so not really reliable). I don't see appeal of the complexity of the commands, they probably serve a purpose. Once you opened a…

How big are the repos you have? The largest git-annex repo I have is multiple TB (spread across multiple systems), with some files 10s of GB.

I'm not sure what you are doing, but from looking at the git-lfs-ls-files manpage `git annex list --in here` is likely what you want?

Re: Git-Annex

#35

Post by aragilar »

seanparsons wrote: post #28
kajika91 wrote: post #21

I'm using my self-hosted forgejo. I don't see any benefit of git-annex over LFS so far, I'm not even sure I could setup annex as easily. Digging a little bit I found that git-annex is coded in haskell (not a fan) and seems to be 50% slower (expected from haskell but also only 1 source so far so not really reliable). I don't see appeal of the complexity of the commands, they probably serve a purpose. Once you opened a…

LFS and git-annex have subtly different use cases in my experience. LFS is for users developing something with git that has large files in the repo like the classic game development example. git-annex is something you'd use to keep some important stuff backed up which happens to involve large files, like a home folder with music or whatever in it. In my case I do the latter.

What it works really well at is storing research data. LFS can't upload to arbitrary webdav/S3/sharepoint/other random cloud service.

Re: Git-Annex

#36

Post by andunie »

I've used this for years, but to me the big selling point was integration with cloud storage providers as a means of backup. That, however, was always flaky and dependent on unmaintained third-party plugins. I think there was also a bug at some point that caused some data inconsistencies, so eventually I stopped.

Does anyone know if the situation has improved on that front in the past 5 years?

Re: Git-Annex

#37

Post by goku12 »

aragilar wrote: post #32
goku12 wrote: post #31

My only problem with git-annex is Haskell. I don't hate the language itself, but the sheer number of dependencies it has to install is staggering. Many of those dependencies are not used by anything else, or may be incompatible versions when more than one application uses it. The pain is when you install them using the system package manager. Just two Haskell applications - annex and pandoc - are enough to fill your…

Which package manager are you using? I've not seen any issues with apt-based systems with Haskell?

I used to have issues on Arch/pacman. Now on ebuilds/Gentoo.

Re: Git-Annex

#38

Post by Borg3 »

albertzeyer wrote: post #17
nolist_policy wrote: post #10

I use git-annex to manage all my data on all my drives. It automatically keeps track of which files are on which drives, it ensures that there are enough copies and it checksums everything. It works perfectly with offline drives. git-annex can be a bit hard to grasp, so I suggest to create a throw-away repository, following the walkthrough[1] and try things out. See also workflows[2]. [1] https://git-annex.branchable…

How much data do you have? I'm using git-annex on my photos, and that are around 100k-1M files, several TB of data, on a ZFS. In the beginning, everything was fine, but it starts to become increasingly slow, such that every operation takes several minutes (5-30 mins or so). I wonder a bit whether that is ZFS, or git-annex, or maybe my disk, or sth else.

Why? WHY?! Why the heck are you using (D)VFS on your immutable data? What is the reasoning? That stuff is immutable and usually incremental.. Just throw proper syncing algoritm on it and sync w/ backups.. thats all. I wonder aby logic behind this...

Docs and other files you often change is completly different story. This is where DVFS shines. I wrote my own very simple DVFS exacly for that case. You just create directory, init repo manager.. and vioala.. Disk wide VFS is kinda useless as most of your data there just sits..

Re: Git-Annex

#39

Post by matrss »

andunie wrote: post #36

I've used this for years, but to me the big selling point was integration with cloud storage providers as a means of backup. That, however, was always flaky and dependent on unmaintained third-party plugins. I think there was also a bug at some point that caused some data inconsistencies, so eventually I stopped. Does anyone know if the situation has improved on that front in the past 5 years?

Depends on the cloud storage provider, I think. The best chances are with those that support the more standard protocols like S3, webdav, sftp, etc.. A relatively new development is the special remote built into rclone, which should be better maintained than some other third-party special remotes and provides access to all rclone-supported remotes.

Re: Git-Annex

#40

Post by BrandiATMuhkuh »

Does this also work if I have data on SharePoint, DropBox, etc. and want to pull them (sync with local machine)?

My use case is mostly ETL related, where I want to pull all customers data (enterprise customer) so I can process them. But also keep the data updated, hence pull?

Post reply on HN