Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git-Annex

git-annex.branchable.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: Git-Annex

#51

Post by fer »

alexdme wrote: post #48
Borg3 wrote: post #38

Earlier quoted context omitted.

Why? WHY?! Why the heck are you using (D)VFS on your immutable data? What is the reasoning? That stuff is immutable and usually incremental.. Just throw proper syncing algoritm on it and sync w/ backups.. thats all. I wonder aby logic behind this... Docs and other files you often change is completly different story. This is where DVFS shines. I wrote my own very simple DVFS exacly for that case. You just create direc…

I also used to use git-annex on my photos, ended up getting frustrated with how slow it was and wrote aegis[1] to solve my use case. I wrote a bit about why in the readme (see archiving vs backup). In my opinion, syncing, snapshots, and backup tools like restic are great but fundamentally solve a different problem from what I want out of an archive tool like aegis, git-annex, or boar[2]. I want my backups to be autom…

While I understand why git-annex wouldn't work for you, what gaps did you find in boar?

Re: Git-Annex

#52

Post by gradientsrneat »

matrss wrote: post #44
BrandiATMuhkuh wrote: post #40

Does this also work if I have data on SharePoint, DropBox, etc. and want to pull them (sync with local machine)? My use case is mostly ETL related, where I want to pull all customers data (enterprise customer) so I can process them. But also keep the data updated, hence pull?

In an ideal world the rclone special remote would support git-annex' importtree feature. Then you could periodically run `git annex import : --from ` to "pull" from those remotes (it is not really a pull as you aren't fetching version-controlled data from those remotes, rather you are importing from a non-version-controlled source and record the result as a new revision). Unfortunately this is not (yet?) supported I…

git-annex does support rclone as a special remote iirc

Re: Git-Annex

#53

Post by seanw444 »

alexdme wrote: post #48
Borg3 wrote: post #38

Earlier quoted context omitted.

Why? WHY?! Why the heck are you using (D)VFS on your immutable data? What is the reasoning? That stuff is immutable and usually incremental.. Just throw proper syncing algoritm on it and sync w/ backups.. thats all. I wonder aby logic behind this... Docs and other files you often change is completly different story. This is where DVFS shines. I wrote my own very simple DVFS exacly for that case. You just create direc…

I also used to use git-annex on my photos, ended up getting frustrated with how slow it was and wrote aegis[1] to solve my use case. I wrote a bit about why in the readme (see archiving vs backup). In my opinion, syncing, snapshots, and backup tools like restic are great but fundamentally solve a different problem from what I want out of an archive tool like aegis, git-annex, or boar[2]. I want my backups to be autom…

I might have to give aegis a try.

Re: Git-Annex

#54

Post by matrss »

gradientsrneat wrote: post #52
matrss wrote: post #44

Earlier quoted context omitted.

In an ideal world the rclone special remote would support git-annex' importtree feature. Then you could periodically run `git annex import : --from ` to "pull" from those remotes (it is not really a pull as you aren't fetching version-controlled data from those remotes, rather you are importing from a non-version-controlled source and record the result as a new revision). Unfortunately this is not (yet?) supported I…

git-annex does support rclone as a special remote iirc

Yes it does, but I don't think the special remote supports the importtree feature, which would be necessary for this.

Re: Git-Annex

#55

Post by geephroh »

Munksgaard wrote: post #13

Git-Annex is a cool piece of technology, but my impression is that it works best for single-user repositories. So for instance, as @nolist_policy described in a sibling comment, managing all your personal files, documents, music, etc. across many different devices. I tried using it for syncing large files in a collaborative repository, and the use of "magic" branches didn't seem to scale well.

YMMV, but it works for my org. We're an archival institution and have been using git-annex for more than a decade as the storage backend for a digital repository system designed for long-term, robust preservation. Admittedly, we only have 15-20 staff; but +30TB of data, ~750K files (binaries + metadata), across hundreds of collection repos.

Re: Git-Annex

#56

Post by mananaysiempre »

Borg3 wrote: post #38
albertzeyer wrote: post #17

Earlier quoted context omitted.

How much data do you have? I'm using git-annex on my photos, and that are around 100k-1M files, several TB of data, on a ZFS. In the beginning, everything was fine, but it starts to become increasingly slow, such that every operation takes several minutes (5-30 mins or so). I wonder a bit whether that is ZFS, or git-annex, or maybe my disk, or sth else.

Why? WHY?! Why the heck are you using (D)VFS on your immutable data? What is the reasoning? That stuff is immutable and usually incremental.. Just throw proper syncing algoritm on it and sync w/ backups.. thats all. I wonder aby logic behind this... Docs and other files you often change is completly different story. This is where DVFS shines. I wrote my own very simple DVFS exacly for that case. You just create direc…

> Why the heck are you using (D)VFS on your immutable data?

Git-annex does not put your data in Git. What it tracks using Git is what’s available where, updating that data on an eventually consistent basis whenever two storage sites come into contact. It also borrows Git functionality for tracking moves, renames, etc. The object-storage parts, on the other hand, are essentially a separate content-addressable store from the normal one Git uses for its objects.

(The concrete form of a git-annex worktree is a Git-tracked tree of symlinks pointing to .git/annex/objects under the repo root, where the actual data is stored as read-only files, plus location-tracking data indexed by object hash in a separate branch called “git-annex”, which the git-annex commands manipulate using special merge strategies.)

Re: Git-Annex

#57

Post by the_absurdist »

Ironically, I just spent a day last weekend writing my own version control system for large files

I dislike git-annex that much.

- it converts your files into blobs and bloats your file system

- As others have previously alluded, my primary use case is to ensure sync between distributed files, not version them (why would anyone possibly need that??)

- You can use AI to build a python based solution that will hash your files and put them into a lookup table, then create some helper methods to sync sources using rclone

Far simpler and more efficient methods exist.

Re: Git-Annex

#58

Post by goku12 »

IsTom wrote: post #33
goku12 wrote: post #31

My only problem with git-annex is Haskell. I don't hate the language itself, but the sheer number of dependencies it has to install is staggering. Many of those dependencies are not used by anything else, or may be incompatible versions when more than one application uses it. The pain is when you install them using the system package manager. Just two Haskell applications - annex and pandoc - are enough to fill your…

> It's quite safe to just statically link most, if not all of them directly into the application If you're talking about distro's repos, isn't this a matter of distro and package manager policy?

Very likely. But then, why is this an issue with Haskell alone?

Re: Git-Annex

#59

Post by goku12 »

GZGavinZhao wrote: post #49
goku12 wrote: post #31

My only problem with git-annex is Haskell. I don't hate the language itself, but the sheer number of dependencies it has to install is staggering. Many of those dependencies are not used by anything else, or may be incompatible versions when more than one application uses it. The pain is when you install them using the system package manager. Just two Haskell applications - annex and pandoc - are enough to fill your…

The Haskell tooling already supports statically linking in the dependencies, I maintain the Haskell stack for Solus and for pandoc we just have a single binary that only depends on libc, all other Haskell dependencies are statically linked inside the binary just like how Rust dependencies work. So it's definitely doable. I think it's more of the distro's maintainers' choice. For Solus the amount of dependencies was j…

Why do some distro maintainers choose otherwise? I have seen some users complaint about this before, without any correction being done. So my guess is that there is something that forces those distro maintainers to make that choice.

Re: Git-Annex

#60

Post by goku12 »

wonger_ wrote: post #46
goku12 wrote: post #31

My only problem with git-annex is Haskell. I don't hate the language itself, but the sheer number of dependencies it has to install is staggering. Many of those dependencies are not used by anything else, or may be incompatible versions when more than one application uses it. The pain is when you install them using the system package manager. Just two Haskell applications - annex and pandoc - are enough to fill your…

Yes! Every time I `pacman -Syu`, half of the updates are Haskell packages. I think from pandoc and shellcheck iirc?

Pandoc and git-annex in my case. Basically any substantial application written in Haskell.
Post reply on HN