Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Git-Annex

git-annex.branchable.com

Post reply on HN
Print view View original

41–50 of 64 posts

Re: Git-Annex

#41

Post by zeendo »

goku12 wrote: post #31

My only problem with git-annex is Haskell. I don't hate the language itself, but the sheer number of dependencies it has to install is staggering. Many of those dependencies are not used by anything else, or may be incompatible versions when more than one application uses it. The pain is when you install them using the system package manager. Just two Haskell applications - annex and pandoc - are enough to fill your…

As a full time Haskell developer, I have a similar aversion to Haskell-based distro packages which aren't statically linked.

There ARE statically linked Haskell packages in the AUR so it's at least feasible. I haven't even dug into the conversations around why packagers are insisting on dynamic linking of distro packages - I just avoid them for the same reasons you mention.

I can't really speak confidently to why it is exactly - I can only guess. Clearly dynamic linking makes sense in a lot of cases for internal application distribution - which is where Haskell is often used - so maybe people are incorrectly projecting that onto distro packages?

Re: Git-Annex

#42

Post by ncann »

Borg3 wrote: post #38
albertzeyer wrote: post #17

Earlier quoted context omitted.

How much data do you have? I'm using git-annex on my photos, and that are around 100k-1M files, several TB of data, on a ZFS. In the beginning, everything was fine, but it starts to become increasingly slow, such that every operation takes several minutes (5-30 mins or so). I wonder a bit whether that is ZFS, or git-annex, or maybe my disk, or sth else.

Why? WHY?! Why the heck are you using (D)VFS on your immutable data? What is the reasoning? That stuff is immutable and usually incremental.. Just throw proper syncing algoritm on it and sync w/ backups.. thats all. I wonder aby logic behind this... Docs and other files you often change is completly different story. This is where DVFS shines. I wrote my own very simple DVFS exacly for that case. You just create direc…

I am looking into using Git for my photos/videos backup external HDDs and the reasoning is simple. It's not about keeping track of changes within the files themselves since like you said, they (almost) never change. Rather, it's about keeping track of changes in _folders_. That is, I want to keep track of when I last copied images from my phones, cameras, etc. to my HDDs, which folders did I touch, if I reorganized existing files into a different folder structure then what are the changes, etc. Also it acts as a rollback mechanism if I ever fat finger and delete something accidentally. I wonder if there's a better tool for this though

Re: Git-Annex

#43

Post by albertzeyer »

Borg3 wrote: post #38
albertzeyer wrote: post #17

Earlier quoted context omitted.

How much data do you have? I'm using git-annex on my photos, and that are around 100k-1M files, several TB of data, on a ZFS. In the beginning, everything was fine, but it starts to become increasingly slow, such that every operation takes several minutes (5-30 mins or so). I wonder a bit whether that is ZFS, or git-annex, or maybe my disk, or sth else.

Why? WHY?! Why the heck are you using (D)VFS on your immutable data? What is the reasoning? That stuff is immutable and usually incremental.. Just throw proper syncing algoritm on it and sync w/ backups.. thats all. I wonder aby logic behind this... Docs and other files you often change is completly different story. This is where DVFS shines. I wrote my own very simple DVFS exacly for that case. You just create direc…

I don't really need the versioning aspect too much, but sometimes I modify the photos a bit (e.g. rotating or so). But all the other things are relevant for me, like having it distributed, syncing, only partially having the data on a particular node, etc.

So, what solution would be better for that? In the end it seems that other solutions provide a similar set of features. E.g. Syncthing.

But what's the downside with Git-annex over Syncthing or other solutions?

Re: Git-Annex

#44

Post by matrss »

BrandiATMuhkuh wrote: post #40

Does this also work if I have data on SharePoint, DropBox, etc. and want to pull them (sync with local machine)? My use case is mostly ETL related, where I want to pull all customers data (enterprise customer) so I can process them. But also keep the data updated, hence pull?

In an ideal world the rclone special remote would support git-annex' importtree feature. Then you could periodically run `git annex import : --from ` to "pull" from those remotes (it is not really a pull as you aren't fetching version-controlled data from those remotes, rather you are importing from a non-version-controlled source and record the result as a new revision).

Unfortunately this is not (yet?) supported I think. But you could also just do something like this: `rclone copy/sync : ./ && git annex add ./ && git commit -m `.

Re: Git-Annex

#45

Post by Borg3 »

ncann wrote: post #42
Borg3 wrote: post #38

Earlier quoted context omitted.

Why? WHY?! Why the heck are you using (D)VFS on your immutable data? What is the reasoning? That stuff is immutable and usually incremental.. Just throw proper syncing algoritm on it and sync w/ backups.. thats all. I wonder aby logic behind this... Docs and other files you often change is completly different story. This is where DVFS shines. I wrote my own very simple DVFS exacly for that case. You just create direc…

I am looking into using Git for my photos/videos backup external HDDs and the reasoning is simple. It's not about keeping track of changes within the files themselves since like you said, they (almost) never change. Rather, it's about keeping track of changes in _folders_. That is, I want to keep track of when I last copied images from my phones, cameras, etc. to my HDDs, which folders did I touch, if I reorganized e…

Then I think some syncing software like rsync will probably be better. Now sure how often you keep changing archived folders. I split that work TRASH like dirs and archives. When I done w/ files, I move them out of TRASH do proper place and that it. I prefer KISS aproach, but whatever works for you :)

Re: Git-Annex

#46

Post by wonger_ »

goku12 wrote: post #31

My only problem with git-annex is Haskell. I don't hate the language itself, but the sheer number of dependencies it has to install is staggering. Many of those dependencies are not used by anything else, or may be incompatible versions when more than one application uses it. The pain is when you install them using the system package manager. Just two Haskell applications - annex and pandoc - are enough to fill your…

Yes! Every time I `pacman -Syu`, half of the updates are Haskell packages. I think from pandoc and shellcheck iirc?

Re: Git-Annex

#47

Post by Borg3 »

albertzeyer wrote: post #43
Borg3 wrote: post #38

Earlier quoted context omitted.

Why? WHY?! Why the heck are you using (D)VFS on your immutable data? What is the reasoning? That stuff is immutable and usually incremental.. Just throw proper syncing algoritm on it and sync w/ backups.. thats all. I wonder aby logic behind this... Docs and other files you often change is completly different story. This is where DVFS shines. I wrote my own very simple DVFS exacly for that case. You just create direc…

I don't really need the versioning aspect too much, but sometimes I modify the photos a bit (e.g. rotating or so). But all the other things are relevant for me, like having it distributed, syncing, only partially having the data on a particular node, etc. So, what solution would be better for that? In the end it seems that other solutions provide a similar set of features. E.g. Syncthing. But what's the downside with…

If you want two-way distributed syncing, that is a bit more complicated and error prone, but most tools support it, even rsync. Simpler aproach is to have central primary node (whatever it desktop or storage) when you sync copy data and sync it to backups.

As I said, handling immutable data (incremental) is easy. You just copy and sync. Kinda trival. The problem I had personaly was all the importand docs (and similar) files I work on. First, I wanted snapshots and history, in case of some mistake or failure. Data checksuming, because they are importand. Also, full peer2peer syncing because I have desktop, servers, VMs, laptop, so I want to sync data around. And because I really like GIT, great tool for VCS, I wanted something similar but for generic binary data. Hence I interested in DVFS system. First I wanted full blown mountable DVFS system, but that is complicated and much harder to make it portable.. Repository aproach is easy to implement and is portable (Cygwin, Linux, UNIX, Posix). Works like a charm.

As for downside, If you think git-annex will work for you, just use it :) For me, it was far too complicated (too much moving parts) even for my DVFS usecase. For immutable data is absolutly overkill, to keep 100s of GBs of data there. I just sync :)

Re: Git-Annex

#48

Post by alexdme »

Borg3 wrote: post #38
albertzeyer wrote: post #17

Earlier quoted context omitted.

How much data do you have? I'm using git-annex on my photos, and that are around 100k-1M files, several TB of data, on a ZFS. In the beginning, everything was fine, but it starts to become increasingly slow, such that every operation takes several minutes (5-30 mins or so). I wonder a bit whether that is ZFS, or git-annex, or maybe my disk, or sth else.

Why? WHY?! Why the heck are you using (D)VFS on your immutable data? What is the reasoning? That stuff is immutable and usually incremental.. Just throw proper syncing algoritm on it and sync w/ backups.. thats all. I wonder aby logic behind this... Docs and other files you often change is completly different story. This is where DVFS shines. I wrote my own very simple DVFS exacly for that case. You just create direc…

I also used to use git-annex on my photos, ended up getting frustrated with how slow it was and wrote aegis[1] to solve my use case.

I wrote a bit about why in the readme (see archiving vs backup). In my opinion, syncing, snapshots, and backup tools like restic are great but fundamentally solve a different problem from what I want out of an archive tool like aegis, git-annex, or boar[2].

I want my backups to be automatic and transparent, for that restic is a great tool. But for my photos, my important documents and other immutable data, I want to manually accept or reject any change that happens to them, since I might not always notice when something changes. For example if I fat finger an rm, or a bug in a program overrides something and I don't notice.

[1]: https://git.sr.ht/~alexdavid/aegis

[2]: https://github.com/mekberg/boar

Re: Git-Annex

#49

Post by GZGavinZhao »

goku12 wrote: post #31

My only problem with git-annex is Haskell. I don't hate the language itself, but the sheer number of dependencies it has to install is staggering. Many of those dependencies are not used by anything else, or may be incompatible versions when more than one application uses it. The pain is when you install them using the system package manager. Just two Haskell applications - annex and pandoc - are enough to fill your…

The Haskell tooling already supports statically linking in the dependencies, I maintain the Haskell stack for Solus and for pandoc we just have a single binary that only depends on libc, all other Haskell dependencies are statically linked inside the binary just like how Rust dependencies work. So it's definitely doable.

I think it's more of the distro's maintainers' choice. For Solus the amount of dependencies was just too much for me to handle so we resorted to static linking.

Re: Git-Annex

#50

Post by kstrauser »

internet_points wrote: post #20

The page doesn't say it, but git-annex was created by https://www.patreon.com/joeyh who also made the wonderful https://joeyh.name/code/moreutils/ and https://etckeeper.branchable.com/

More famously (to me), he was a core Debian contributor to Debian for a couple of decades, starting in 1996. A pretty big chunk of what we think of as Linux came from his keyboard.
Post reply on HN