Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unison File Synchronizer

cis.upenn.edu

Post reply on HN
Print view View original

91–100 of 102 posts

Re: Unison File Synchronizer

#91

Post by nelgaard »

rwmj wrote: post #10
avhception wrote: post #4

I can't praise Unison enough. I've been using it every day for over 10 years and is easily one of the most useful pieces of software I've ever come across. I've donated multiple times over the years.

I'm a maintainer of the Unison package in Fedora and I'd love to know which version(s) of Unison you use day to day. In Fedora (and I think this applies in Debian too) we have to maintain 3 versions because Unison isn't interoperable across minor releases. For this reason we package 2.13, 2.27 and 2.40, and I think there is discussion about packaging the latest release too. Keeping these ancient (esp 2.13) versions g…

2.48.4 mostly. I use it on different versions of Debian and Ubuntu. For a while I had to compile Unison from source to make it work everywhere.

Re: Unison File Synchronizer

#92

Post by rstuart4133 »

mjn wrote: post #21
LeoPanthera wrote: post #19

Earlier quoted context omitted.

This incompatibility is one of the reasons why I reduced my dependency on it, though I still use it for minor things. The last straw came when it turns out that unison was incompatible with the same version, if that version had been built on a different system. I can't remember the details but there was some library version difference that results in a unison that has the same version number but wouldn't talk to one…

That can happen because unison uses OCaml's built in serialization, which isn't guaranteed to be stable across OCaml versions. It doesn't often change, but the representation of arrays was changed in OCaml 4.02, which meant unison compiled with 4.01 couldn't sync with unison compiled with 4.02. Debian bug discussing the issue: https://bugs.debian.org/802919

> which isn't guaranteed to be stable across OCaml versions

Oh, so OCaml is a toy language from academics. I've read so many good things about it over the years, but no one mentioned that aspect of it. There should be a law about putting a warning in Big Red Letters on the box so someone doesn't make the mistake of using it in a large, long lived project.

Re: Unison File Synchronizer

#93

Post by exadeci »

stavros wrote: post #65
comex wrote: post #39

Earlier quoted context omitted.

I recently switched from Syncthing to Unison, after getting tired of Syncthing either taking forever to sync anything or getting stuck altogether, with very little visibility or control over what it was doing. So far Unison (with fsmonitor) has ‘just worked’, with files updating immediately on save as I’d expect. (That said, one of the problems I encountered with Syncthing seemed to be that it wouldn’t properly recon…

Huh, I've had the opposite experience with Syncthing. It's basically just worked, it's been bulletproof. I'm not sure why the disparity.

Syncthing can behave oddly, sometimes it works great and others it just starts looping and doing nothing.

Of course I've not tried it in a while because of those issues so it could be fixed now.

Re: Unison File Synchronizer

#94

Post by ernst_klim »

rstuart4133 wrote: post #92
mjn wrote: post #21

Earlier quoted context omitted.

That can happen because unison uses OCaml's built in serialization, which isn't guaranteed to be stable across OCaml versions. It doesn't often change, but the representation of arrays was changed in OCaml 4.02, which meant unison compiled with 4.01 couldn't sync with unison compiled with 4.02. Debian bug discussing the issue: https://bugs.debian.org/802919

> which isn't guaranteed to be stable across OCaml versions Oh, so OCaml is a toy language from academics. I've read so many good things about it over the years, but no one mentioned that aspect of it. There should be a law about putting a warning in Big Red Letters on the box so someone doesn't make the mistake of using it in a large, long lived project.

> Oh, so OCaml is a toy language from academics.

What? Why did you conclude so?

> but no one mentioned that aspect of it.

Really? Did you open the documentation? It's stated explicitly.

https://caml.inria.fr/pub/docs/manual-ocaml/libref/Marshal.h...

> There should be a law about putting a warning in Big Red Letters on the box so someone doesn't make the mistake of using it in a large, long lived project.

It's documented, and it's not intended to be used for serialization, you use it in very particular cases, for example for faster object sharing with IPC for multicore. We use Json or S-expressions or other stuff for that purposes.

I really don't know why unison people don't switch to a proper serialization.

Re: Unison File Synchronizer

#96

Post by sedachv »

mindslight wrote: post #47
sedachv wrote: post #22

Earlier quoted context omitted.

> - Unison synchronizes files to files, but for a Dropbox-like system you really want deduplication for space savings (i.e. server-side storage is a bunch of pointers to content-addressed blocks.) > - in general, it's not clear to me that the client is really the hard part of Dropbox. Note e.g. the part where Dropbox now runs its own data centers for cost reasons. The whole point of Unison is that you do not need a s…

Erm, with pairwise syncing you still want a "server". I've used unison as my primarily file syncing for a decade, and trying to maintain a spanning tree among partially-available nodes/disks is a pain in the ass. And creating loops means you can't rely on file deletions. I agree it doesn't have to be a corporate-owned, or even corporate-snoopable "cloud" server, and if this is your goal then unison will work well. Al…

> Erm, with pairwise syncing you still want a "server". I've used unison as my primarily file syncing for a decade, and trying to maintain a spanning tree among partially-available nodes/disks is a pain in the ass. And creating loops means you can't rely on file deletions.

You do not need a spanning tree or loops, you just need to do a topological sort on the nodes/disks and then consistently sync them in that order. There is no magic algorithm that will handle conflict resolution for arbitrary syncing.

Re: Unison File Synchronizer

#97

Post by mindslight »

sedachv wrote: post #96
mindslight wrote: post #47

Earlier quoted context omitted.

Erm, with pairwise syncing you still want a "server". I've used unison as my primarily file syncing for a decade, and trying to maintain a spanning tree among partially-available nodes/disks is a pain in the ass. And creating loops means you can't rely on file deletions. I agree it doesn't have to be a corporate-owned, or even corporate-snoopable "cloud" server, and if this is your goal then unison will work well. Al…

> Erm, with pairwise syncing you still want a "server". I've used unison as my primarily file syncing for a decade, and trying to maintain a spanning tree among partially-available nodes/disks is a pain in the ass. And creating loops means you can't rely on file deletions. You do not need a spanning tree or loops, you just need to do a topological sort on the nodes/disks and then consistently sync them in that order.…

The problem I'm referring to isn't conflict resolution, but rather spurious re-creation of files that have been deleted. Unison's pairwise syncing cannot distinguish a file that has been deleted from one that has yet to be created (whereas say vector clocks do).

Create file F on A. Sync A-B, A-C. Delete F on A. Sync A-B. ?????. Sync B-C. Sync A-B. File F now re-exists on A (and B and C).

??? is some event where you cannot sync A-C to directly save A's changes on C, yet you still want to save any changes from B on C. Say a remote machine crashes and becomes unavailable, you didn't have the time over 3G, or some other type of ad-hoc craziness which is why you're using distributed syncing in the first place.

Which implies you need to choose one node from {A,B,C} that is the most likely to be available to sync the other two to. That node can also provide connectivity to a larger network, which generalizes to a spanning tree.

IMO a topological sort would be an even stricter requirement than spanning tree, in that if one node becomes available, you can't sync anything "below" it! Also what does syncing disks "in order" mean when changes can happen at any time? (eg I use unison for maildir).

Re: Unison File Synchronizer

#98

Post by mindslight »

icebraining wrote: post #67
mindslight wrote: post #47

Earlier quoted context omitted.

Erm, with pairwise syncing you still want a "server". I've used unison as my primarily file syncing for a decade, and trying to maintain a spanning tree among partially-available nodes/disks is a pain in the ass. And creating loops means you can't rely on file deletions. I agree it doesn't have to be a corporate-owned, or even corporate-snoopable "cloud" server, and if this is your goal then unison will work well. Al…

There are multiple dedup utilities. I use fdupes, which can not only list but also remove or replace duplicates with hardlinks.

Thanks! I guess formulating that question, I was halfway to apt-cache search. I had envisioned a tool that would find an entire equivalent dir tree (two unpacked source tarballs -> one match), and at first glance I don't know if fdupes will do that. But it might make sense to use an off the shelf tool with a little more manual labor.

Re: Unison File Synchronizer

#99

Post by a3n »

stewbrew wrote: post #81
a3n wrote: post #53

Earlier quoted context omitted.

Can you not mount the android file system on your laptop, and sync the two different directory trees, running exclusively from the laptop?

Modern Androids only support mtp for data access, which doesn't work well for that use case.

I haven't tried unison yet, but you can mount android on your laptop wirelessly over ssh.

Enable your android phone's hotspot, and connect your laptop to that hotspot.

Install termux on android (playstore). Install openssh in termux. Setup passwordless pub/private keys to log in from laptop to android termux sshd.

  cat ~/.ssh/config # Laptop.

  Host someName # Your android.
    User u0_a168 # Whatever username termux gave you.
    HostName 192.168.43.1 # 'ifconfig' in termux for this.
    IdentityFile /home/you/.ssh/someName_id_rsa
    Port     8022 # Default termux sshd port.
Now this should work from your laptop:

  ssh someName
Log out of your laptop's termux session.

Install sshfs on your laptop, through your package manager or the manly way from github. I'm not manly.

Here's part of a script I use to get photos from my phone to my laptop.

  # Mount the phone's DCIM directory locally over ssh.
  # I had trouble with termux symlinks, so I went directly to /storage/...
  sshfs someName:/storage/emulated/0/DCIM ~/mnt/someName

  cd ~/mnt/someName/Camera

  cp -vn * ~/Pictures/. # Copy all pictures that aren't yet on laptop.

  mv * ../Saved/. # So ops on dcim/Camera don't take longer with more pics.

  cd ~/Pictures # Can't unmount until leave mount.

  fusermount -u ~/mnt/someName # Dismount. Gymnastic salute.

  # Do whatever postprocessing you like on your photos ...
https://wiki.termux.com/wiki/Main_Page

https://wiki.termux.com/wiki/Remote_Access

https://wiki.termux.com/wiki/Internal_and_external_storage

https://en.wikipedia.org/wiki/SSHFS

man sshfs

stackoverflow

Re: Unison File Synchronizer

#100

Post by sedachv »

mindslight wrote: post #97
sedachv wrote: post #96

Earlier quoted context omitted.

> Erm, with pairwise syncing you still want a "server". I've used unison as my primarily file syncing for a decade, and trying to maintain a spanning tree among partially-available nodes/disks is a pain in the ass. And creating loops means you can't rely on file deletions. You do not need a spanning tree or loops, you just need to do a topological sort on the nodes/disks and then consistently sync them in that order.…

The problem I'm referring to isn't conflict resolution, but rather spurious re-creation of files that have been deleted. Unison's pairwise syncing cannot distinguish a file that has been deleted from one that has yet to be created (whereas say vector clocks do). Create file F on A. Sync A-B, A-C. Delete F on A. Sync A-B. ?????. Sync B-C. Sync A-B. File F now re-exists on A (and B and C). ??? is some event where you c…

> The problem I'm referring to isn't conflict resolution, but rather spurious re-creation of files that have been deleted.

That is conflict resolution, over the set of files. Which is why Unison provides the "reconciling changes" UI.

> Which implies you need to choose one node from {A,B,C} that is the most likely to be available to sync the other two to.

Yes, that is a way to avoid update conflicts.

> IMO a topological sort would be an even stricter requirement than spanning tree, in that if one node becomes available, you can't sync anything "below" it!

Yes, it is also a way to avoid update conflicts.

> (eg I use unison for maildir).

Why? IMAP already takes care of synchronization, without any potential problems.

Post reply on HN