Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast file synchronization and network forwarding for remote development

github.com

Post reply on HN
Print view View original

11–20 of 64 posts

Re: Fast file synchronization and network forwarding for remote development

#11

Post by xenoscopic »

eddyg wrote: post #10

This sounds useful. But one question that comes to mind right away: Does Mutagen handle the case where “local tools” (running on a completely different architecture than the remote) still need to “know” about include/header/library/etc. files from the remote machine in order to provide working “intelligence” capabilities? It’s one thing to efficiently sync “code”, but it’s another to make local tools fully-aware of t…

On the synchronization front, Mutagen's only goal is to facilitate the synchronization of files (albeit with a focus on development-related settings and low-latency for a "real time" feel). It doesn't attempt to integrate with any higher-level tooling (except in the cases of Docker Desktop and Compose, which is facilitated via external projects). That sort of tooling, language, and framework-specific integration is a bit outside the project's target scope (and something that becomes very domain-specific).

Mutagen will, however, happily operate between different operating systems and architectures, so things like working with a remote amd64-based Docker engine from your local arm64-based laptop are totally possible.

Also, several external projects (such as DDEV[0] and Garden[1]) do use Mutagen as a low-level component in their stack to provide synchronization that does "know" a bit more about the framework that you're using.

[0]: https://ddev.com/ [1]: https://garden.io/

Re: Fast file synchronization and network forwarding for remote development

#12

Post by ta988 »

I've been using mutagen for over 6 months now to sync over an M1 Linux VM. The only thing I miss is an option that would say "force everything from A" or "force everything from B" I've had rare cases where there were conflicts that I only could resolve by pausing mutagen and running rsync. But I appreciate that mutagen warns you and just doesn't overwrite silently like syncthing can do sometimes.

Re: Fast file synchronization and network forwarding for remote development

#13

Post by xenoscopic »

xani_ wrote: post #7

How's that compared to sshfs (wth cache/kernel_cache enabled) ? I've used it few times where I had need to dev like that and it was generally just fine for just editing a file, where performance tanked was doing a lot of file I/O at once (say updating git repo)

The benchmarks will likely be highly dependent on your use case, but SSHFS-style virtual filesystems (specifically those backed by FUSE) typically have significantly lower performance than something like an APFS/ext4/NTFS filesystem that Mutagen could target with synchronization.

All of your readdir()/stat()/open()/read()-style calls will suffer significantly on virtual filesystems, and unfortunately these get hit a lot by things like IDEs (e.g. when indexing code), compilers, and dynamic language runtimes (especially PHP).

No tool is at fault in this chain, of course, it's a hard problem. Mutagen is able to offer better performance by being a little less dynamic and creating "real" copies of all the files on a more persistent filesystem.

Re: Fast file synchronization and network forwarding for remote development

#14

Post by ta988 »

xani_ wrote: post #7

How's that compared to sshfs (wth cache/kernel_cache enabled) ? I've used it few times where I had need to dev like that and it was generally just fine for just editing a file, where performance tanked was doing a lot of file I/O at once (say updating git repo)

Advantage of mutagen is that it works on OSes that can't do sshfs. It felt faster too especially with a lot of IOs like node modules or other things that touch a lot of files. But I never ran a benchmark , it is so much faster by at least a factor 10 than whatever is in docker desktop when populating node modules that I don't even need a benchmark.

Re: Fast file synchronization and network forwarding for remote development

#15

Post by jedisct1 »

Super useful tool!

Plus, it's multi platform. I'm using it to synchronize directories between hosts running macOS, OpenBSD and Linux. Everything works fine.

I haven't tried the Docker Desktop extension since I switched to Colima (Docker Desktop is constantly broken on Apple Silicon).

Re: Fast file synchronization and network forwarding for remote development

#16

Post by _0w8t »

cassianoleal wrote: post #6
karamanolev wrote: post #2

This sounds like my dream tool - I've always loved how quickly and well local tools work and remote environments cut into that good experience significantly. For me to be productive, I really need an instant feedback loop where tools work fast and I can immediately experience the result of some small piece of work. Has anyone tried this for a real-world project and can share feedback?

I find that VS Code's Remote-* extensions work well. I'm currently writing a Terraform provider on a remote Linux box using Remote-SSH and everything feels local. Compilation, etc happens on the remote and if I were serving requests it's dead easy to forward a port.

Mutagen tries to be secure so in principle one can develop on untrusted remote machine. VSCode remote always assumes that the remote part is trusted.

Re: Fast file synchronization and network forwarding for remote development

#17

Post by ta988 »

karamanolev wrote: post #2

This sounds like my dream tool - I've always loved how quickly and well local tools work and remote environments cut into that good experience significantly. For me to be productive, I really need an instant feedback loop where tools work fast and I can immediately experience the result of some small piece of work. Has anyone tried this for a real-world project and can share feedback?

Yes it is excellent, syncing macos (Jetbrains tools and a few other things) with a Linux VM .

Re: Fast file synchronization and network forwarding for remote development

#18

Post by xenoscopic »

AnthonBerg wrote: post #8

I’d like to know more about the theory behind the synchronisation — how the syncing is known to be safe and non-destructive.

The synchronization uses a repeated three-way merge algorithm, very similar to Git's merge when merging branches. It is triggered by recursive filesystem watching, which is also used to accelerate filesystem rescans. It maintains a virtual most-recent-ancestor and uses the two synchronization endpoints as the "branches" being merged. Much like Git has "-X ours" and "-X theirs" options, Mutagen also has automated conflict resolution[0] modes that can be specified. You can find the reconciliation algorithm here[1] (and there are an exhaustive set of test cases in the corresponding _test.go file).

To avoid a large class of race conditions (at least to the extent possible allowed by POSIX and Windows), Mutagen will use `*at` style system calls for all filesystem traversal on POSIX systems, with a similar strategy on Windows.

Also, to avoid race conditions due to filesystem changes between scan time and change-application time, Mutagen will perform just-in-time checks that filesystem contents haven't changed from what was fed into the reconciliation algorithm.

[0]: https://mutagen.io/documentation/synchronization#modes [1]: https://github.com/mutagen-io/mutagen/blob/master/pkg/synchr...

Re: Fast file synchronization and network forwarding for remote development

#19

Post by cassianoleal »

_0w8t wrote: post #16
cassianoleal wrote: post #6

Earlier quoted context omitted.

I find that VS Code's Remote-* extensions work well. I'm currently writing a Terraform provider on a remote Linux box using Remote-SSH and everything feels local. Compilation, etc happens on the remote and if I were serving requests it's dead easy to forward a port.

Mutagen tries to be secure so in principle one can develop on untrusted remote machine. VSCode remote always assumes that the remote part is trusted.

That sounds interesting but I can't find any mention to it in the docs. In fact, it sounds like it's just copying files over to the remote and running commands there.

Are you able to provide a reference to how Mutagen secures my code on an untrusted remote?

Re: Fast file synchronization and network forwarding for remote development

#20

Post by xenoscopic »

xenoscopic wrote: post #18
AnthonBerg wrote: post #8

I’d like to know more about the theory behind the synchronisation — how the syncing is known to be safe and non-destructive.

The synchronization uses a repeated three-way merge algorithm, very similar to Git's merge when merging branches. It is triggered by recursive filesystem watching, which is also used to accelerate filesystem rescans. It maintains a virtual most-recent-ancestor and uses the two synchronization endpoints as the "branches" being merged. Much like Git has "-X ours" and "-X theirs" options, Mutagen also has automated conf…

Also, while Mutagen's exact implementation is novel in a number of ways, I would be remiss to not point out that huge amount of academic work in this field was done by Benjamin Pierce[0] and later implemented in Unison[1].

[0]: https://www.cis.upenn.edu/~bcpierce/papers/index.shtml#Synch... [1]: https://www.cis.upenn.edu/~bcpierce/unison/

Post reply on HN