Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast file synchronization and network forwarding for remote development

github.com

Post reply on HN
Print view View original

21–30 of 64 posts

Re: Fast file synchronization and network forwarding for remote development

#21

Post by AnthonBerg »

xenoscopic wrote: post #20
xenoscopic wrote: post #18

Earlier quoted context omitted.

The synchronization uses a repeated three-way merge algorithm, very similar to Git's merge when merging branches. It is triggered by recursive filesystem watching, which is also used to accelerate filesystem rescans. It maintains a virtual most-recent-ancestor and uses the two synchronization endpoints as the "branches" being merged. Much like Git has "-X ours" and "-X theirs" options, Mutagen also has automated conf…

Also, while Mutagen's exact implementation is novel in a number of ways, I would be remiss to not point out that huge amount of academic work in this field was done by Benjamin Pierce[0] and later implemented in Unison[1]. [0]: https://www.cis.upenn.edu/~bcpierce/papers/index.shtml#Synch... [1]: https://www.cis.upenn.edu/~bcpierce/unison/

Thank you so much for the great replies!

Re: Fast file synchronization and network forwarding for remote development

#22

Post by xenoscopic »

cassianoleal wrote: post #19
_0w8t wrote: post #16

Earlier quoted context omitted.

Mutagen tries to be secure so in principle one can develop on untrusted remote machine. VSCode remote always assumes that the remote part is trusted.

That sounds interesting but I can't find any mention to it in the docs. In fact, it sounds like it's just copying files over to the remote and running commands there. Are you able to provide a reference to how Mutagen secures my code on an untrusted remote?

The general philosophy with Mutagen is to (a) delegate encryption to other tools and (b) use secure defaults (especially for permissions).

So, for example, Mutagen doesn't implement any encryption, instead relying on transports like OpenSSH to provide the underlying transport encryption. In the Docker case, Mutagen does rely on the user securing the Docker transport if using TCP, but works to make this clear in the docs, and Mutagen is generally using the Docker Unix Domain Socket transport anyway. When communicating with itself, Mutagen also only uses secure Unix Domain Sockets and Windows Named Pipes.

When it comes to permissions, Mutagen doesn't do a blanket transfer of file ownership and permissions. Ownership defaults to the user under which the mutagen-agent binary is operating and permissions default to 0700/0600. The only permission bits that Mutagen transfers are executability bits, and only to entities with a corresponding read bit set. The idea is that synchronizing files to a remote, multi-user system shouldn't automatically expose your files to everyone on that system. These settings can be tweaked, of course, and in certain cases (specifically the Docker Desktop extension), broader permissions are used by default to emulate the behavior of the existing virtual filesystems that Mutagen is replacing.

Re: Fast file synchronization and network forwarding for remote development

#24

Post by cube2222 »

We've been using Mutagen extensively for remote development with an EC2 instance hosting a docker-compose with a couple of services and live rebuild+reload, and it's been working fantastic.

It's also nice for automatically managing port forwards.

Re: Fast file synchronization and network forwarding for remote development

#25

Post by ithkuil »

ta988 wrote: post #12

I've been using mutagen for over 6 months now to sync over an M1 Linux VM. The only thing I miss is an option that would say "force everything from A" or "force everything from B" I've had rare cases where there were conflicts that I only could resolve by pausing mutagen and running rsync. But I appreciate that mutagen warns you and just doesn't overwrite silently like syncthing can do sometimes.

Mutagen allows to choose a replication mode: https://mutagen.io/documentation/synchronization

Do you want something different from the "one-way-replica" mode?

Re: Fast file synchronization and network forwarding for remote development

#27

Post by notemaker »

Any user stories with *vim + mutagen for _large_ remote code bases? Vs code remote is the only thing that has been fast enough in my experience, but I would love to be able to use my local neovim instance for remote development instead and this tool looks promising.

Re: Fast file synchronization and network forwarding for remote development

#29

Post by xenoscopic »

notemaker wrote: post #27

Any user stories with *vim + mutagen for _large_ remote code bases? Vs code remote is the only thing that has been fast enough in my experience, but I would love to be able to use my local neovim instance for remote development instead and this tool looks promising.

It should work fine. Many users use Mutagen on multi-GB codebases. If we're talking something larger (say 10s of GBs or TB-sized monorepos), then there are some tweaks you can do to make life with Mutagen a little easier. Feel free to reach out to jacob[-at-]mutagen.io if you have a specific use case, or pop over to the Mutagen Community Slack Workspace[0] to chat.

[0]: https://mutagen.io/slack

Re: Fast file synchronization and network forwarding for remote development

#30

Post by xenoscopic »

Naac wrote: post #26

I haven't found anything better than using Unison. Maybe the linked README could compare prior art?

Conceptually speaking, Mutagen and Unison are very similar (and actually I mentioned Benjamin Pierce's work in another comment here asking about the sync algorithm - fantastic stuff!). I tend to avoid direct comparisons because they always come across one-sided, but some cursory differences:

- Mutagen tries to integrate recursive filesystem watching very tightly into its synchronization loop to drive synchronization and allow for near-instant filesystem rescans

- Mutagen automatically copies an "agent" binary to remote systems to support synchronization, so no remote install is required

- Mutagen uses Protocol Buffers for its data storage, so synchronization sessions created with older versions continue to work with newer versions

- Mutagen written in Go, Unison in OCaml (which allows Mutagen broader platform support "for free")

- Mutagen tries to treat Windows as a first-class citizen

- Mutagen uses race-free traversal (e.g. openat, fstatat, unlinkat, etc.) to perform operations

Obviously the internal implementations are different, but both use differential (rsync-style) file transfers, both use the same reconciliation concepts, etc.

Mutagen has the advantage of Go, recursive filesystem watching, and modern POSIX/Windows APIs that didn't exist when Unison was originally written, though some of that functionality has been brought into Unison.

For a comparison with Syncthing (and to some extent Unison), check out this comment[0].

[0]: https://news.ycombinator.com/item?id=30966448

Post reply on HN