Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast file synchronization and network forwarding for remote development

github.com

Post reply on HN
Print view View original

31–40 of 64 posts

Re: Fast file synchronization and network forwarding for remote development

#31

Post by xenoscopic »

emrah wrote: post #23

What is the benefit over rsync which is the perfect tool for this at the moment? Maybe add an faq section to the readme for questions like this?

The primary benefits:

- Mutagen performs bidirectional synchronization (though it can also operate unidirectionally); rsync is unidirectional

- Mutagen uses recursive filesystem watching to avoid full filesystem rescans (whereas rsync always does a full filesystem rescan). This allows Mutagen to provide a more "real time" sync.

- Mutagen has an active synchronization loop that doesn't require manual invocation.

- Mutagen has more idiomatic Windows support.

- Mutagen doesn't require that it be pre-installed on both endpoints.

Both use differential transfers (i.e. the "rsync algorithm") for transferring individual files.

There are other differences, of course, as well as similarities. Mutagen's design is tuned for development work, rsync's design is tuned for replication. I still use rsync for archival operations on a daily basis - it's great!

Re: Fast file synchronization and network forwarding for remote development

#32

Post by liketochill »

xenoscopic wrote: post #20
xenoscopic wrote: post #18

Earlier quoted context omitted.

The synchronization uses a repeated three-way merge algorithm, very similar to Git's merge when merging branches. It is triggered by recursive filesystem watching, which is also used to accelerate filesystem rescans. It maintains a virtual most-recent-ancestor and uses the two synchronization endpoints as the "branches" being merged. Much like Git has "-X ours" and "-X theirs" options, Mutagen also has automated conf…

Also, while Mutagen's exact implementation is novel in a number of ways, I would be remiss to not point out that huge amount of academic work in this field was done by Benjamin Pierce[0] and later implemented in Unison[1]. [0]: https://www.cis.upenn.edu/~bcpierce/papers/index.shtml#Synch... [1]: https://www.cis.upenn.edu/~bcpierce/unison/

I’ve been using unison for what feels like 14 years. Once working it was great but it always took me a while to figure out the exact command line options I wanted. Beautiful tool.

Re: Fast file synchronization and network forwarding for remote development

#34

Post by quechimba »

notemaker wrote: post #27

Any user stories with *vim + mutagen for _large_ remote code bases? Vs code remote is the only thing that has been fast enough in my experience, but I would love to be able to use my local neovim instance for remote development instead and this tool looks promising.

I did this and Mutagen worked great. It was pretty awesome to not have any lag while developing and Mutagen was really fast to detect and sync changes. Never really noticed any delays, everything felt instant

Re: Fast file synchronization and network forwarding for remote development

#35

Post by angrais »

cube2222 wrote: post #24

We've been using Mutagen extensively for remote development with an EC2 instance hosting a docker-compose with a couple of services and live rebuild+reload, and it's been working fantastic. It's also nice for automatically managing port forwards.

Have you considered plugins for vscode, e.g., vscode remote and vs code containers?

Re: Fast file synchronization and network forwarding for remote development

#36

Post by cube2222 »

angrais wrote: post #35
cube2222 wrote: post #24

We've been using Mutagen extensively for remote development with an EC2 instance hosting a docker-compose with a couple of services and live rebuild+reload, and it's been working fantastic. It's also nice for automatically managing port forwards.

Have you considered plugins for vscode, e.g., vscode remote and vs code containers?

That's actually a different use case than ours. We only use it to run the builder and app services (which can use a bunch of resources), while keeping the editor/IDE tooling local.

That said, we have developers using VS Code, JetBrains, as well as Vim. The Mutagen-based workflow works well with all of them.

Re: Fast file synchronization and network forwarding for remote development

#37

Post by cassianoleal »

xenoscopic wrote: post #22
cassianoleal wrote: post #19

Earlier quoted context omitted.

That sounds interesting but I can't find any mention to it in the docs. In fact, it sounds like it's just copying files over to the remote and running commands there. Are you able to provide a reference to how Mutagen secures my code on an untrusted remote?

The general philosophy with Mutagen is to (a) delegate encryption to other tools and (b) use secure defaults (especially for permissions). So, for example, Mutagen doesn't implement any encryption, instead relying on transports like OpenSSH to provide the underlying transport encryption. In the Docker case, Mutagen does rely on the user securing the Docker transport if using TCP, but works to make this clear in the d…

So, transport-wise they're the same.

For files at rest on the remote, I guess I assumed files would be encryted on the remote with a local key since GP said "one can develop on untrusted remote machine" and "VSCode remote always assumes that the remote part is trusted".

On an actually untrusted remote, removing group read permissions doesn't do much to secure my code.

The only scenario where it's helpful is a system with multiple non-admin users, perhaps like a university lab computer but who's doing sensitive work on those anyway?

Re: Fast file synchronization and network forwarding for remote development

#38

Post by CoastalCoder »

liketochill wrote: post #32
xenoscopic wrote: post #20

Earlier quoted context omitted.

Also, while Mutagen's exact implementation is novel in a number of ways, I would be remiss to not point out that huge amount of academic work in this field was done by Benjamin Pierce[0] and later implemented in Unison[1]. [0]: https://www.cis.upenn.edu/~bcpierce/papers/index.shtml#Synch... [1]: https://www.cis.upenn.edu/~bcpierce/unison/

I’ve been using unison for what feels like 14 years. Once working it was great but it always took me a while to figure out the exact command line options I wanted. Beautiful tool.

> but it always took me a while to figure out the exact command line options I wanted.

Reminds me of my occasional use of rsync. I'm always afraid of invoking it wrong and synching in the wrong direction.

Re: Fast file synchronization and network forwarding for remote development

#39

Post by danrochman »

jedisct1 wrote: post #15

Super useful tool! Plus, it's multi platform. I'm using it to synchronize directories between hosts running macOS, OpenBSD and Linux. Everything works fine. I haven't tried the Docker Desktop extension since I switched to Colima (Docker Desktop is constantly broken on Apple Silicon).

TIL about Colima - thanks! Grabbing it now on my M1 Mac.
Post reply on HN