Live data from Hacker News

Fast file synchronization and network forwarding for remote development

github.com

41–50 of 64 posts

Re: Fast file synchronization and network forwarding for remote development

#41
post #12

I've been using mutagen for over 6 months now to sync over an M1 Linux VM. The only thing I miss is an option that would say "force everything from A" or "force everything from B" I've had rare cases where there were conflicts that I only could resolve by pausing mutagen and running rsync. But I appreciate that mutagen warns you and just doesn't overwrite silently like syncthing can do sometimes.

when does syncthing overwrite changes. that should not happen and would be a major bug

Re: Fast file synchronization and network forwarding for remote development

#42

Earlier quoted context omitted.

The general philosophy with Mutagen is to (a) delegate encryption to other tools and (b) use secure defaults (especially for permissions). So, for example, Mutagen doesn't implement any encryption, instead relying on transports like OpenSSH to provide the underlying transport encryption. In the Docker case, Mutagen does rely on the user securing the Docker transport if using TCP, but works to make this clear in the d…

So, transport-wise they're the same. For files at rest on the remote, I guess I assumed files would be encryted on the remote with a local key since GP said "one can develop on untrusted remote machine" and "VSCode remote always assumes that the remote part is trusted". On an actually untrusted remote, removing group read permissions doesn't do much to secure my code. The only scenario where it's helpful is a system…

In many ways Mutagen and VSCode's remote extensions are the same idea, with trade-offs in terms of flexibility vs. integration.

Shared systems with multiple non-admin users was one of the original motivating use cases for tighter default permissions.

I don't think there's any scenario where one can perform truly secure development work on an untrusted system. You could certainly store encrypted code in an untrusted location, but there's not much you could do with it on that system (without a hypothetical compiler or tool that maybe supported some sort of homomorphic-encryption compilation operations?). Even decryption on-the-fly for processing by regular tools wouldn't be secure on an untrusted system. And running any code there would be equally insecure.

I'd imagine that for any seriously sensitive work, one would only want to work in highly controlled, trusted, and firewalled environments. If there's a scenario I'm missing though, definitely let me know.

Re: Fast file synchronization and network forwarding for remote development

#43
this sounds really impressive.

unison has a UI to manage conflicts on a file level, whereas synchthing only allows a manual overwrite of all changes at once, and i have to log into the remote daemon if i want to overwrite changes there instead of locally.

syncthing also will not tell me if all local changes arrived at the remote side, or if the remote is still busy downloading, without logging into the remote daemon

on the other hand, unison only manages one source/target per process if i remember correctly while syncthing can manage many different source/target pairs conveniently.

how does mutagen compare in these two areas?

Re: Fast file synchronization and network forwarding for remote development

#44

Mutagen author here — happy to answer any questions about Mutagen[0], its Docker Desktop extension[1], its Compose integration[2], or anything else! [0]: https://mutagen.io/ [1]: https://mutagen.io/documentation/docker-desktop-extension [2]: https://mutagen.io/documentation/orchestration/compose

Nice work. Depending on your secure networking goals, you could evaluate using the OpenZiti Golang SDK to embed zero trust networking in Mutagen as an alternative to VPNs.

Re: Fast file synchronization and network forwarding for remote development

#45
post #25
post #12

I've been using mutagen for over 6 months now to sync over an M1 Linux VM. The only thing I miss is an option that would say "force everything from A" or "force everything from B" I've had rare cases where there were conflicts that I only could resolve by pausing mutagen and running rsync. But I appreciate that mutagen warns you and just doesn't overwrite silently like syncthing can do sometimes.

Mutagen allows to choose a replication mode: https://mutagen.io/documentation/synchronization Do you want something different from the "one-way-replica" mode?

Yes because I have generated files too that I can edit further, and need to send log files, screenshots etc to other people.

Re: Fast file synchronization and network forwarding for remote development

#46
post #41
post #12

I've been using mutagen for over 6 months now to sync over an M1 Linux VM. The only thing I miss is an option that would say "force everything from A" or "force everything from B" I've had rare cases where there were conflicts that I only could resolve by pausing mutagen and running rsync. But I appreciate that mutagen warns you and just doesn't overwrite silently like syncthing can do sometimes.

when does syncthing overwrite changes. that should not happen and would be a major bug

When machines were not online at the same time or with unreliable connectivity. I've managed to never loose data because I have staggered backups so I could always revert to the in-between version.

Re: Fast file synchronization and network forwarding for remote development

#47
Mutagen sounds impressive. I wonder how well it works synchronizing a large directory. My company has two servers, in different states, connected via a VPN. We currently use rsync and cron. The directory contains thousands of files. Would Mutagen be able handle that?

Re: Fast file synchronization and network forwarding for remote development

#48

Mutagen sounds impressive. I wonder how well it works synchronizing a large directory. My company has two servers, in different states, connected via a VPN. We currently use rsync and cron. The directory contains thousands of files. Would Mutagen be able handle that?

I use mutagen daily on a very large monolithic application repository and while the initial sync can take a couple of minutes, subsequent edits are replicated sub second - even when changing multiple files e.g. via linter auto fixing files

Re: Fast file synchronization and network forwarding for remote development

#49

Mutagen author here — happy to answer any questions about Mutagen[0], its Docker Desktop extension[1], its Compose integration[2], or anything else! [0]: https://mutagen.io/ [1]: https://mutagen.io/documentation/docker-desktop-extension [2]: https://mutagen.io/documentation/orchestration/compose

Thank you for all your hard work, amazing tool!

Re: Fast file synchronization and network forwarding for remote development

#50
post #45
post #25

Earlier quoted context omitted.

Mutagen allows to choose a replication mode: https://mutagen.io/documentation/synchronization Do you want something different from the "one-way-replica" mode?

Yes because I have generated files too that I can edit further, and need to send log files, screenshots etc to other people.

One option I often recommend is setting up multiple synchronization sessions targeting different parts of a particular codebase with different synchronization configurations (kept mutually exclusive via ignore specifications). Quite often people want unidirectional replicas for a certain folder (e.g. a build directory) with more standard bidirectional synchronization for the rest of the code. It can be a little more complex to orchestrate, but you can use Mutagen's project functionality or a shell script to automate the setup a bit.
Post reply on HN