Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast file synchronization and network forwarding for remote development

github.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: Fast file synchronization and network forwarding for remote development

#51

Post by xenoscopic »

em-bee wrote: post #43

this sounds really impressive. unison has a UI to manage conflicts on a file level, whereas synchthing only allows a manual overwrite of all changes at once, and i have to log into the remote daemon if i want to overwrite changes there instead of locally. syncthing also will not tell me if all local changes arrived at the remote side, or if the remote is still busy downloading, without logging into the remote daemon…

There's no conflict resolution UI at the moment (either graphical or command-line based). Mutagen's conflict resolution is primarily performed via the specification of its synchronization mode (which can automate the resolution of most conflicts) and (in the case where conflicts can't be resolved automatically) by manually deleting the file that should "lose" the conflict (because Mutagen will synchronize a modification over a deletion).

Mutagen provides fairly detailed reporting of synchronization status, file staging progress, and change application problems via its "mutagen sync monitor" and "mutagen sync list" commands. These also support JSON output (and Go-template-style formatting), so you can pipe this information into other tooling. If you need REALLY detailed information, you can look at the debug or trace-level logs from the Mutagen daemon, but that's typically only for debugging during development of Mutagen itself.

Mutagen is somewhere in between Unison and Syncthing on the topology front, but closer to Unison. It still only supports two endpoints per synchronization session, but unlike Unison it doesn't require that one is local (i.e. you can do remote-to-remote sync using your local system as a proxy). But with both Mutagen (and Unison), you can set up a hub-and-spoke topology if you want to sync with multiple nodes.

Re: Fast file synchronization and network forwarding for remote development

#52

Post by kyriakos »

I've been using mutagen for a long time as a workaround for WSL2 slow Windows storage access. I setup 2 copies of my project code, one in Windows and one in WSL2. IDE sees Windows drive while build tools and docker use Linux drive. Mutagen keeps them in sync near instantly via ssh.

Thanks for this great tool.

Re: Fast file synchronization and network forwarding for remote development

#53

Post by memco »

cassianoleal wrote: post #6
karamanolev wrote: post #2

This sounds like my dream tool - I've always loved how quickly and well local tools work and remote environments cut into that good experience significantly. For me to be productive, I really need an instant feedback loop where tools work fast and I can immediately experience the result of some small piece of work. Has anyone tried this for a real-world project and can share feedback?

I find that VS Code's Remote-* extensions work well. I'm currently writing a Terraform provider on a remote Linux box using Remote-SSH and everything feels local. Compilation, etc happens on the remote and if I were serving requests it's dead easy to forward a port.

I like the remote editing stuff in VSCode, but I often end up having to rsync things at some point since I need to test code on remote servers, but I sometimes need access to other resources only available on my local machine. I can use VSCode for the editing, but I can use mutagen for the syncing. VSCode also ate some of my remote work before so I started using an extension that created periodic backups on the remote so that if it happened again I wouldn't lose much if any work. That hasn't happened in a while, but it was bad enough that I want to ensure it doesn't happen again. I tried an rsync extension, but it was flaky and poorly maintained. With something like git I'd have to create WIP commits, remember to push & pull and then remember to clean up the changes before merging.

Re: Fast file synchronization and network forwarding for remote development

#54

Post by ajvs »

xenoscopic wrote: post #40
ajvs wrote: post #33

How does it compare to syncthing?

Here's a bit of a comparison that I wrote the last time Mutagen was posted: https://news.ycombinator.com/item?id=30966448

Thanks, very interesting read. One-sided installation and low-latency sound particularly appealing.

Two features I didn't see addressed:

- Is there a GUI of any kind? I use Syncthing's web GUI + a tray applet for monitoring syncing on desktop, and on mobile there's an app for syncing also. Makes it easy to detect if there's a problem with any of the folders being synced, and easy to add new folders this way also.

- Is there an equivalent to Syncthing's untrusted devices feature? It basically allows for client-side encrypting the files, so you can have say one remote with the encrypted files and another remote with unencrypted files.

Re: Fast file synchronization and network forwarding for remote development

#55

Post by cassianoleal »

xenoscopic wrote: post #42
cassianoleal wrote: post #37

Earlier quoted context omitted.

So, transport-wise they're the same. For files at rest on the remote, I guess I assumed files would be encryted on the remote with a local key since GP said "one can develop on untrusted remote machine" and "VSCode remote always assumes that the remote part is trusted". On an actually untrusted remote, removing group read permissions doesn't do much to secure my code. The only scenario where it's helpful is a system…

In many ways Mutagen and VSCode's remote extensions are the same idea, with trade-offs in terms of flexibility vs. integration. Shared systems with multiple non-admin users was one of the original motivating use cases for tighter default permissions. I don't think there's any scenario where one can perform truly secure development work on an untrusted system. You could certainly store encrypted code in an untrusted l…

No, I think this clarifies it - thanks!

Just to be clear, when I mention sensitive work, I'm not necessarily talking about national security, military, etc. kind of work. Any work for a client is sensitive enough that I wouldn't do it on any remote I (or my client assuming there is approval) don't control.

I will try Mutagen at some point. The fact that it's editor agnostic is certainly a big sell!

Re: Fast file synchronization and network forwarding for remote development

#56

Post by eddyg »

xenoscopic wrote: post #11
eddyg wrote: post #10

This sounds useful. But one question that comes to mind right away: Does Mutagen handle the case where “local tools” (running on a completely different architecture than the remote) still need to “know” about include/header/library/etc. files from the remote machine in order to provide working “intelligence” capabilities? It’s one thing to efficiently sync “code”, but it’s another to make local tools fully-aware of t…

On the synchronization front, Mutagen's only goal is to facilitate the synchronization of files (albeit with a focus on development-related settings and low-latency for a "real time" feel). It doesn't attempt to integrate with any higher-level tooling (except in the cases of Docker Desktop and Compose, which is facilitated via external projects). That sort of tooling, language, and framework-specific integration is a…

Just wanted to say thanks for taking the time to reply, and for the links to those other projects that use Mutagen!

Re: Fast file synchronization and network forwarding for remote development

#58

Post by comfypotato »

kyriakos wrote: post #52

I've been using mutagen for a long time as a workaround for WSL2 slow Windows storage access. I setup 2 copies of my project code, one in Windows and one in WSL2. IDE sees Windows drive while build tools and docker use Linux drive. Mutagen keeps them in sync near instantly via ssh. Thanks for this great tool.

Can you elaborate on the effectiveness of this methodology? It sounds too good to be true (so I’m tentative to explore). What hang ups if any do you encounter? Did you have any issues working around a corporate security policy? Have you ever run into any issues executing a test too quickly on the windows side for the sync to catch up?

Re: Fast file synchronization and network forwarding for remote development

#59

Post by comfypotato »

em-bee wrote: post #41
ta988 wrote: post #12

I've been using mutagen for over 6 months now to sync over an M1 Linux VM. The only thing I miss is an option that would say "force everything from A" or "force everything from B" I've had rare cases where there were conflicts that I only could resolve by pausing mutagen and running rsync. But I appreciate that mutagen warns you and just doesn't overwrite silently like syncthing can do sometimes.

when does syncthing overwrite changes. that should not happen and would be a major bug

I’ve only ever casually browsed information about syncthing. Even I’ve stumbled across stories of people with a naive setup that, once fired up, started erasing all their files.

I understand that this is probably an oversimplification, but I’ve read at least two separate stories that are along those lines. The comments for both stories were basically “yeah that sounds right because of the way you set it up”. It’s not a bug so much as how synchthing works. My understanding is that there is a circumstance where you sync an empty folder with a folder with your documents, and it decides to synch the empty folder such that it erases the documents. Without any kind of warning. Wild.

Re: Fast file synchronization and network forwarding for remote development

#60

Post by comfypotato »

xenoscopic wrote: post #9

Mutagen author here — happy to answer any questions about Mutagen[0], its Docker Desktop extension[1], its Compose integration[2], or anything else! [0]: https://mutagen.io/ [1]: https://mutagen.io/documentation/docker-desktop-extension [2]: https://mutagen.io/documentation/orchestration/compose

Can you comment on the extent to which it will be maintained? I understand that you can’t predict the future, but the ever-looming threat of abandonware is always prominent on my mind when I see a tool that looks too good to be true.
Post reply on HN