Fast file synchronization and network forwarding for remote development • Hacker News Board

Re: Fast file synchronization and network forwarding for remote development

#61

Post by kyriakos » Mon, Oct 17, 2022, 12:14 PM UTC

comfypotato wrote: post #58

kyriakos wrote: post #52

I've been using mutagen for a long time as a workaround for WSL2 slow Windows storage access. I setup 2 copies of my project code, one in Windows and one in WSL2. IDE sees Windows drive while build tools and docker use Linux drive. Mutagen keeps them in sync near instantly via ssh. Thanks for this great tool.

Can you elaborate on the effectiveness of this methodology? It sounds too good to be true (so I’m tentative to explore). What hang ups if any do you encounter? Did you have any issues working around a corporate security policy? Have you ever run into any issues executing a test too quickly on the windows side for the sync to catch up?

corporate policy problems would be only if mutagen.exe is banned by your IT department. usually open source projects are ok with most companies.

there are issues occasionally with symlinks (especially with large node_modules folders) but most of the times nothing breaks and they are easily fixable - running mutagen sync monitor shows you things as they happen. one thing to consider is where .git directory is hosted. I personally keep git on windows side - you should make sure it only exists on one side and add an exclusion (mutagen has a parameter for that). performance wise, on SSD, an npm install from zero takes at most 5 seconds to sync (running npm on wsl2 -> files appearing on IDE).

Re: Fast file synchronization and network forwarding for remote development

#62

Post by ta988 » Mon, Oct 17, 2022, 1:47 PM UTC

xenoscopic wrote: post #50

ta988 wrote: post #45

Earlier quoted context omitted.

Yes because I have generated files too that I can edit further, and need to send log files, screenshots etc to other people.

One option I often recommend is setting up multiple synchronization sessions targeting different parts of a particular codebase with different synchronization configurations (kept mutually exclusive via ignore specifications). Quite often people want unidirectional replicas for a certain folder (e.g. a build directory) with more standard bidirectional synchronization for the rest of the code. It can be a little more…

Yes I have that too, node modules are one way toward my host machine so my IDE can use them and a few things like that like logs

Re: Fast file synchronization and network forwarding for remote development

#63

Post by xenoscopic » Mon, Oct 17, 2022, 2:32 PM UTC

comfypotato wrote: post #60

xenoscopic wrote: post #9

Mutagen author here — happy to answer any questions about Mutagen[0], its Docker Desktop extension[1], its Compose integration[2], or anything else! [0]: https://mutagen.io/ [1]: https://mutagen.io/documentation/docker-desktop-extension [2]: https://mutagen.io/documentation/orchestration/compose

Can you comment on the extent to which it will be maintained? I understand that you can’t predict the future, but the ever-looming threat of abandonware is always prominent on my mind when I see a tool that looks too good to be true.

I can't predict ultra far into the future (who can these days... :|), but Mutagen has been under active development for about 6 years now[0]. At the moment I have enough funding to work on it full-time until at least the middle of next year, though I also do Mutagen-related contracting and consulting work to support the project. Mutagen's Docker Desktop extension is going to be a freemium product designed to support the project more directly, which will hopefully allow development to continue indefinitely.

[0]: https://github.com/mutagen-io/mutagen/graphs/contributors

Re: Fast file synchronization and network forwarding for remote development

#64

Post by em-bee » Mon, Oct 17, 2022, 2:42 PM UTC

comfypotato wrote: post #59

em-bee wrote: post #41

Earlier quoted context omitted.

when does syncthing overwrite changes. that should not happen and would be a major bug

I’ve only ever casually browsed information about syncthing. Even I’ve stumbled across stories of people with a naive setup that, once fired up, started erasing all their files. I understand that this is probably an oversimplification, but I’ve read at least two separate stories that are along those lines. The comments for both stories were basically “yeah that sounds right because of the way you set it up”. It’s not…

maybe some people get confused thinking that unidirectional sync (send only) works like a backup?

it is possible to use syncthing for backups, but that is actually one of the pain points. there is an option to specify to ignore deletes, but that option is hidden in advanced settings, and there is no quick way to check if it is turned on. every time i delete something that i want to keep on the backup side. (like pictures from my phone, where i am running out of space) i first go to the destination/backup server and double check that this option is turned on to make sure that i won't accidentally have the backup deleted too.

it's also an issue that i can't see the remote settings from the client side. i have to connect to the remote server to check.

there is also a problem with the way it resolves conflicts. if you have "ignore delete" set up, and then you delete files, syncthing will tell you that the two locations are out of sync. it offers a button to force a sync, but there is no preview explaining what that button will do. there is also no choice in how the out-of-sync status should be resolved (in which direction the sync should happen to get the two locations in sync).

i have always been afraid to ever push that button, and there is no way to make it go away. i think this is an UX bug, because if i intentionally configure to ignore deletes, then deletes should not count as a difference. they should actually be ignored.