Live data from Hacker News

Mutagen – Cloud-based development using your local tools

mutagen.io

51–58 of 58 posts

Re: Mutagen – Cloud-based development using your local tools

#51

Author of Mutagen here, happy to answer any questions you might have, whether it's about its use with Docker, SSH, its historical integration into Docker Desktop, or its related Mutagen Compose[0][1] project. [0]: https://github.com/mutagen-io/mutagen-compose [1]: https://mutagen.io/documentation/orchestration/compose

Hi Jacob. I am one of the founders of Okteto (https://okteto.com/), a remote development platform for Compose and Kubernetes applications. We use Syncthing to sync code between the developer laptop and pods running in Kubernetes. I would love to know your thoughts on the strengths and weak points of Mutagen vs Syncthing for this use case. Thanks!

Re: Mutagen – Cloud-based development using your local tools

#52

Author of Mutagen here, happy to answer any questions you might have, whether it's about its use with Docker, SSH, its historical integration into Docker Desktop, or its related Mutagen Compose[0][1] project. [0]: https://github.com/mutagen-io/mutagen-compose [1]: https://mutagen.io/documentation/orchestration/compose

Hi Jacob. I am one of the founders of Okteto ( https://okteto.com/ ), a remote development platform for Compose and Kubernetes applications. We use Syncthing to sync code between the developer laptop and pods running in Kubernetes. I would love to know your thoughts on the strengths and weak points of Mutagen vs Syncthing for this use case. Thanks!

Also, how hard would it be to run mutagen on web assembly?

Re: Mutagen – Cloud-based development using your local tools

#53

Just curious, when you develop for the Cloud where to you debug your code? In the cloud or on your work-station?

For integration tests and other code meant to run like external facing code I use my local machine for debugging. If my local machine is too slow (like running the whole integ test suite) I use a cloud VM. For internal code or slow to build code I use a remote VM hooked up to the internal network. That allows me to build quickly and test in an integration environment. I use a file synchronization tool like mutagen at…

VSCode makes it very easy to selectively install plugins on a remote that doesn’t have internet access and allows fairly seamless editing, debugging and terminal access to a remote with very little effort. But I’ve been looking for alternatives to VSCode’s remote editing experience because I’ve generally got about 6-8 projects that I keep open all the time and it can eat up a lot of RAM and CPU especially when I have several of them running and debugging stuff in tandem. The trick for me is that beyond file editing synchronization I need to be able to debug the code. There’s some promising ideas here, but I haven’t been able to make the jump yet. The jet brains link you shared is the most similar offering I’ve seen, but I’m hopeful distant.nvim or mutagen (plus whatever else I’d need) can be used to get more tools on a similar level of seamlessness.

Re: Mutagen – Cloud-based development using your local tools

#54
post #53

Earlier quoted context omitted.

For integration tests and other code meant to run like external facing code I use my local machine for debugging. If my local machine is too slow (like running the whole integ test suite) I use a cloud VM. For internal code or slow to build code I use a remote VM hooked up to the internal network. That allows me to build quickly and test in an integration environment. I use a file synchronization tool like mutagen at…

VSCode makes it very easy to selectively install plugins on a remote that doesn’t have internet access and allows fairly seamless editing, debugging and terminal access to a remote with very little effort. But I’ve been looking for alternatives to VSCode’s remote editing experience because I’ve generally got about 6-8 projects that I keep open all the time and it can eat up a lot of RAM and CPU especially when I have…

I'm asking because I'm contemplating on doing some work on AWS. If code runs on AWS doesn't it mean it makes lots of calls to the different AWS APIs. And then in which environment do you debug the code that makes those API-calls? Is it even possible to debug it locally?

Re: Mutagen – Cloud-based development using your local tools

#56
post #53

Earlier quoted context omitted.

For integration tests and other code meant to run like external facing code I use my local machine for debugging. If my local machine is too slow (like running the whole integ test suite) I use a cloud VM. For internal code or slow to build code I use a remote VM hooked up to the internal network. That allows me to build quickly and test in an integration environment. I use a file synchronization tool like mutagen at…

VSCode makes it very easy to selectively install plugins on a remote that doesn’t have internet access and allows fairly seamless editing, debugging and terminal access to a remote with very little effort. But I’ve been looking for alternatives to VSCode’s remote editing experience because I’ve generally got about 6-8 projects that I keep open all the time and it can eat up a lot of RAM and CPU especially when I have…

So specifically for debugging I have used the remote interpreter feature in most jetbrains IDEs: https://www.jetbrains.com/help/pycharm/remote-debugging-with...

For web dev: https://www.jetbrains.com/help/webstorm/debugging-javascript...

I use jetbrains IDEs cause you get this stuff out the box without much fiddling with configs and it’s all integrated together.

For lower ram when working on multiple projects they did build a new ide for lightweight remote dev but I havent tried it: https://www.jetbrains.com/fleet/

Could be worth a shot.

Re: Mutagen – Cloud-based development using your local tools

#57

Author of Mutagen here, happy to answer any questions you might have, whether it's about its use with Docker, SSH, its historical integration into Docker Desktop, or its related Mutagen Compose[0][1] project. [0]: https://github.com/mutagen-io/mutagen-compose [1]: https://mutagen.io/documentation/orchestration/compose

Hi Jacob. I am one of the founders of Okteto ( https://okteto.com/ ), a remote development platform for Compose and Kubernetes applications. We use Syncthing to sync code between the developer laptop and pods running in Kubernetes. I would love to know your thoughts on the strengths and weak points of Mutagen vs Syncthing for this use case. Thanks!

Sure, that's a great question. I'll preface my response by saying that I'm a huge fan of Syncthing (and that Mutagen's use cases form a Venn diagram with those of Syncthing (as well as tools like rsync)). Technologically, all three of these tools are very similar in terms of using the rsync differential transfer algorithm, but their architectures and primary use cases differ. I think the core differentiators with Mutagen are:

Development-oriented: Mutagen's sync configuration is primarily focused on development, so it adds more granular controls for things like uni-/bi-directionality, conflict resolution, ignores, symbolic link handling, etc. It also has a permission propagation model that's focused on things like cross-platform executability propagation and preservation (i.e. between Windows and POSIX), as well as operating in multi-service environments where many different process UIDs/GIDs might be in play. It also handles weird filesystem quirks (like macOS Unicode decomposition).

Low-latency: Mutagen's goal is to reduce the latency of sync cycles (i.e. time from local edit to change reflection on the remote) to an imperceptible level. It uses a lot of tricks to try to do this, but the goal is really to use the absolute best filesystem watching mechanism for each platform and to integrate that tightly with the sync loop. On Linux, for example, Mutagen is now starting to experiment with the recently revamped fanotify[0] API to get highly scalable but low-latency watching (as opposed to the emulated and janky recursive watching emulation with inotify that most tools use). It also uses tricks like rsync-diffing of the metadata snapshots that it transfers to get latency as low as possible. The eventual goal is to reach sub-100ms sync cycles for multi-GB codebases, and I think that's pretty close.

Git-like sync: Conceptually, Mutagen's sync algorithm is like a filesystem watcher + repetitive three-way Git merge (with the difference being that file transfers are deltified and the merging (potentially) affects both endpoints). This means it tracks content in a manner very similar to Git's CAS and branches, which is a little different than the way that Syncthing does it. This affords (in my opinion) more precise identification of conflicts.

Distrust: Mutagen takes a more aggressive approach to mutual distrust between endpoints, working hard to ensure that a malicious endpoint can't read outside the synchronization root on the other endpoint via symbolic links or maliciously crafted paths. It does this by using POSIX *at functions to traverse the filesystem and perform operations. This avoids issues like CVE-2017-1000420. You can harden this further by using unidirectional sync and other configuration options. This makes it well-suited to cases where multiple users might be syncing to file storage on a shared system (say on a SaaS platform) (though, at least in that case, you can protect yourself and users with the filesystem namespacing afforded by containers).

One-sided installation and flexible topology: Mutagen's primary M.O. is injecting small "agent" binaries to remote systems via a copy mechanism (such as `scp` or `docker cp`), so you don't have to manually install it on both endpoints. This is less important to "full stack" cases like Okteto, where your tooling can handle the setup of Syncthing on the remote, but it makes working directly over SSH or in ephemeral containers significantly more convenient. And Mutagen's architecture is also really flexible, allowing it to sync files and forward traffic between any combination of local and remote endpoints (including remote-to-remote, proxied via the local Mutagen daemon).

Command-based transports: Mutagen uses the standard I/O streams of commands like `ssh` and `docker exec` as its transport (similar to tools like Git or rsync), making it easier to target remote environments with your existing tooling and configuration. Again, this is less of an issue for a case like Okteto's, but is useful in the standalone case.

Network forwarding: This is outside the scope of sync, but Mutagen offers OpenSSH-style TCP/UDS forwarding (with the difference from OpenSSH being that Mutagen's forwarding is persistent and managed by a background daemon). This offers support for doing things like forwarding a local socket to a remote Docker daemon over SSH, and then forwarding web application traffic over that underlying forwarding by using Mutagen over `docker exec` (or reverse forwarding, or forwarding between two remotes and bridging them via your laptop, or loads of other crazy shenanigans).

I hope that clarifies things a bit. Ping me via email if you want an expanded comparison.

[0]: https://man7.org/linux/man-pages/man7/fanotify.7.html

Re: Mutagen – Cloud-based development using your local tools

#58

Earlier quoted context omitted.

Hi Jacob. I am one of the founders of Okteto ( https://okteto.com/ ), a remote development platform for Compose and Kubernetes applications. We use Syncthing to sync code between the developer laptop and pods running in Kubernetes. I would love to know your thoughts on the strengths and weak points of Mutagen vs Syncthing for this use case. Thanks!

Also, how hard would it be to run mutagen on web assembly?

It should be possible, especially since Mutagen is already built for almost all of Go's supported architectures. The biggest issue would just be implementing the race-free filesystem traversal that's used on Windows and POSIX (or potentially living without it given that WASM would probably provide sufficient sandboxing). In any case, most of the work would take place in Mutagen's `filesystem` package, where those syscall equivalents would have to be figured out. The rest of Mutagen should compile without modification. You'd also need to define a transport for Mutagen to use, which would depend on the exact mode of operation you're looking at, but that's the easier problem to solve. Mutagen v0.15 is going to be focused on extensibility (including custom transports), so something like this may become a reality soon.
Post reply on HN