Earlier quoted context omitted.
Sort of. Not really. spiped operates at the level of individual stream connections, so you can e.g. make one end a local socket in a filesystem and use UNIX permissions to control access to it. In fact that's exactly why I wrote it -- so I could have a set of daemons designed to communicate via local sockets and transparently (aside from performance) have them running on different systems.
Is it possible to use tarsnap's deduplication code on my own server? We're setting up an ML dataset distribution box, and I was hoping to avoid storing e.g. imagenet as a tarball + untar'd (so that nginx can serve each photo individually) + imagenet in TFDS format. https://github.com/xolox/dedupfs was the closest I found, but it has a lot of downsides. Has anyone made an interface to tarsnap's tarball dedup code? A p…
Tarsnap's deduplication code is not available under an open-source license.