Seems like there's some overlap with the iceoryx project https://iceoryx.io/
Show HN: Threadprocs – executables sharing one address space (0-copy pointers)
41–48 of 48 posts
Re: Show HN: Threadprocs – executables sharing one address space (0-copy pointers)
#42> Unlike dlopen-based plugin systems, threadprocs run traditional executables with a `main()` function. Why not dlopen with something that calls plugin_main() (etc.) in its own thread?
What if you're trying to run multiple instances of something that uses global state? Or that uses an incompatible library version? (I guess those are technically the same thing.)
Good point! (another reason not to use global state heh)
Re: Show HN: Threadprocs – executables sharing one address space (0-copy pointers)
#43Earlier quoted context omitted.
Judging by the description, it's exactly like AmigaOS, Windows 3.x or early Apple. Or MS-DOS things like DESQview. I fail to see the point - if you control the code and need performance so much that an occassional copy bites, you can as well just link it all into a single address space without those hoopjumps. It won't function as separate processes if it's modified to rely on passing pointers around anyway. And if y…
> Besides, what's wrong with shared memory? I generally think that it's bad to share memory for anything with concurrency, simply because it can make it very hard to reason about the code. Mutexes are hard to get right for anything that's not completely trivial, and I find that it's almost always better to figure out a way to do work without directly sharing memory if possible (or do some kind of borrow/ownership thi…
I agree, locks are best avoided and explicit message passing is much better for maintanability.
But OP talks about removing address space isolation entirely and sharing all memory, not just more-or-less well-defined bits.
Re: Show HN: Threadprocs – executables sharing one address space (0-copy pointers)
#44This is basically 'De-ASLR' is it not?
Could you clarify what you mean by that? This does heavily rely on loaded code being position-independent, because the memory used will go into whatever regions `mmap(..., ~MAP_FIXED)` returns.
I think it was meant not in a literal sense. ASLR is meant to make it hard to access memory which isn't yours. Your system makes it easy to access memory which isn't yours.
I wonder if the Rust checker could be made "extra process" aware in your scenario and thus allow rust programs to "connect to each other" in this shared memory space.
Re: Show HN: Threadprocs – executables sharing one address space (0-copy pointers)
#45Earlier quoted context omitted.
Could you clarify what you mean by that? This does heavily rely on loaded code being position-independent, because the memory used will go into whatever regions `mmap(..., ~MAP_FIXED)` returns.
I think it was meant not in a literal sense. ASLR is meant to make it hard to access memory which isn't yours. Your system makes it easy to access memory which isn't yours. I wonder if the Rust checker could be made "extra process" aware in your scenario and thus allow rust programs to "connect to each other" in this shared memory space.
That's a confusing perspective of ASLR
Re: Show HN: Threadprocs – executables sharing one address space (0-copy pointers)
#46I actually just published a paper about something like this, which I implemented in both libriscv and TinyKVM called "Inter-Process Remote Execution (IPRE): Low-latency IPC/RPC using merged address spaces". Here is the abstract: This paper introduces Inter-Process Remote Execution (IPRE), whose primary function is enabling gated persistence for per-request isolation architectures with microsecond-latency access to pe…
> Low-latency IPC/RPC using merged address spaces".
Can't this be achieved with a small block of shared memory, between processes that are otherwise isolated?
Re: Show HN: Threadprocs – executables sharing one address space (0-copy pointers)
#47I actually just published a paper about something like this, which I implemented in both libriscv and TinyKVM called "Inter-Process Remote Execution (IPRE): Low-latency IPC/RPC using merged address spaces". Here is the abstract: This paper introduces Inter-Process Remote Execution (IPRE), whose primary function is enabling gated persistence for per-request isolation architectures with microsecond-latency access to pe…
> Low-latency IPC/RPC using merged address spaces". Can't this be achieved with a small block of shared memory, between processes that are otherwise isolated?
You either have to pause the caller or prevent the caller from trampling the memory while the callee is using it. If you look at previous work like VMRPC (https://ieeexplore.ieee.org/document/5542746/) they make the shared area read-only while the callee is using it.
In IPRE, I am pausing the caller while the remote call is on-going, which means "just having a shared block" is inferior to just sharing everything. It's just so much easier and nicer to be able to pass literally anything you want.
The caveat is that the callee has to wait, but I think the fact that the remote is now running in the SAME THREAD without any scheduling involved makes up for it. It's a true synchronous remote function call with some overhead.
Re: Show HN: Threadprocs – executables sharing one address space (0-copy pointers)
#48Seems like there's some overlap with the iceoryx project https://iceoryx.io/
I'm one of the iceoryx maintainer :)
Yes, there is some overlap but iceoryx and threadprocs could complement each other. Currently, we achieve true zero copy only with data types that are shared memory compatible, i.e. no heap, no self references, etc., basically the data type must be memcopyable. Cap'n Proto or the Apache Arrow columnar format would also work. For other data, e.g one that uses the heap, there would still a serialization step be required but at least there is no further copy between the processes.
With threadprocs, it would be possible to also send data structures that use the heap or are self-referential. iceoryx2 already works in the same way as the proposed solution for the deallocation. There is a submission channel to provide the data and a completion channel where the pointer are sent back to the producer. Currently, we do not call any d'tor, since it is not necessary, but that should not be too difficult to implement.