Show HN: Iceoryx2 – Fast IPC Library for Rust, C++, and C

I agree with your parallelism counter-argument in principle. However even there it would probably make sense to not trust each other, to limit the blast radius of successful attacks.

In your next point the "careful" illustrates exactly my point. Using shared memory for IPC is like using C or C++ and saying "well I'll be careful then". It can work but it will be very dangerous and most likely there will be security issues. You are much better off not doing it.

Postgres is a beautiful argument in that respect. Yes you can write a database in C or C++ and have it use shared memory. It's just not recommended because you need professionals of the caliber of the Postgres people to pull it off. I understand many organizations think they have those. I don't think they actually do though.

Yes, we are using shared memory, and I agree that shared memory is a challenge but there are some mechanisms that can make it secure.

The main problem with shared memory is, that one process can corrupt the data structure while another process is consuming it. Even verifying the contents of the data structure is insufficient since it can always be corrupted afterwards. We have named the problem "modify-after-delivery problem" - a sender modifies the data after it has been delivered to a receiver.

This can be handled with:

1. memfd: The sender acquires it, writes its payload, seals it so that it is read-only and then transfers the file descriptor to all receivers. The receiver can verify the read-only seal with fcntl. Since linux guarantees us that it cannot be reverted the receiver can now safely consume the data. This allows it to be used even in a zero-trust environment. [1] provides a good introduction (see the File-Sealing IPC subsection). 2. Memory protection keys [2]: I do not have too much experience with them, but as far as I understand, they solve the problem with mprotect, meaning, the sender can call mprotect and make the segment read only for it self, but the receiver has no way of verifying it or to prevent the sender from calling mprotect again and granting it read/write access again to corrupt the data.

So, the approach is that a sender acquires shared memory, writes its payload into it, makes it read-only, and then transfers it to the receivers.

> Shared memory works as a transport if you either assume that all parties are trusted (in which case why do IPC in the first place?

Robustness is another use case. In mission-critical systems you trust each process but a crash caused by a bug in one sub-system shall not bring down the whole system. So you split up the monolith in many processes and the overall system survives if one process goes down or deadlocks, assuming you have a shared memory library that itself is safe. If you detect a process crash, you can restart it and continue operations.

[1] https://dvdhrm.wordpress.com/2014/06/10/memfd_create2/ [2] https://www.kernel.org/doc/html/latest/core-api/protection-k...

So far for this topic, we have only done some brief research on the options available to us, such us going over the C API or using something like PyO3.

I second that. It took us quite some time to get the correct architecture. After all, iceoryx2 is the third incarnation of this piece of software, with elfepiff an me working on the last two.

> 1. Unless you're using either fixed sized or specially allocated structures, you end up paying for serialization anyhow (zero copy is actually one copy).

Indeed, we are using fixed size structures with a bucket allocator. We have ideas on how to enable the usage on types which support custom allocators and even with raw pointers but that is just a crazy idea which might not pan out to work.

> 2. There's no way to reference count the shared memory - if a reader crashes, it holds on to the memory it was reading. You can get around this with some form of watchdog process, or by other schemes with a side channel, but it's not "easy". > > 3. Similar to 2, if a writer crashes, it will leave behind junk in whatever filesystem you are using to hold the shared memory.

Indeed, this is a complicated topic and support from the OS would be appreciated. We found a few ways on how to make this feasible, though.

The origins of iceoryx are in automotive and there it is required to split functionality up into multiple processes. When one process goes down, the system can still operate in a degraded mode or just restart the faulty process. With this, one needs an efficient and low-latency solution else the CPU is spending more time on copying data than on doing real work.

Of course there are issues like the producer mutating data after delivery, but here are also solutions for this. It will of course affect the latency but should still be better than using e.g. unix domain sockets.

Fun fact. For iceoryx1 we supported only 4GB memory chunks and some time ago someone came and asked if we could lift this limitation since he wanted to transfer a 92GB large language model via shared memory.