Is this project kind of an amalgamation of Kafka and something like Flight/nats?
Iggy.rs – building message streaming in Rust
31–40 of 63 posts
Re: Iggy.rs – building message streaming in Rust
#32Is this project kind of an amalgamation of Kafka and something like Flight/nats?
Can't say for sure, we share some of the same concepts that can be found in Kafka or RabbitMQ Streams, but at the same time, try to build something from the ground up and focus on low-level I/O improvements :)
Re: Iggy.rs – building message streaming in Rust
#33I'm surprised about the selection of monoio. AFAIK it requires the usage of the nightly compiler which I've found to not be a great choice when maintaining a project.
Monoio seems to be the most performant runtime, and actually easy to use - we have decided to go with "bleeding edge" approach, as it will still take at least a few months to implement io_uring and other optimizations, as we'll have to rewrite some of the core parts and most likely shift towards thread-per-core architecture.
For what it’s worth I don’t believe it meaningfully outperforms glommio. There are some spots where it outperforms glommio but that’s just a missing optimization that could be added. The biggest advantage monoio really has is that it currently has support for other platforms.
Not saying it’s the wrong choice, just highlighting the trade offs.
I’m not sure what you mean by io_uring as glommio/monoio should be hiding the io_uring details behind the runtime.
Re: Iggy.rs – building message streaming in Rust
#34Definitely wins a prize for cute dog picture
Love it
Re: Iggy.rs – building message streaming in Rust
#35Definitely wins a prize for cute dog picture Love it
Thanks, I do agree, they're both lovely! :D
Re: Iggy.rs – building message streaming in Rust
#36Earlier quoted context omitted.
Monoio seems to be the most performant runtime, and actually easy to use - we have decided to go with "bleeding edge" approach, as it will still take at least a few months to implement io_uring and other optimizations, as we'll have to rewrite some of the core parts and most likely shift towards thread-per-core architecture.
For what it’s worth I don’t believe it meaningfully outperforms glommio. There are some spots where it outperforms glommio but that’s just a missing optimization that could be added. The biggest advantage monoio really has is that it currently has support for other platforms. Not saying it’s the wrong choice, just highlighting the trade offs. I’m not sure what you mean by io_uring as glommio/monoio should be hiding t…
Of course, they hide the details behind io_uring, it's just that monoio seemed even easier to work with. It's not set in stone, though, actually there are new runtimes being developed as we speak, for example mfio - we'll see what will be our final choice, but we've decided to start with monoio.
Speaking of the other platforms, having the fallback to epoll or kqueue (they've even announced some Windows integration) is nice to have, however, at some point we might just purely focus on Linux development anyway (io_uring only), if there'd be any issues e.g. when it comes to the code design to provide the compatibility across multiple OS.
Re: Iggy.rs – building message streaming in Rust
#37Built something similar in Go with a friend a few years ago.
Re: Iggy.rs – building message streaming in Rust
#38Earlier quoted context omitted.
It does require nightly, but it only uses five features, one of which could be removed by adding an external crate, and the others are like, not anything super wild, for the most part. I haven't looked too deeply into the code but they all make sense to me, for example one of them is a standard library API for creating uninitialized containers, which allows you to eliminate a copy. I have not compared monoio with glo…
It’s not that the features are unreasonable but more that anything that uses it as a dependency also inherits the nightly requirement which makes things a bit more unstable because nightly is a floating version. I did compare with glommio a bit and disliked the file API in monoio but that’s probably more a matter of taste.
Yeah I hear you. I much prefer staying on stable for that reason. Sometimes (though much more rarely than non-Rust users seem to think) it is unavoidable though, and once you do go into nightly, you can kinda go two ways:
* only use what is absolutely necessary
* go wild and add tons of stuff
As well as there being two kinds of nightly features:
* stuff that's not stable but has a clear path to becoming stable on some time frame
* stuff that's not stable but who knows when that will change, it may never
And to me, this dependency feels kinda closer to choosing the first option from both lists, which is the most conservative way of using nightly.
Re: Iggy.rs – building message streaming in Rust
#39Quite an aspirational read especially about how the team came to be. I love open source.
Re: Iggy.rs – building message streaming in Rust
#40It seems like such a sharp, smart move to start with QUIC. It provides really useful SCTP-like multi-streaming, which is a great base to start from. There's already so many good libraries the project or others can work from, and it'll just keep getting better (and better optimized, likely). https://github.com/xileteam/awesome-quic?tab=readme-ov-file#... This seems like such a natural place where of course having a so…
I've started with QUIC, as I wanted to try out something new. Still, TCP protocol we have in place is a bit faster than QUIC, but it might be due to the lack of some additional tuning or so. On a side note, QUIC on MacOS is slow, when compared to Linux.
Is TCP + TLS still faster than QUIC, or do you mean TCP without encryption? (I'm assuming you're using Quinn with rustls, so encryption is always enabled.)
I'm also curious about your macOS vs Linux numbers. As a Quinn maintainer, we're always happy to hear feedback including about our performance!