Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Iggy.rs – building message streaming in Rust

blog.iggy.rs

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Iggy.rs – building message streaming in Rust

#41

Post by spetz »

steveklabnik wrote: post #38
vlovich123 wrote: post #21

Earlier quoted context omitted.

It’s not that the features are unreasonable but more that anything that uses it as a dependency also inherits the nightly requirement which makes things a bit more unstable because nightly is a floating version. I did compare with glommio a bit and disliked the file API in monoio but that’s probably more a matter of taste.

Yeah I hear you. I much prefer staying on stable for that reason. Sometimes (though much more rarely than non-Rust users seem to think) it is unavoidable though, and once you do go into nightly, you can kinda go two ways: * only use what is absolutely necessary * go wild and add tons of stuff As well as there being two kinds of nightly features: * stuff that's not stable but has a clear path to becoming stable on som…

As already mentioned, we've decived to use monoio, just to have something to begin with, it will take at least a few months to rewrite the core parts to make use of io_uring and thread-per-core approach (if feasible), so staying on the bleeding edge is more like a sandbox for us - we might decide to go with another runtime, like glommio, mfio or something else.

Re: Iggy.rs – building message streaming in Rust

#42

Post by spetz »

lysecret wrote: post #39

Quite an aspirational read especially about how the team came to be. I love open source.

I did some dotnet OSS in the past, but there's something special about Rust community and I guess that if I'd picked another language instead, I wouldn't gather such an amazing team :)

Re: Iggy.rs – building message streaming in Rust

#44

Post by steveklabnik »

spetz wrote: post #41
steveklabnik wrote: post #38

Earlier quoted context omitted.

Yeah I hear you. I much prefer staying on stable for that reason. Sometimes (though much more rarely than non-Rust users seem to think) it is unavoidable though, and once you do go into nightly, you can kinda go two ways: * only use what is absolutely necessary * go wild and add tons of stuff As well as there being two kinds of nightly features: * stuff that's not stable but has a clear path to becoming stable on som…

As already mentioned, we've decived to use monoio, just to have something to begin with, it will take at least a few months to rewrite the core parts to make use of io_uring and thread-per-core approach (if feasible), so staying on the bleeding edge is more like a sandbox for us - we might decide to go with another runtime, like glommio, mfio or something else.

Ah cool! Just to be clear, I think you should do whatever you feel is best for your project. I don't see this in any sort of moral terms, purely engineering ones, and there are good reasons to use nightly just like there are good reasons to use stable.

Re: Iggy.rs – building message streaming in Rust

#45

Post by spetz »

dochtman wrote: post #40
spetz wrote: post #25

Earlier quoted context omitted.

I've started with QUIC, as I wanted to try out something new. Still, TCP protocol we have in place is a bit faster than QUIC, but it might be due to the lack of some additional tuning or so. On a side note, QUIC on MacOS is slow, when compared to Linux.

Is TCP + TLS still faster than QUIC, or do you mean TCP without encryption? (I'm assuming you're using Quinn with rustls, so encryption is always enabled.) I'm also curious about your macOS vs Linux numbers. As a Quinn maintainer, we're always happy to hear feedback including about our performance!

Right, I meant TCP without TLS, and when it comes to QUIC I only did benchmark using the "dangerous" mode (disabled local cert validation). I don't have the exact numbers, but AFAIR on MacOS raw TCP was a few times faster than QUIC, while on Linux (e.g. PopOS distro) TCP was maybe around 20-30% faster?

Still, there are plenty of options to be configured in Quinn (buffer size, send/receive window etc.), so it might be that with some tweaking the performance would be similar or QUIC would be even faster than TCP - this is also one of the things that I'd like to spend more time with in the future. We expose most of these options in the server configuration file, so it's easy to adjust.

Thank you for contributing to such a great library, really easy to work with :)

Re: Iggy.rs – building message streaming in Rust

#46

Post by esafak »

the_duke wrote: post #12

I'm not quite sure how this compares to Kafka and fluvio [1], a Kafka competitor also written in Rust? Is it more of a message queue like rabbitmq? [1] https://www.fluvio.io/

Doesn't Fluvio aim to replace both Flink and Kafka? I just heard about it so I am trying to understand.

Re: Iggy.rs – building message streaming in Rust

#47

Post by bsaul »

spetz wrote: post #28
bsaul wrote: post #6

How fast can one realistically expect to be able to build decent quality system-level rust code, provided the person is already familiar with C-style programming languages and has 0 rust experience ?

I had 0 Rust experience and close to 0 of system programming experience (except playing with some lower level communication protocols etc. but nothing fancy), but I'd say that something like message streaming isn't the system programming - it's close to building the database, but nowhere close to making direct usage of low level system APIs.

Yes i didn’t know exactly how to call it. Let’s say « middleware programming »

Re: Iggy.rs – building message streaming in Rust

#48

Post by spetz »

bsaul wrote: post #47
spetz wrote: post #28

Earlier quoted context omitted.

I had 0 Rust experience and close to 0 of system programming experience (except playing with some lower level communication protocols etc. but nothing fancy), but I'd say that something like message streaming isn't the system programming - it's close to building the database, but nowhere close to making direct usage of low level system APIs.

Yes i didn’t know exactly how to call it. Let’s say « middleware programming »

That's a good name! :)

Re: Iggy.rs – building message streaming in Rust

#49

Post by tizz98 »

Really cool idea and project! There are 2 things I would need to understand before trying it out though:

1. How can I run more than one instance of the server?

2. When running more than one instance of the server, how would the filesystem interaction work between servers?

Re: Iggy.rs – building message streaming in Rust

#50

Post by tialaramex »

bsaul wrote: post #6

How fast can one realistically expect to be able to build decent quality system-level rust code, provided the person is already familiar with C-style programming languages and has 0 rust experience ?

It's hard to judge what counts as "decent quality". However, much of the very low level understanding translates 1:1 from C to Rust. For example in term of the ordering rules to deliver a multi-threaded program with sequential consistency (which thus can be understood by humans) those are identical, both C and Rust "stole" the C++ 11 model.

For ABI Rust has an explicit way to say "Lay this out the way C does" which works for things that you could have done in C. How is everything else laid out? Not your business. So again, your C expertise applies.

Neither Rust nor C formally have agreed pointer provenance rules, if you've learned things that are promised to work in this domain in C most of them also work in Rust. Most things aren't promised to work in C, Aria's Strict Provenance experiment and associated APIs are more than you're going to get from ISO C any time soon.

Aliasing rules are stricter in Rust than C, but also I'd argue much simpler. So that seems not to be a huge amount of extra work.

Post reply on HN