Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RaptorQ and performance optimization in Rust

cberner.com

Post reply on HN
Print view View original

21–30 of 51 posts

Re: RaptorQ and performance optimization in Rust

#21

Post by ignoramous »

sudhirj wrote: post #15
layoutIfNeeded wrote: post #6

What is the patent situation with Raptor codes nowadays?

We have a license and use them for satellite broadcasting, but my understanding is Qualcomm has made statements in https://datatracker.ietf.org/ipr/1511/ that if you use it for a "wireless wide-area standard (for example, a UMTS-compatible handset or Infrastructure equipment)" you'll be charged a standard royalty fee, otherwise they don't care.

Noob question: Any reason folks would use FountainCode/RaptorQ over TurboCode [0] / PolarCode [1]? What's the difference?

[0] https://en.wikipedia.org/wiki/Turbo_code

[1] https://en.wikipedia.org/wiki/Polar_code_(coding_theory)

Re: RaptorQ and performance optimization in Rust

#22

Post by pornel »

empath75 wrote: post #2

Anybody want to comment on whether all of his unsafe code was actually necessary? Seems bad for rust that safe code is 25x slower.

I don't think 25x speed difference can be attributed to safe code. In most cases safe code just means extra bounds checking or refcounting here and there, which aren't that bad even in hot loops.

The fragment of code in the article with `get_unchecked_mut()` shouldn't be necessary. It's a simple case that LLVM should be able to optimize. And if it didn't, it could be helped by either iterating both slices with `zip()` or a trick `let slice = &slice[0..len];` which proves to LLVM that you have the required length and it doesn't need to be checked again.

But overall the article seems in line what you'd expect from Rust: you have low-level control over memory layout and safety checks, and you can make trade-offs to squeeze maximum performance out of an algorithm if you need to.

Re: RaptorQ and performance optimization in Rust

#23

Post by lower »

When comparing a Rust rewrite of a C++ codebase, I noticed that Rust's default HashMap and HashSet were noticably slower than unordered_map and unordered_set. The reason is that Rust uses a hash function that has better properties, but that is a bit slower. If the performance of a HashMap is a bottleneck, one can use a simpler hash function to get the same performance as in C++. With FnvHashMap, the performance was the same as unordered_map.

Re: RaptorQ and performance optimization in Rust

#25

Post by v_lisivka »

lower wrote: post #23

When comparing a Rust rewrite of a C++ codebase, I noticed that Rust's default HashMap and HashSet were noticably slower than unordered_map and unordered_set. The reason is that Rust uses a hash function that has better properties, but that is a bit slower. If the performance of a HashMap is a bottleneck, one can use a simpler hash function to get the same performance as in C++. With FnvHashMap, the performance was t…

Try hashbrown. Author declares 8x improvement over standard HashMap.

https://github.com/Amanieu/hashbrown

https://blog.waffles.space/2018/12/07/deep-dive-into-hashbro...

https://gankro.github.io/blah/hashbrown-insert/

Re: RaptorQ and performance optimization in Rust

#26

Post by steveklabnik »

v_lisivka wrote: post #25
lower wrote: post #23

When comparing a Rust rewrite of a C++ codebase, I noticed that Rust's default HashMap and HashSet were noticably slower than unordered_map and unordered_set. The reason is that Rust uses a hash function that has better properties, but that is a bit slower. If the performance of a HashMap is a bottleneck, one can use a simpler hash function to get the same performance as in C++. With FnvHashMap, the performance was t…

Try hashbrown. Author declares 8x improvement over standard HashMap. https://github.com/Amanieu/hashbrown https://blog.waffles.space/2018/12/07/deep-dive-into-hashbro... https://gankro.github.io/blah/hashbrown-insert/

It soon will be the standard hashmap!

Re: RaptorQ and performance optimization in Rust

#29

Post by tgtweak »

sudhirj wrote: post #16
tgtweak wrote: post #4

Very cool efforts here. Curious what the speed is without those instructions present on the cpu, and what hardware this was run on. The concept of a fountain seems interesting but what is a good use case? Variable strength error correction?

Let's say you want to send something (say 100KB) to a million listeners, but you don't know which of them is going to be listening when. You'd feed that 100KB into a RaptorQ encoder, configure it to 1KB packets, and it would give you stream of near-infinite 1KB packets that you could broadcast (usually over a satellite, but UDP, multicast, QR codes all work). Receivers would listen for as many of these packets as the…

Very interesting. Still struggling to see how this would be more efficient than looping that data for the use case of getting beacon data out. Perhaps it's more consistent/predictable in scenarios of high or targeted loss (ie first X bytes keep getting dropped)?

Re: RaptorQ and performance optimization in Rust

#30

Post by lower »

v_lisivka wrote: post #25
lower wrote: post #23

When comparing a Rust rewrite of a C++ codebase, I noticed that Rust's default HashMap and HashSet were noticably slower than unordered_map and unordered_set. The reason is that Rust uses a hash function that has better properties, but that is a bit slower. If the performance of a HashMap is a bottleneck, one can use a simpler hash function to get the same performance as in C++. With FnvHashMap, the performance was t…

Try hashbrown. Author declares 8x improvement over standard HashMap. https://github.com/Amanieu/hashbrown https://blog.waffles.space/2018/12/07/deep-dive-into-hashbro... https://gankro.github.io/blah/hashbrown-insert/

Thanks! I just replaced FnvHashMap with hashbrown::HashMap and get a 20% overall performance improvement in my workload (which is rather HashMap-heavy).
Post reply on HN