Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rust Atomics and Locks: Low-Level Concurrency in Practice

marabos.nl

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Rust Atomics and Locks: Low-Level Concurrency in Practice

#21

Post by throwaway17_17 »

cwzwarich wrote: post #11
seanhunter wrote: post #6

Earlier quoted context omitted.

In case anyone here doesn’t know, Paul McKenney was one of the main contributors to the RCU[1] implementation for the Linux kernel. So the guy knows a thing or two about concurrency. [1] https://en.wikipedia.org/wiki/Read-copy-update if you just want to know what it is and http://www.rdrop.com/users/paulmck/RCU/rclock_OLS.2001.05.01... if you want the good stuff

Although, somewhat amusingly, neither C++ nor Rust (which basically just copies the C++ atomics and memory model) can be used to correctly express RCU written within the language. The leading proposal for C++ (at least AFAICT) is to just add RCU primitives to the library and make their correctness the implementation's concern: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p25... I hope we get a next genera…

Can you provide any extra information about what is unable to be expressed? I’d like to look into what sort of constructs could be useful in this context.

Re: Rust Atomics and Locks: Low-Level Concurrency in Practice

#23

Post by cwzwarich »

throwaway17_17 wrote: post #21
cwzwarich wrote: post #11

Earlier quoted context omitted.

Although, somewhat amusingly, neither C++ nor Rust (which basically just copies the C++ atomics and memory model) can be used to correctly express RCU written within the language. The leading proposal for C++ (at least AFAICT) is to just add RCU primitives to the library and make their correctness the implementation's concern: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p25... I hope we get a next genera…

Can you provide any extra information about what is unable to be expressed? I’d like to look into what sort of constructs could be useful in this context.

The keyword to search for is "memory_order_consume", which was C++11's proposed solution to the problem that turned out to not work in practice. Here are some of the C++ WG documents describing the issues:

https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p00... https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p01... https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p04... https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p07...

A lot of the problems center around the reliance on data dependencies for ordering, and the difficulty of expressing these requirements in a manner that is compatible with separate compilation and compiler optimizations.

Re: Rust Atomics and Locks: Low-Level Concurrency in Practice

#25

Post by chandlore »

seanhunter wrote: post #6
estebank wrote: post #3

The foreword by Paul E. McKenney makes a great case to read this book even if you don't care about Rust at all: > Which brings us to another group of potential readers, the Rust skeptics. While I do believe that most Rust skeptics are doing the community a valuable service by pointing out opportunities for improvement, all but the most Rust-savvy of skeptics would benefit from reading this book. If nothing else, doin…

In case anyone here doesn’t know, Paul McKenney was one of the main contributors to the RCU[1] implementation for the Linux kernel. So the guy knows a thing or two about concurrency. [1] https://en.wikipedia.org/wiki/Read-copy-update if you just want to know what it is and http://www.rdrop.com/users/paulmck/RCU/rclock_OLS.2001.05.01... if you want the good stuff

He also wrote a good introduction to parallel programming that's freely available as well:

https://mirrors.edge.kernel.org/pub/linux/kernel/people/paul...

Re: Rust Atomics and Locks: Low-Level Concurrency in Practice

#26

Post by penguin_booze »

estebank wrote: post #3

The foreword by Paul E. McKenney makes a great case to read this book even if you don't care about Rust at all: > Which brings us to another group of potential readers, the Rust skeptics. While I do believe that most Rust skeptics are doing the community a valuable service by pointing out opportunities for improvement, all but the most Rust-savvy of skeptics would benefit from reading this book. If nothing else, doin…

Also from Paul: The Perf Book: https://mirrors.edge.kernel.org/pub/linux/kernel/people/paul....

Re: Rust Atomics and Locks: Low-Level Concurrency in Practice

#27

Post by gpderetta »

cwzwarich wrote: post #11
seanhunter wrote: post #6

Earlier quoted context omitted.

In case anyone here doesn’t know, Paul McKenney was one of the main contributors to the RCU[1] implementation for the Linux kernel. So the guy knows a thing or two about concurrency. [1] https://en.wikipedia.org/wiki/Read-copy-update if you just want to know what it is and http://www.rdrop.com/users/paulmck/RCU/rclock_OLS.2001.05.01... if you want the good stuff

Although, somewhat amusingly, neither C++ nor Rust (which basically just copies the C++ atomics and memory model) can be used to correctly express RCU written within the language. The leading proposal for C++ (at least AFAICT) is to just add RCU primitives to the library and make their correctness the implementation's concern: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p25... I hope we get a next genera…

> Although, somewhat amusingly neither C++ nor Rust [...] can be used to correctly express RCU written within the language.

What do you mean exactly? Of course the required remote memory barrier need to be a primitive as it is provided by the OS, and ultimately the hardware, but AFAIK the asymmetric synchronization needed by RCU can be modelled within the C++ as a signal and a atomic signal fence.

Re: Rust Atomics and Locks: Low-Level Concurrency in Practice

#28

Post by cwzwarich »

gpderetta wrote: post #27
cwzwarich wrote: post #11

Earlier quoted context omitted.

Although, somewhat amusingly, neither C++ nor Rust (which basically just copies the C++ atomics and memory model) can be used to correctly express RCU written within the language. The leading proposal for C++ (at least AFAICT) is to just add RCU primitives to the library and make their correctness the implementation's concern: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p25... I hope we get a next genera…

> Although, somewhat amusingly neither C++ nor Rust [...] can be used to correctly express RCU written within the language. What do you mean exactly? Of course the required remote memory barrier need to be a primitive as it is provided by the OS, and ultimately the hardware, but AFAIK the asymmetric synchronization needed by RCU can be modelled within the C++ as a signal and a atomic signal fence.

You can't express the equivalent of Linux's rcu_dereference primitive in a way that avoids data races without using memory_order_acquire (or memory_order_consume, which is treated like memory_order_acquire in all implementations, and is fundamentally broken as I link to in my sibling reply). With a memory consistency model weaker than TSO (e.g. ARM, RISC-V, POWER), this imposes unnecessary overhead and eliminates some of the benefits of RCU.

To work around this flaw, you can use inline assembly, implementation-specific "compiler-only" barriers, etc. but that's outside of the language. It also forces the compiler to be much more pessimistic about optimizations than is actually necessary to preserve dependency ordering.

Re: Rust Atomics and Locks: Low-Level Concurrency in Practice

#29

Post by ecnahc515 »

peoplefromibiza wrote: post #17
estebank wrote: post #15

Earlier quoted context omitted.

Someone has to implement those frameworks, and those people need to understand pretty much everything covered by this book :)

not to disagree or antagonize, but the question was > How on earth are you doing concurrency without touching these two key primitives and not how to write high level concurrency frameworks. I wouldn't trust very much a concurrency framework written by someone who just learned about locks and atomics on a book. I wouldn't even trust myself to write low level concurrent code using locks and atomics. I have 26 years of…

Sometimes all you need is an atomic counter though (or lock). Like, for the commonly used Prometheus metrics format, you usually just have global, atomic counters, because it's arguably the most efficient/easiest way to do it. Higher level primitives are nice, but even lower level APIs might be "the right choice" for even high-level problems.

Re: Rust Atomics and Locks: Low-Level Concurrency in Practice

#30

Post by lilyball »

gavinray wrote: post #22

I'm curious how folks in Rust implement striped-locking? Striped locking is where you chunk sections of data behind the same lock to improve scalability. So, 1 lock might protect 1,000 items in an array, for example.

I've never done this but I imagine you could use something like

  [Mutex; 10]
Post reply on HN