Comparison of Rust async and Linux thread context switch time and memory use • Hacker News Board

Re: Comparison of Rust async and Linux thread context switch time and memory use

#81

Post by _vvhw » Fri, Feb 12, 2021, 9:32 AM UTC

klodolph wrote: post #41

scottlamb wrote: post #39

Earlier quoted context omitted.

> What io_uring does do is provide a way to poll without needing to wait, but if you haven’t received new events when you poll, you’re not on the fast path any more. Isn't "not on the fast path any more" a bit absolutist? io_uring's "slow" path is roughly one syscall per iteration, right? That's still many fewer syscalls than one syscall per IO operation (or more if any return EAGAIN/EWOULDBLOCK) as you'd be doing wi…

io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall. I am a bit baffled how this could possibly be considered an “absolutist” viewpoint—I am just saying that there exist scenarios where io_uring is not helpful. This should be uncontroversial.

> io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall.

That's not correct, io_uring was "absolutely" designed, at least in the technical sense, for zero syscalls in the slow path (if you want to):

  IORING_SETUP_SQPOLL
  When this flag is specified, a kernel thread is created to perform submission queue polling.
  An io_uring instance configured in this way enables an application to issue I/O without ever
  context switching into the kernel. By using the submission queue to fill in new submission
  queue entries and watching for completions on the completion queue, the application can submit
  and reap I/Os without doing a single system call.

From the man page: https://manpages.debian.org/unstable/liburing-dev/io_uring_s...

This mode required privileges in early kernel versions but that's already changed. Things are moving fast.

Re: Comparison of Rust async and Linux thread context switch time and memory use

#82

Post by _vvhw » Fri, Feb 12, 2021, 9:41 AM UTC

josephg wrote: post #55

KirillPanov wrote: post #19

> But this advantage goes away if the context switch is due to I/O readiness This is not at all a fair comparison unless you're using io_uring.

Good point - It’d be very interesting to see how io_uring changes those numbers if anyone has some time to make a fork / PR!

Not Rust, but you may be interested in colorless async io_uring using Zig: https://news.ycombinator.com/item?id=26111847

Also (very rough) benchmarks (take with a pinch of salt) comparing various styles of fs and network IO (blocking, epoll, io_uring) for C and Zig: https://github.com/coilhq/tigerbeetle/tree/master/demos/io_u...

Re: Comparison of Rust async and Linux thread context switch time and memory use

#83

Post by Ciantic » Fri, Feb 12, 2021, 9:46 AM UTC

josephg wrote: post #53

masklinn wrote: post #46

Earlier quoted context omitted.

> So involving "async" is totally the wrong tool for the job. Sadly with so many things having gone async-first (or only) it’s become difficult not to end up with an async runtime anyway, or not be forced to use an async system. I wanted to build a small web-based tool for local, didn’t really find anything which was not async.

I’d happily take an async-by-default world over a world where some APIs only exist through blocking calls. A classically threaded program can easily block on a future, but wrapping a blocking call in an otherwise asynchronous program is complicated, expensive and error prone work.

Btw, have you read this: https://async.rs/blog/stop-worrying-about-blocking-the-new-a... async-std allows to run blocking calls without hoops rather efficiently:

   async fn read_to_string(path: impl AsRef) -> io::Result {
       std::fs::read_to_string(path)
   }

It doesn't have await inside! My mind was blown as I saw that.

Re: Comparison of Rust async and Linux thread context switch time and memory use

#84

Post by richardwhiuk » Fri, Feb 12, 2021, 9:55 AM UTC

The problem with threads is you need to correctly size your thread pool. That's hugely difficult if you have unknown lengths of blocking IO.

Re: Comparison of Rust async and Linux thread context switch time and memory use

#85

Post by masklinn » Fri, Feb 12, 2021, 9:55 AM UTC

josephg wrote: post #53

masklinn wrote: post #46

Earlier quoted context omitted.

> So involving "async" is totally the wrong tool for the job. Sadly with so many things having gone async-first (or only) it’s become difficult not to end up with an async runtime anyway, or not be forced to use an async system. I wanted to build a small web-based tool for local, didn’t really find anything which was not async.

I’d happily take an async-by-default world over a world where some APIs only exist through blocking calls. A classically threaded program can easily block on a future, but wrapping a blocking call in an otherwise asynchronous program is complicated, expensive and error prone work.

> A classically threaded program can easily block on a future, but wrapping a blocking call in an otherwise asynchronous program is complicated, expensive and error prone work.

It’s really not though, at least as long as the parameters and results are Send. For instance Tokio has a spawn_blocking which runs the function on one of the blocking threads it spawns on-demand specifically for that use.

Meanwhile « blocking on a future » requires adding and managing an entire async runtime and its interactions with the rest of the program, and locking up the runtime is a very real possibility.

Re: Comparison of Rust async and Linux thread context switch time and memory use

#86

Post by masklinn » Fri, Feb 12, 2021, 10:09 AM UTC

Ciantic wrote: post #83

josephg wrote: post #53

Earlier quoted context omitted.

I’d happily take an async-by-default world over a world where some APIs only exist through blocking calls. A classically threaded program can easily block on a future, but wrapping a blocking call in an otherwise asynchronous program is complicated, expensive and error prone work.

Btw, have you read this: https://async.rs/blog/stop-worrying-about-blocking-the-new-a... async-std allows to run blocking calls without hoops rather efficiently: async fn read_to_string(path: impl AsRef ) -> io::Result { std::fs::read_to_string(path) } It doesn't have await inside! My mind was blown as I saw that.

> should a task execute for too long, the runtime will automatically react by spawning a new executor thread taking over the current thread’s work.

That is a super interesting strategy, though obviously only works when you can « afford » a multithreaded scheduler.

Anyway I wonder how they manage this, signals?

Re: Comparison of Rust async and Linux thread context switch time and memory use

#87

Post by pas » Fri, Feb 12, 2021, 10:12 AM UTC

kzrdude wrote: post #77

lxpz wrote: post #75

> People often see that there's some theoretical benefit of async and then they accept far less ergonomic coding styles and the additional bug classes that only happen on async due to accidental blocking etc... despite the fact that when you consider a real-world deployed application, those "benefits" become indistinguishable from noise. However, due to the additional bug classes and worse ergonomics, there is now le…

That's interesting to hear - but how much of an investment is it to climb the that mountain (or hill) that makes you comfortable with working with the async model?

Rust's async model takes very little time to grasp. It's very explicit. Nothing runs in the background (contrast that with NodeJS). You have strong static types to help you to know when you got a Future, you can decide where/when to await it.

It's programming with threads, where you have a thread pool and pipes to put tasks onto that and a helper function/macro. Await does this under the hood of course.

Re: Comparison of Rust async and Linux thread context switch time and memory use

#88

Post by jiggawatts » Fri, Feb 12, 2021, 10:16 AM UTC

lxpz wrote: post #75

> People often see that there's some theoretical benefit of async and then they accept far less ergonomic coding styles and the additional bug classes that only happen on async due to accidental blocking etc... despite the fact that when you consider a real-world deployed application, those "benefits" become indistinguishable from noise. However, due to the additional bug classes and worse ergonomics, there is now le…

Speaking of futures_unordered and similar patterns, I think a part of the "async promise" that has failed is the lack of concurrency for a single user request by default in most languages.

That is, the 'easy' path is to write code such as the following (in vaguely C# pseudocode):

    var p = await GetUserPermission( username );
    var c = await GetServerConfig();
    var m = await GetMessageOfTheDay();

Assume each await call is potentially an expensive SQL query or REST API call.

The problem with that is that this is strictly sequential, synchronous code that is merely "dehydrated" and "rehydrated" to reduce overheads during the waiting periods. It is strictly slower when executed on a server that is not very busy! It must be, because it does the exact same work in the exact same order as the ordinary synchronous version, except now with extra state machinery and complex error handling woven throughout by the compiler.

Scalability is not everyone's concern. Scalability is for the FAANG sized companies. I care about the individual user experience, and async does nothing for that by default.

I mean, sure, you can write much more verbose code along the lines of:

        var p_t = GetUserPermission(username);
        var c_t = GetServerConfig();
        var m_t = GetMessageOfTheDay();
        await Task.WhenAll(new Task[] { p_t, c_t, m_t });
        var p = p_t.Result;
        var c = c_t.Result;
        var m = m_t.Result;

But noone does this, for some values of noone. I've never seen code like this in the field.

In fact, let's test this. I'm reviewing an asynchronous ASP.NET application developed in 2020 right now. It's a large app, with literally thousands of uses of the "await" keyword, at least 3500 files use it.

The only use of "Task" static methods are seven uses of FromResult(). That's it. Zero uses of WaitAll(), WaitAny(), or ContinueWith()!

This is typical.

It's not that asynchronous programming is hard, it's that it is unergonomic to gain a latency benefit out of it. Most applications need lower latency, not higher throughput. Hence, for most programmers, most of the time, asynchronous programming is next to useless. It's just extra noise and more failure modes.

Re: Comparison of Rust async and Linux thread context switch time and memory use

#89

Post by pron » Fri, Feb 12, 2021, 10:16 AM UTC

The throughput increase in I/O scenarios with many tasks is due to the number of supported concurrent processes and Little's law; it has little to do with context switching time, which has a negligible impact on the throughput in these use-cases: https://inside.java/2020/08/07/loom-performance/

Low context switch latency only matters when the number of tasks is very small (their data all fits in the cache), and the workload is entirely computational. Otherwise, even the fastest implementation is ~60 ns, which is the cost of a cache-miss, and the compiler can't optimise things into a simple goto because the dispatch goes through a scheduler that has a megamorphic call-site.

So memory is much more important for I/O use-case throughput, and while it is true that the kernel doesn't commit the full stack memory on thread creation, it's misleading to think that you get good memory usage. For one, once the memory is committed, it's never uncommitted (although it can be paged out). For another, the granularity is that of a page, i.e. at least 4K, which can often be much higher than what a task requires.

> It is hard to pin down exactly how the alleged advantages would arise.

For I/O use-cases the answer is here:

> the async version uses about 1/20th as much memory as the threaded version.

This could translate to 20x throughput -- due to Little's law -- although usually less because there are other limits, like network saturation.

Re: Comparison of Rust async and Linux thread context switch time and memory use

#90

Post by ceronman » Fri, Feb 12, 2021, 10:16 AM UTC

lxpz wrote: post #75

> People often see that there's some theoretical benefit of async and then they accept far less ergonomic coding styles and the additional bug classes that only happen on async due to accidental blocking etc... despite the fact that when you consider a real-world deployed application, those "benefits" become indistinguishable from noise. However, due to the additional bug classes and worse ergonomics, there is now le…

It seems to me that these are two orthogonal topics. One thing is how you represent tasks, either using OS threads or async tasks. And the other is how you structure concurrency. Maybe I'm missing something, but I think there is nothing preventing the use of those structured concurrency patterns using OS threads as the base for tasks. Then you get some nice benefits of doing this such as proper stack-traces and easier debugging.

The killer use case for async tasks is when you need hyper-concurrency, e.g. hundreds of thousands of concurrent tasks. In that case, as the article mentions, you can't use OS threads anymore. Of course there are some use cases requiring this level of concurrency, messaging servers come to my mind, but there are also many, many use cases were you need a lower level of concurrency, like a few hundred concurrent tasks max. In those cases I think using OS threads can work pretty well with less complexity.