Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing an OS in Rust: Async/Await

os.phil-opp.com

Post reply on HN
Print view View original

51–60 of 107 posts

Re: Writing an OS in Rust: Async/Await

#51

Post by yazaddaruvala »

jerf wrote: post #44
jcranmer wrote: post #42

Earlier quoted context omitted.

There's a few different pieces to your question. Async/await is a programming paradigm that moves the state management of an asynchronous program into the compiler rather than being explicitly managed by the user. The paradigm is almost universally a win: if you don't want to write asynchronous code, just don't use it; it doesn't impact your code at all. A second question is how the language actually drives the execu…

"Now do note that this requires a far more complex executor model than most async/await toolkits (including this tutorial) provide." I'm assuming the async/await language support in Rust is intrinsically tied to a cooperative approach, which is the part I'm questioning. Obviously a kernel needs to be able to generically work in an asynchronous fashion... the question is, is this asynchronous fashion appropriate? If t…

At a high level, your concern is that OSes should not use cooperative multitasking for OS Processes. i.e. using Rust's async-await as the only technology within the OS Scheduler would be a mistake.

The article isn't discussing schedulers, but async-await within the OS codebase in general. Forgetting the scheduler, for example, Interrupt handling and driver code is generally async in nature, and at a base level needs to be cooperative.

    while let Some(byte) = keyboard.interrupt().await() {
        ...
    }
In general, certain drivers also need pre-emption, however, given all of the code is written and curated by the kernel devs, they can ensure the code cooperates well.

Re: Writing an OS in Rust: Async/Await

#52

Post by ccktlmazeltov »

zackmorris wrote: post #11

I've shied away from async/await because I haven't seen a good writeup on how to make it deterministic. Come to think of it, all of the times I've encountered it, there was no way to analyze the codepaths and prove that every exceptional situation and failure mode was handled. Maybe I missed something along the way? So my feeling about it is that it may turn out to be an evolutionary dead end, or anti-pattern at the…

This is a huge issue when trying to fuzz async code also.

Re: Writing an OS in Rust: Async/Await

#53

Post by kbenson »

Gladdyu wrote: post #49
kbenson wrote: post #46

Earlier quoted context omitted.

> If the Rust async/await support is indeed so flexible that with some work you could tie it together with a pre-emptive runtime of your own custom creation I think that's exactly what it is. AIUI it's basically an API for async/await (and maybe a default runtime?) that different back-ends can be plugged into. At least that's what I've come away with over the months of reading random bits about it here.

As described in the blog post, the rust compiler generates a state machine for every async function and generates the appropriate poll methods for the state transition. This is fundamentally at odds with the preemption, which would then have to indroduce new intermediate states into the diagram which it won't be able to do at runtime.

But if Rust is the operating system/kernel, whenever it decides to to schedule something is preemption for anything downstream, right?

I mean, you don't actually use preemption in the kernel right? Don't you have to handle all that yourself, since there's nothing higher level than you to handle it for you? In that respect, doesn't plugging in a Futures runtime that looks for device flags and stuff as appropriate and flags tasks to wake up/finish accomplish the same thing? (those are actual questions, not leading statements)

Re: Writing an OS in Rust: Async/Await

#54

Post by Gladdyu »

kbenson wrote: post #53
Gladdyu wrote: post #49

Earlier quoted context omitted.

As described in the blog post, the rust compiler generates a state machine for every async function and generates the appropriate poll methods for the state transition. This is fundamentally at odds with the preemption, which would then have to indroduce new intermediate states into the diagram which it won't be able to do at runtime.

But if Rust is the operating system/kernel, whenever it decides to to schedule something is preemption for anything downstream, right? I mean, you don't actually use preemption in the kernel right? Don't you have to handle all that yourself, since there's nothing higher level than you to handle it for you? In that respect, doesn't plugging in a Futures runtime that looks for device flags and stuff as appropriate and…

If you would write a basic scheduler, at some point you'd have to await the userspace code but you wouldn't have any way to force it to stop running. If the userspace code would enter an infinite loop it would hold the kernel thread forever. Within a constrained environment, eg. the kernel itself (and even that's sufficiently complex with loadable drivers that you might end up with bad interactions) I could see some use for async await, but you'd still need to be able to preempt untrusted code.

Re: Writing an OS in Rust: Async/Await

#55

Post by toast0 »

jerf wrote: post #15

Is async/await a good idea for an OS kernel, even a toy one? Cooperative multitasking tends to break down at scale, because the probability that all the "threads" you're cooperating with are playing nice goes to zero as the number of threads increases. An OS will tend to have a concentrated number of the pathological cases in it as it deals with hardware and all the other hardest timing & concurrency problems. It's a…

> Is async/await a good idea for an OS kernel, even a toy one? Cooperative multitasking tends to break down at scale,

> Embedding islands of cooperative multitasking in a sea of pre-emptive multitasking seems to make a lot more sense than the other way around.

In my reading of the post, the proposal is to do cooperative multitasking within the kernel, and preemptive multitasking between the kernel and the userspace. I think this is tenable, within the kernel, you pretty much need to play nice with the rest of the kernel anyway. Most kernels don't have effective protection for one part of the kernel causing problems with the rest; although, some do have limited protection against drivers that crash or loop.

Re: Writing an OS in Rust: Async/Await

#56

Post by jfkebwjsbx »

bitwize wrote: post #33
jerf wrote: post #15

Is async/await a good idea for an OS kernel, even a toy one? Cooperative multitasking tends to break down at scale, because the probability that all the "threads" you're cooperating with are playing nice goes to zero as the number of threads increases. An OS will tend to have a concentrated number of the pathological cases in it as it deals with hardware and all the other hardest timing & concurrency problems. It's a…

All I/O is asynchronous. Or rather, synchronous I/O is a subset of asynchronous. Often, in kernel I/O systems, you are communicating with, say, a disk controller that will take some time to process your request. You will want to do other stuff while the device finishes and process the results only when the device signals success or failure.

Async and sync I/O are not subsets of each other, although you can use one to implement the other.

The issue OP is discussing is cooperative vs. preemptive, not async vs. sync.

Re: Writing an OS in Rust: Async/Await

#57

Post by monocasa »

jerf wrote: post #15

Is async/await a good idea for an OS kernel, even a toy one? Cooperative multitasking tends to break down at scale, because the probability that all the "threads" you're cooperating with are playing nice goes to zero as the number of threads increases. An OS will tend to have a concentrated number of the pathological cases in it as it deals with hardware and all the other hardest timing & concurrency problems. It's a…

Coopertively scheduling across your other isolation boundaries is a pain point, but it's a great architecture for within a single context (ie. within the kernel, or within a single process).

Linux has a lot of cruft and isn't as async as some would like, but NT is pretty async underneath it all (although they still allow you to do syncronous work as well).

Re: Writing an OS in Rust: Async/Await

#58

Post by geogriffin »

toast0 wrote: post #55
jerf wrote: post #15

Is async/await a good idea for an OS kernel, even a toy one? Cooperative multitasking tends to break down at scale, because the probability that all the "threads" you're cooperating with are playing nice goes to zero as the number of threads increases. An OS will tend to have a concentrated number of the pathological cases in it as it deals with hardware and all the other hardest timing & concurrency problems. It's a…

> Is async/await a good idea for an OS kernel, even a toy one? Cooperative multitasking tends to break down at scale, > Embedding islands of cooperative multitasking in a sea of pre-emptive multitasking seems to make a lot more sense than the other way around. In my reading of the post, the proposal is to do cooperative multitasking within the kernel, and preemptive multitasking between the kernel and the userspace.…

well.. there is CONFIG_PREEMPT and the even broader CONFIG_PREEMPT_RT in linux [1].

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Re: Writing an OS in Rust: Async/Await

#59

Post by nindalf »

animalnewbie wrote: post #40
zackmorris wrote: post #11

I've shied away from async/await because I haven't seen a good writeup on how to make it deterministic. Come to think of it, all of the times I've encountered it, there was no way to analyze the codepaths and prove that every exceptional situation and failure mode was handled. Maybe I missed something along the way? So my feeling about it is that it may turn out to be an evolutionary dead end, or anti-pattern at the…

I am no expert and I wish there were a start from 0 guide with examples not involving external crates but my sense is that you can write your own executor and do whatever you want. (Maybe I'm wrong?)

The linked article finally helped me understand how Rust implements async. Give it a shot.

Re: Writing an OS in Rust: Async/Await

#60

Post by saagarjha »

hope-striker wrote: post #10
amelius wrote: post #4

An OS with async/await sounds awfully similar to Windows 3.1 with its cooperative multitasking model. What's old is new again ...

Didn't most old operating systems use cooperative multitasking? I remember, at least, that classic Mac OS (i.e. pre-OSX) didn't use preemptive multitasking, either. Anyway, this SO answer[0] explains why early Linux, much like the hobby kernel in this article, used cooperative scheduling inside the kernel, and only preempted user-space stuff. [0]: https://stackoverflow.com/a/16766562

Mac OS badly needed preemptive multitasking (and memory protection!), and got it when the NeXT bits were transformed into Mac OS X.
Post reply on HN