Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing an OS in Rust: Async/Await

os.phil-opp.com

Post reply on HN
Print view View original

71–80 of 107 posts

Re: Writing an OS in Rust: Async/Await

#71

Post by lmm »

jfkebwjsbx wrote: post #68
lmm wrote: post #64

Earlier quoted context omitted.

async/await-based code is deterministic until you deliberately introduce the ability for tasks to race. Look at Noether for a language design that explicitly makes those steps. Whereas if you have channels (and don't have some kind of single ownership) then you have nondeterministic races right from the start. So I rather doubt that any such thin wrapper could be formed. Certainly as a human reader, Future-based code…

A state machine's control flow is even easier to follow, although more verbose.

I disagree, I find state machines almost impossible to reason about. Has it already been through state X? Who knows. Will it come back to this state in the future? Maybe. Is there a route from state Y to state Z? Shrug. State machines are effectively goto writ large, and there's a reason we switched to structured programming.

Re: Writing an OS in Rust: Async/Await

#72

Post by jcranmer »

jfkebwjsbx wrote: post #69
jcranmer wrote: post #67

Earlier quoted context omitted.

And what are those threads doing all the time? On N-core, 2-way SMT hardware (which describes pretty much all consumer hardware), the maximum amount of threads that can be usefully doing work in parallel is between N and 2 * N. Any more threads than that, and it must be the case that you expect those threads to spend a lot of time doing nothing to justify the cost of their creation. There's two main categories that c…

That does not follow. Long-running CPU-only tasks are a thing and they may not be aware of each other. With cooperative multitasking, launching something like a parallel compilation would make your machine unusable.

It is true that it's possible to screw your machine your over in that manner. I was pointing out that the large number of threads wasn't evidence in support of that fact, but rather the contrary; the threads have to already be using existing facilities that provide cooperative sleeping points.

Re: Writing an OS in Rust: Async/Await

#73

Post by elihu »

CuriousSkeptic wrote: post #62
jerf wrote: post #15

Is async/await a good idea for an OS kernel, even a toy one? Cooperative multitasking tends to break down at scale, because the probability that all the "threads" you're cooperating with are playing nice goes to zero as the number of threads increases. An OS will tend to have a concentrated number of the pathological cases in it as it deals with hardware and all the other hardest timing & concurrency problems. It's a…

Probably not at all related to the OP. But I found the thought entertaining, so offering it as such. One way to address you concern would be to guarantee that task always terminate in a bounded amount of instructions. One way to solve that would be offer a limited buffer in which the program instructions for the task might be expressed, and a language to express them in that is strongly normalizing (think something l…

The Linux kernel makes a somewhat similar distinction between code that's executing in an atomic context and code that isn't. Atomic code can get stuck in an infinite loop or whatever, but what it isn't allowed to do is to go to sleep (i.e. put itself into a sleeping state and invoke the scheduler).

Atomic tasks are generally expected to do a finite amount of work that can be finished in a short time without waiting for anything else (except maybe a spinlock, which should be held by some other similarly-atomic task that will finish in a short time).

Re: Writing an OS in Rust: Async/Await

#74

Post by ccktlmazeltov »

staticassertion wrote: post #61
ccktlmazeltov wrote: post #52

Earlier quoted context omitted.

This is a huge issue when trying to fuzz async code also.

Dropbox does, essentially, fuzzing of Rust async code and it is determistic. I couldn't find a post about it, sadly, but I know the information is out there somewhere.

source? Can't find anything on that either

Re: Writing an OS in Rust: Async/Await

#75

Post by RossBencina »

jerf wrote: post #15

Is async/await a good idea for an OS kernel, even a toy one? Cooperative multitasking tends to break down at scale, because the probability that all the "threads" you're cooperating with are playing nice goes to zero as the number of threads increases. An OS will tend to have a concentrated number of the pathological cases in it as it deals with hardware and all the other hardest timing & concurrency problems. It's a…

> Is async/await a good idea for an OS kernel, even a toy one?

You might want to look at Joe Duffy's posts about Microsoft's Midori OS. In particular the post "Asynchronous Everything":

http://joeduffyblog.com/2015/11/03/blogging-about-midori/

Their findings sound pretty compelling to me. Personally I'm convinced that eventually we'll see much more system-level use of async/await style mechanisms. Perhaps with Rust, perhaps with C++ once coroutines land in C++20.

Re: Writing an OS in Rust: Async/Await

#76

Post by xedrac »

lmm wrote: post #71
jfkebwjsbx wrote: post #68

Earlier quoted context omitted.

A state machine's control flow is even easier to follow, although more verbose.

I disagree, I find state machines almost impossible to reason about. Has it already been through state X? Who knows. Will it come back to this state in the future? Maybe. Is there a route from state Y to state Z? Shrug. State machines are effectively goto writ large, and there's a reason we switched to structured programming.

Well-designed deterministic state machines are incredibly easy to reason about. You simply render them as a graph and at a glance you can understand every possible transition and state.

> Will it come back to this state in the future?

If there's a path from the current state back to itself and it gets the correct inputs, yes. Otherwise no.

> Is there a route from state Y to state Z?

A simple glance at the graph will suffice to answer this.

Of course poorly written code, regardless of the constructs, will be hard to reason about.

Re: Writing an OS in Rust: Async/Await

#77

Post by staticassertion »

ccktlmazeltov wrote: post #74
staticassertion wrote: post #61

Earlier quoted context omitted.

Dropbox does, essentially, fuzzing of Rust async code and it is determistic. I couldn't find a post about it, sadly, but I know the information is out there somewhere.

source? Can't find anything on that either

https://dropbox.tech/infrastructure/rewriting-the-heart-of-o...

> The Control thread is designed to be entirely deterministic when its inputs and scheduling decisions are fixed. We use this property to fuzz it with pseudorandom simulation testing. With a seed for our random number generator, we can generate random initial filesystem state, schedules, and system perturbations and let the engine run to completion. Then, if we fail any of our sync correctness checks, we can always reproduce the bug from the original seed. We run millions of scenarios every day in our testing infrastructure.

Re: Writing an OS in Rust: Async/Await

#78

Post by Matthias247 »

jcranmer wrote: post #50
jerf wrote: post #44

Earlier quoted context omitted.

"Now do note that this requires a far more complex executor model than most async/await toolkits (including this tutorial) provide." I'm assuming the async/await language support in Rust is intrinsically tied to a cooperative approach, which is the part I'm questioning. Obviously a kernel needs to be able to generically work in an asynchronous fashion... the question is, is this asynchronous fashion appropriate? If t…

The Rust async/await amounts to the following: * Magic keywords (i.e., async and await) to manage the state machine aspect of the function for you. * A standard interface for describing an asynchronous task (core::future::Future). The magic keywords create an implementation of this interface. * An interface for saying that task progress can be made (core::task::Waker). That's it. There is no runtime provided, not eve…

> Rust's model definitely maps much more easily to a kernel, since the dominant pattern will be to drain a buffer, poke a device, and then wait for an interrupt to happen (the interrupt handler calling the wake function).

What is nice about Rusts model is that it prevents the continuation from running inline the event handler which signaled the completion. That avoids a ton of reentrancy issues and questions around "where does my code actually run"? That indeed makes it also interesting to use for interrupt handlers, since it is guaranteed that the code which waits for the interrupt to happen runs purely in the executors thread and will not accidentally take over the interrupt handler.

Re: Writing an OS in Rust: Async/Await

#79

Post by ahasani »

lmm wrote: post #71
jfkebwjsbx wrote: post #68

Earlier quoted context omitted.

A state machine's control flow is even easier to follow, although more verbose.

I disagree, I find state machines almost impossible to reason about. Has it already been through state X? Who knows. Will it come back to this state in the future? Maybe. Is there a route from state Y to state Z? Shrug. State machines are effectively goto writ large, and there's a reason we switched to structured programming.

What do you think about hierarchical state machine [1] ?

>Has it already been through state X?

Like everything else, logging

>Will it come back to this state in the future?

Mentioned in the comment before, there is a transition graph that you can do static analysis on, note that you might run into a halting problem

>Is there a route from state Y to state Z?

There is a transition graph

>State machines are effectively goto writ large...

quoting Prof Carl Hewitt of Actor Model, "goto is harmless"[2], function/procedure call is a goto, much like sending an named event while in particular state with/out parameters

[1] https://en.wikipedia.org/wiki/UML_state_machine

[2] https://www.youtube.com/watch?v=7erJ1DV_Tlo&t=34m54s

note: edited for formatting

Re: Writing an OS in Rust: Async/Await

#80

Post by spease »

jerf wrote: post #44
jcranmer wrote: post #42

Earlier quoted context omitted.

There's a few different pieces to your question. Async/await is a programming paradigm that moves the state management of an asynchronous program into the compiler rather than being explicitly managed by the user. The paradigm is almost universally a win: if you don't want to write asynchronous code, just don't use it; it doesn't impact your code at all. A second question is how the language actually drives the execu…

"Now do note that this requires a far more complex executor model than most async/await toolkits (including this tutorial) provide." I'm assuming the async/await language support in Rust is intrinsically tied to a cooperative approach, which is the part I'm questioning. Obviously a kernel needs to be able to generically work in an asynchronous fashion... the question is, is this asynchronous fashion appropriate? If t…

> But a fundamentally cooperative kernel would really worry me. You'd be looking at freezing the whole kernel, and likely the whole machine, if anything, anywhere, goes wrong with the cooperation. It won't take a lot of scale before that's a problem.

This is already the case, isn’t it? If you do a ‘while(!operation_complete) {}’ anywhere in the kernel and it never happens then I’d expect it’s pretty much done.

Whereas with async/await it’d be more likely that the coroutine would simply never resume. It might wind up being a memory leak, yeah, but not freeze your system.

Post reply on HN