Supporting Linux kernel development in Rust • Hacker News Board

Re: Supporting Linux kernel development in Rust

#101

Post by geofft » Mon, Aug 31, 2020, 10:38 PM UTC

ghostwriter wrote: post #30

I won't get tired of repeating the same comment in every topic that suggests Rust to be a great replacement of C in existing projects , that it isn't. The safety guarantees that Rust provides are neither unique nor complete, and if we discuss the amount of effort necessary for bringing new interfaces like the one this article mentions ("one can define a kmalloc_for_rust() symbol containing an un-inlined version"), we…

As one of the presenters of this session, I'd like to reply by saying, this is cool! I'm hopeful it shows some good results! I'm not wedded to Rust in particular - it just seems to have the right balance of solving the particular problems the kernel wants to solve (there are plenty of other great languages like Go and Java and Python and Haskell and C++ that aren't suitable), easy to learn for people who have been working with C and have picked up an intuition for things like lifetimes and ownership that matter in C but aren't concrete, and popular (which is important because it means there are good learning resources, communities building things, etc.).

It looks like the "ATS Linux" work is happening on a Git fork at http://git.bejocama.org/ats-linux-5.7.2 . However, I don't see any commits there when I clone it - it looks like it's just the v5.7.2 tag from upstream. Do you know if there's any sample code that's been written demonstrating kernel code in ATS?

I think that, empirically, we have built something that works. You can check it out (https://github.com/fishinabarrel/linux-kernel-module-rust - see tests/* in particular). I'd be delighted to see folks who are excited about other languages build something in their language, too.

I'd like to correct two specific things about your comment:

> if we discuss the amount of effort necessary for bringing new interfaces like the one this article mentions ("one can define a kmalloc_for_rust() symbol containing an un-inlined version"), we should compare it to other existing solutions, like ATS

Here is the amount of effort necessary to work around that:

https://github.com/fishinabarrel/linux-kernel-module-rust/bl...

I took a semester-long MIT class on a proof assistant (Coq) and I thought it was great but I'm still not fully comfortable with it. I realize that Coq and ATS are different languages, but I would humbly submit that if you compare the amount of effort spent calling krealloc instead of kmalloc vs. learning ATS, the former is probably lower.

> without giving up on existing C interfaces and the toolchain

Looking at Chapter 8 "Interaction with C" from the ATS book http://ats-lang.github.io/DOCUMENT/INT2PROGINATS/HTML/c2016.... , it seems like ATS's interoperability with C is pretty comparable to Rust's https://doc.rust-lang.org/book/ch19-01-unsafe-rust.html#usin... - I don't think either of them require giving up on existing C interfaces or the C toolchain. One of the specific reasons to use Rust in this context instead of Go (which is a great language too) is that Rust is designed to fit in very closely with the C toolchain. The first paragraph of that chapter, which discusses how ATS's data structures can be zero-overhead bridged with C and you can view ATS as a better-typed frontend for C, seems to equally well apply to Rust.

(In fact, as mentioned in the article, we'd love to automatically generate bindings from existing C interfaces. It turns out that most interfaces lack documentation - certainly machine-parseable docs, but often human-readable docs too - that indicates their ownership properties and locking constraints and so forth, but definitely the best approach is to get this sort of information into C annotations instead of having manual bindings in Rust. As a bonus, if it turns out that another language like ATS or Sing# or whatever is a better fit than Rust, the information is right there.)

Re: Supporting Linux kernel development in Rust

#102

Post by kazinator » Mon, Aug 31, 2020, 10:38 PM UTC

kbenson wrote: post #61

ghostwriter wrote: post #30

I won't get tired of repeating the same comment in every topic that suggests Rust to be a great replacement of C in existing projects , that it isn't. The safety guarantees that Rust provides are neither unique nor complete, and if we discuss the amount of effort necessary for bringing new interfaces like the one this article mentions ("one can define a kmalloc_for_rust() symbol containing an un-inlined version"), we…

> I won't get tired of repeating the same comment in every topic that suggests Rust to be a great replacement of C in existing projects, that it isn't. That's not what this is about, so even though you aren't tired of posting this, I'm not sure it's warranted. This is about the kernel supporting other languages in select places where it makes sense, such as kernel modules, where there's already an API in place to con…

> They are simply making sure those interfaces are not actively hostile to languages that aren't C ...

I.e. not actively small and fast, like an OS should be.

Re: Supporting Linux kernel development in Rust

#103

Post by Foxboron » Mon, Aug 31, 2020, 10:39 PM UTC

boogies wrote: post #56

ColanR wrote: post #35

Earlier quoted context omitted.

I think you were being "inaccurate and hyperbolic". > Linux is not "forcing adoption of HDCP" (it's just something Linux has a driver for) What they actually said in the article: > Historically, some features began as optional ones until they reached total functionality. Then they became forced and difficult to patch out. Even if this does not happen in the case of HDCP, we remain cautious about such implementations.

I was going to comment similarly. The comment could be described as “inaccurate and hyperbolic”: if spelling counts, the second link does not say Linux was “forcing adoption [emphasis added] of HDCP.” Systemd stinks and I don’t care how it’s spelled as long as it’s sent to … nvm. And the Hyperbola devs have shown up, done the work, and ported pacman to their kernel, not making the mistake of listening to hyperbolic h…

> And the Hyperbola devs have shown up, done the work, and ported pacman to their kernel, not making the mistake of listening to hyperbolic haters.

Where? I was curious as there has been no upstream improvements. And there are no new code in what I assume is their upstream repository?

https://libregit.org/Hyperbola/hyperman

Re: Supporting Linux kernel development in Rust

#104

Post by Waterluvian » Mon, Aug 31, 2020, 10:43 PM UTC

ghostwriter wrote: post #30

I won't get tired of repeating the same comment in every topic that suggests Rust to be a great replacement of C in existing projects , that it isn't. The safety guarantees that Rust provides are neither unique nor complete, and if we discuss the amount of effort necessary for bringing new interfaces like the one this article mentions ("one can define a kmalloc_for_rust() symbol containing an un-inlined version"), we…

It honestly doesn’t matter a whole lot what makes people excited about contributing to Linux. I think it’s worth embracing whatever does as long as it’s at worst harmless.

Edit: I think what I’m trying to say is, let’s embrace what excites people.

Re: Supporting Linux kernel development in Rust

#105

Post by Spivak » Mon, Aug 31, 2020, 10:45 PM UTC

gok wrote: post #89

cbmuser wrote: post #80

If we get the Rust backend for gcc finalized and merged first, it would be much easier to get Rust code into the kernel: > https://github.com/philberty/gccrs/

That's a front end?

Compile C to Rust! Easiest language migration ever.

Re: Supporting Linux kernel development in Rust

#106

Post by geofft » Mon, Aug 31, 2020, 10:46 PM UTC

ghostwriter wrote: post #68

johncolanduoni wrote: post #52

Earlier quoted context omitted.

How does ATS provide safety guarantees without additional semantic information about the C code it’s linking to? If it does require that, then you’re in the same boat as Rust: your choices are trivial interoperation with no safety at the interfaces or encoding the semantics of the C interface in the new language. The only difference I can think of is built in ability to inline C from headers (based on your comment ab…

It's not only inclusion of the headers. ATS compiles to C, and it can inline C without FFI. The benefit that you receive is the ability to introduce gradual automated proofs around unchanged stable interfaces that are known to be safe. You start with inlining everything but the small core that is proven to be safe. Then you can iterate indefinitely at the desired pace to bring new layers of formally verified API in p…

> it can inline C without FFI

I'm not sure what you mean by "FFI" (I would apply the term "FFI" to the way ATS interoperates with C, too), but you can certainly get cross-language LTO between C and Rust, since they both compile to LLVM IR.

> you can iterate indefinitely at the desired pace to bring new layers of formally verified API in places of previously unverified C calls without breaking backward compatibility and expected runtime profiles

You can do this in Rust, too.

Re: Supporting Linux kernel development in Rust

#107

Post by rumanator » Mon, Aug 31, 2020, 10:47 PM UTC

dbaupp wrote: post #100

ghostwriter wrote: post #87

Earlier quoted context omitted.

In case of ATS, any C library can be treated as a "unsafe-marked" ATS library, so there's no problem with library support, the problem is with their formal verification. And Rust libraries regularly suffer from the same lack of formally-verified and proven-to-be-safe APIs.

That helps with the "library availability" sub-point specifically, yes. However, there's a more than that when evaluating a language, as suggested by my comment above. In any case, if one is discussing modern and safe languages, there's a dramatic difference between using a C library and using a library native to the language. Even ignoring safety, the ergonomics/developer experience is dramatically different and the…

> In any case, if one is discussing modern and safe languages, there's a dramatic difference between using a C library and using a library native to the language.

The dramatic nature of that difference is whether that library exists or not. Odds are, if exists then it's written in C. Thus this point is mute with regards to Rust because at best it's relegated to a nice-to-have, in the sense you can enjoy the same features that are already available in C but with language-specific assertions.

Re: Supporting Linux kernel development in Rust

#108

Post by simias » Mon, Aug 31, 2020, 10:52 PM UTC

ghostwriter wrote: post #30

I won't get tired of repeating the same comment in every topic that suggests Rust to be a great replacement of C in existing projects , that it isn't. The safety guarantees that Rust provides are neither unique nor complete, and if we discuss the amount of effort necessary for bringing new interfaces like the one this article mentions ("one can define a kmalloc_for_rust() symbol containing an un-inlined version"), we…

The Linux kernel is fairly modular, I could definitely imagine rewriting subsystems by subsystems with fairly good ergonomics and safety. Like rewrite serial_core.c in Rust and then let people write serial drivers in Rust. Then the SPI code. Then the I2C. Then USB etc...

Easier said than done of course, and definitely a huge effort especially for such a huge project as the Linux kernel, but there's also a significant amount of resources that could be allocated to that task by several large institutions.

I don't know if it's a good idea, or if ATS or something else could be better, but I think you're overly pessimistic.

Also "the safety guarantees that Rust provides are neither unique nor complete", while technically true, is also borderline FUD-y. At least you should expand on what you have in mind here, because I doubt that there exists a programing language that deals with low level hardware configuration that could ever provide "complete guarantees" about anything.

And beyond that, keep in mind that Rust provides a lot more than just safety guarantees, its syntax is lot more advanced that good old C in particular for anything dealing with types (including destructuring, matching, inference etc...). Even if you don't get "full" safety right away it can still be a much nicer language to code in (and I say that as somebody who knows C well and quite like it overall).

Re: Supporting Linux kernel development in Rust

#109

Post by fluffy87 » Mon, Aug 31, 2020, 10:53 PM UTC

johncolanduoni wrote: post #83

ghostwriter wrote: post #68

Earlier quoted context omitted.

It's not only inclusion of the headers. ATS compiles to C, and it can inline C without FFI. The benefit that you receive is the ability to introduce gradual automated proofs around unchanged stable interfaces that are known to be safe. You start with inlining everything but the small core that is proven to be safe. Then you can iterate indefinitely at the desired pace to bring new layers of formally verified API in p…

This may need fewer code changes (if the code in question is correct and the interfaces can possible be implemented in a safe way, that is) but it still requires developers to write proofs in a dependent type theory. I’ve only looked at ATS briefly, but I’m familiar with a lot of the other languages in the space (Idris, Fstar, etc.). Are ATS’s proof tactics drastically more productive/brief than those (F* in particul…

I’ve used ATS, Idris and Rust, and I find ATS way more cumbersome to use for writing proofs than idris.

I’ve never used ATS to do what the GP is suggesting of incrementally wrapping C with proofs, but I can’t imagine this being simpler than just rewriting the C in Rust. You wouldn’t get the same proofs, but you would get memory and thread safety, which for many apps would be an incremental improvement.

I haven’t taught ATS, but I found it harder to learn than Idris, and I can’t imagine C programmers which have a hard time with Rust learning it quicker than they would learn Idris or Rust.

Re: Supporting Linux kernel development in Rust

#110

Post by progval » Mon, Aug 31, 2020, 10:53 PM UTC

aspaceman wrote: post #64

JoshTriplett wrote: post #29

Earlier quoted context omitted.

The point of this LPC session was how to incrementally introduce Rust in the existing Linux kernel, with all its existing drivers and syscalls and similar. A rewrite would indeed be a daunting and problematic proposition. There are kernels written in Rust, such as Redox, but those are separate projects, and that's not what this conference session or article are talking about. Standing reminder: the Rust project is no…

> the Rust project is not a fan of rabid Rust over-evangelism (e.g. "Rewrite It In Rust", "Rust Evangelism Strike Force"), and sees it as damaging and unhelpful. I was really caught off guard by this stuff recently. I was around the rust subreddit a lot back in 2013-2015 and never saw stuff like that. Stepped out while working on other stuff and now it seems common. Super strange.

As an occasional rust programmer: you don't see this kind of behavior from the rust community. You see it when a random person opens an issue on a bugtracker you are subscribed to. Then it's usually prompty closed by a maintainer because the issue is very low effort, and the author doesn't offer to do any work themselves.

Examples: https://gitlab.com/fdroid/fdroidclient/-/issues/1049 and https://github.com/rapid7/metasploit-framework/issues/9092