Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rust Libs Blitz

blog.rust-lang.org

Post reply on HN
Print view View original

51–60 of 125 posts

Re: The Rust Libs Blitz

#51

Post by eridius »

agwa wrote: post #48
Manishearth wrote: post #35

Earlier quoted context omitted.

It's pretty easy to audit a couple lines of unsafe in a library. It is not easy to do the same with a C library where it's basically prone to overflow issues everywhere. These are vastly different issues. Yes, we should totally be strict on unsafe code. No, it is not the end of the world when the stdlib hashmap uses unsafe code. Unsafe is designed exactly for this purpose, dealing with the innards of safe abstraction…

> It's pretty easy to audit a couple lines of unsafe in a library. You have to audit more than just the lines within the unsafe block. For example, I discovered a buffer overflow in a Rust library this week that was caused by an integer overflow outside the unsafe block[1]. The unsafe code itself was written correctly. [1] https://github.com/RustSec/advisory-db/blob/master/crates/ba...

Sounds to me like the unsafe block had the wrong scope. If safe Rust called a function with a buffer, and that buffer was too small, and that function internally used unsafe code to write to the buffer, then that function is leaking the unsafety past the `unsafe {}` scope, which is incorrect. So that function should be marked as `unsafe`, and the calling code (which calculates the buffer) is then responsible for ensuring that it's invoking the unsafe function with an appropriately-sized buffer.

Re: The Rust Libs Blitz

#52

Post by Animats »

Manishearth wrote: post #24
Animats wrote: post #21

Earlier quoted context omitted.

Let's see the benchmarks justifying the use of "unsafe" for maps. Maybe there's a better way to do it without much of a performance penalty. It's a good way to find out what optimizations the compiler is missing. It may even turn out that unsafe code written early no longer is a performance win, since the Rust compiler is getting better at optimizing out redundant subscript checks. When you start looking through Rust…

(The hashmap that is used in the winning rust benchmarksgame entry is 100% safe, fwiw) > When you start looking through Rust libraries, "unsafe" turns up way too often. You keep making this claim without substantiation. Yes, there is some level of unnecessary unsafe, but certainly not "way too often". I recall going through all the crates in my .cargo and finding very little unnecessary unsafe, and showing you the au…

We've seen this unsubstantiated denial before on YC, at "Why I’m dropping Rust"[1] and "Rust sucks if I fail to write X".[2] I once started going through the Rust library packages and listed uses of "unsafe". Try doing that.

The previous discussions made a few things clear:

- The big design-level problems with data structure safety are 1) partially initialized arrays, and 2) backlinks. The first is needed for growing arrays in Vec, and the second is needed for doubly-linked lists and some kinds of trees. It's very hard to handle either of those in safe Rust. A very small number of packages need unsafe code for those functions, and those should be tightly controlled.

- Foreign code remains a problem, and is inherently unsafe when calling unsafe languages. The most downloaded Rust crate is "libc". What could possibly go wrong there? Was it really a good idea to import unsafe "strcpy" into Rust?

    pub fn strcpy(dst: *mut c_char, src: *const c_char) -> *mut c_char;
- There's a lot of use of "unsafe" code that's not strictly necessary. You find unsafe code like "from_utf8_unchecked", which then turns up in the JSON decoder at (https://github.com/rust-lang-deprecated/rustc-serialize/blob...). There are no comments on the safety of that. Is there some way to create bad JSON, get bad UTF-8 into a string, and cause trouble further upstream? I don't know, but somebody "optimized" there, and created a potential problem.

I could give many more examples.

Most of these problems are fixable. They're not inherent in Rust. Fixing them is important to Rust's credibility. If Rust is going to replace C++, which it should, the holes have to be plugged. It only takes one hole to create a security vulnerability.

Denial is not a river in Egypt.

[1] https://news.ycombinator.com/item?id=12474445 [2] https://news.ycombinator.com/item?id=13670366

Re: The Rust Libs Blitz

#53

Post by hackcrafter »

microcolonel wrote: post #45

Rust sorta has a de-facto code style. It'd be interesting to add tooling to cargo to make it obvious how to comply with the evolved standard style for Rust.

There is definitely a changing Rust definition of good code hygiene in terms of how to write code in the most "Rustic" way, but in terms of code formatting, there is already rustfmt[0]

[0] https://github.com/rust-lang-nursery/rustfmt

Re: The Rust Libs Blitz

#54

Post by Manishearth »

moosingin3space wrote: post #47
Manishearth wrote: post #37

Earlier quoted context omitted.

A guideline like "your tests should test all major functionalities of an API" can't be statically checked for example. It's not about the crate being non checkable, it's about the check being something that needs a human to look at.

I realize that statically checking for tests is reducible to the halting problem, but would it be possible to have some sort of code-coverage checker that would make sure all externally-facing functions have non-zero coverage? That could be used to construct a checklist for a library developer.

Yes, there are code coverage tools and stuff :)

This could be used, sure.

Re: The Rust Libs Blitz

#55

Post by Manishearth »

agwa wrote: post #48
Manishearth wrote: post #35

Earlier quoted context omitted.

It's pretty easy to audit a couple lines of unsafe in a library. It is not easy to do the same with a C library where it's basically prone to overflow issues everywhere. These are vastly different issues. Yes, we should totally be strict on unsafe code. No, it is not the end of the world when the stdlib hashmap uses unsafe code. Unsafe is designed exactly for this purpose, dealing with the innards of safe abstraction…

> It's pretty easy to audit a couple lines of unsafe in a library. You have to audit more than just the lines within the unsafe block. For example, I discovered a buffer overflow in a Rust library this week that was caused by an integer overflow outside the unsafe block[1]. The unsafe code itself was written correctly. [1] https://github.com/RustSec/advisory-db/blob/master/crates/ba...

I'm aware. When I say "audit a line of unsafe" that includes all the invariants that it relies on, which could be outside the block.

This is still not that hard.

Re: The Rust Libs Blitz

#56

Post by Manishearth »

eridius wrote: post #51
agwa wrote: post #48

Earlier quoted context omitted.

> It's pretty easy to audit a couple lines of unsafe in a library. You have to audit more than just the lines within the unsafe block. For example, I discovered a buffer overflow in a Rust library this week that was caused by an integer overflow outside the unsafe block[1]. The unsafe code itself was written correctly. [1] https://github.com/RustSec/advisory-db/blob/master/crates/ba...

Sounds to me like the unsafe block had the wrong scope. If safe Rust called a function with a buffer, and that buffer was too small, and that function internally used unsafe code to write to the buffer, then that function is leaking the unsafety past the `unsafe {}` scope, which is incorrect. So that function should be marked as `unsafe`, and the calling code (which calculates the buffer) is then responsible for ensu…

Not necessarily. This isn't a question with a single answer within the community; there are reasons to tightly scope unsafe, and there are reasons to scope it to the full extent of unsafe-affecting invariants.

Scope your unsafe however you want, but check the invariants when auditing, and make sure they don't escape the module.

Re: The Rust Libs Blitz

#57

Post by kibwen »

I'm going to use this opportunity to second the suggestion that people consider taking this year's Rust community survey: https://blog.rust-lang.org/2017/05/03/survey.html . I know it's mentioned in the post, but I figure the number of people reading the comments is much larger than the number who actually click through the link. :P And even if you don't or have never used Rust, we still value your feedback!

Re: The Rust Libs Blitz

#58

Post by lossolo »

Manishearth wrote: post #24
Animats wrote: post #21

Earlier quoted context omitted.

Let's see the benchmarks justifying the use of "unsafe" for maps. Maybe there's a better way to do it without much of a performance penalty. It's a good way to find out what optimizations the compiler is missing. It may even turn out that unsafe code written early no longer is a performance win, since the Rust compiler is getting better at optimizing out redundant subscript checks. When you start looking through Rust…

(The hashmap that is used in the winning rust benchmarksgame entry is 100% safe, fwiw) > When you start looking through Rust libraries, "unsafe" turns up way too often. You keep making this claim without substantiation. Yes, there is some level of unnecessary unsafe, but certainly not "way too often". I recall going through all the crates in my .cargo and finding very little unnecessary unsafe, and showing you the au…

> winning rust benchmarksgame entry

C is winning that benchmark, around 2x faster than Rust.

http://benchmarksgame.alioth.debian.org/u64q/regexredux.html

Re: The Rust Libs Blitz

#59

Post by crzwdjk »

Animats wrote: post #52
Manishearth wrote: post #24

Earlier quoted context omitted.

(The hashmap that is used in the winning rust benchmarksgame entry is 100% safe, fwiw) > When you start looking through Rust libraries, "unsafe" turns up way too often. You keep making this claim without substantiation. Yes, there is some level of unnecessary unsafe, but certainly not "way too often". I recall going through all the crates in my .cargo and finding very little unnecessary unsafe, and showing you the au…

We've seen this unsubstantiated denial before on YC, at "Why I’m dropping Rust"[1] and "Rust sucks if I fail to write X".[2] I once started going through the Rust library packages and listed uses of "unsafe". Try doing that. The previous discussions made a few things clear: - The big design-level problems with data structure safety are 1) partially initialized arrays, and 2) backlinks. The first is needed for growing…

> which then turns up in the JSON decoder at (https://github.com/rust-lang-deprecated/rustc-serialize/blob...). There are no comments on the safety of that.

Maybe there's a reason it's in a repository labeled "deprecated", namely that it's deprecated. The replacement serialization framework, serde, has exactly one appearance of "unsafe", in a function that is only compiled when you explicitly enable the "unstable" feature flag and that in fact appears to be a reasonably safe use of unsafe.

Re: The Rust Libs Blitz

#60

Post by Manishearth »

Animats wrote: post #52
Manishearth wrote: post #24

Earlier quoted context omitted.

(The hashmap that is used in the winning rust benchmarksgame entry is 100% safe, fwiw) > When you start looking through Rust libraries, "unsafe" turns up way too often. You keep making this claim without substantiation. Yes, there is some level of unnecessary unsafe, but certainly not "way too often". I recall going through all the crates in my .cargo and finding very little unnecessary unsafe, and showing you the au…

We've seen this unsubstantiated denial before on YC, at "Why I’m dropping Rust"[1] and "Rust sucks if I fail to write X".[2] I once started going through the Rust library packages and listed uses of "unsafe". Try doing that. The previous discussions made a few things clear: - The big design-level problems with data structure safety are 1) partially initialized arrays, and 2) backlinks. The first is needed for growing…

> I once started going through the Rust library packages and listed uses of "unsafe". Try doing that.

That's ... exactly what I did? Talk about denial. I went through the libraries in my .cargo folder (which filters for libraries that actually get used, not just random libraries out there). I linked you to that audit in the comment.

Yes, libstd contains a lot more unsafe, but that's kind of the raison d'etre of libstd -- to contain OS-abstractions and very common internally-unsafe abstractions. It's better to have one unsafe implementation of Vec (in a library with a lot of eyeballs on it) than to have ten that the ecosystem relies on.

> which then turns up in the JSON decoder at

rustc-serialize is deprecated. It was deprecated before 1.0, and went into maintenance mode. It was still kinda-maintained because of the difficulty of using serde on stable, but now it's basically going to be full maintenance mode.

Also, that line is trivially safe to execute on untrusted input; `char` is utf8. Yes, a comment would be nice, but like I said, maintenance mode.

It is possible to do what that function does in safe Rust today, but it needs an API that probably didn't exist ~two years ago when that library was actually relevant. Fixing.

> The most downloaded Rust crate is "libc". What could possibly go wrong there? Was it really a good idea to import unsafe "strcpy" into Rust?

libc is just bindings (to everything in libc). You still need to use unsafe to call those functions (everything in `extern "C"` is unsafe to call even if not marked explicitly as such). This is not a valid example.

> A very small number of packages need unsafe code for those functions, and those should be tightly controlled.

which is pretty true already? The "partially initialized arrays" problem is generally handled by just using Vec. Yes, you need unsafe to write vec, but then you can just build things out of it.

Backlinks turn up rarely -- if perf isn't involved folks just use Weak safely, but otherwise people use petgraph or something.

> I could give many more examples.

Go ahead then. You literally never have, and none of these examples are valid "unnecessary unsafe" for reasons I gave above. I did substantiate with an audit. I'm genuinely interested in finding places where we have too much unsafe code, because I'd like to avoid depending on those crates and/or fix them.

> Most of these problems are fixable. They're not inherent in Rust. Fixing them is important to Rust's credibility.

Sure. And folks are always looking to improve this. But this doesn't mean that there's some widespread problem of unsafe being used too much in Rust. I'm not denying that this shouldn't be improved, I'm denying your allegations that `"unsafe" turns up way too often`.

Post reply on HN