Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rust Libs Blitz

blog.rust-lang.org

Post reply on HN
Print view View original

41–50 of 125 posts

Re: The Rust Libs Blitz

#41

Post by skybrian »

stcredzero wrote: post #38

There’s a countervailing mindset which, in its harshest terms, says “the standard library is where code goes to die” This can be addressed in a language with sufficient annotation and good parser tools. In some future language, there should be a unification between the version control, the de-facto codesharing site, language/library versions, and syntax-driven tools to automatically rewrite code. It should be possibl…

The problem is that the number of possible combinations of versions grows rapidly as you add versions. This makes testing harder, as it spreads the community thin - everyone's using a different combo than everyone else.

To avoid that, you need to standardize on a blessed set of versions to be tested together, much like assembling a release of a Linux distro.

People will still swap in alternate versions of libraries occasionally, but keeping things mostly standard and a few cherry-picks is still better than everyone choosing differently.

Re: The Rust Libs Blitz

#42

Post by kbenson »

stcredzero wrote: post #38

There’s a countervailing mindset which, in its harshest terms, says “the standard library is where code goes to die” This can be addressed in a language with sufficient annotation and good parser tools. In some future language, there should be a unification between the version control, the de-facto codesharing site, language/library versions, and syntax-driven tools to automatically rewrite code. It should be possibl…

> In some future language, there should be a unification between the version control, the de-facto codesharing site, language/library versions, and syntax-driven tools to automatically rewrite code.

You actually get a lot of that in Perl 6, which identified modules not only by name, but by version and authority (so you can choose to load author Bar's version of Widget instead of author Foo's version.

Re: The Rust Libs Blitz

#43

Post by stcredzero »

skybrian wrote: post #41
stcredzero wrote: post #38

There’s a countervailing mindset which, in its harshest terms, says “the standard library is where code goes to die” This can be addressed in a language with sufficient annotation and good parser tools. In some future language, there should be a unification between the version control, the de-facto codesharing site, language/library versions, and syntax-driven tools to automatically rewrite code. It should be possibl…

The problem is that the number of possible combinations of versions grows rapidly as you add versions. This makes testing harder, as it spreads the community thin - everyone's using a different combo than everyone else. To avoid that, you need to standardize on a blessed set of versions to be tested together, much like assembling a release of a Linux distro. People will still swap in alternate versions of libraries o…

The problem is that the number of possible combinations of versions grows rapidly as you add versions.

The point is not to let people hang out in whatever obscure snowflake version-set they want to. The point is to make migration going forward as painless as possible. However, that expectation is not so much about the tooling as it is about the developer/language community.

To avoid that, you need to standardize on a blessed set of versions to be tested together, much like assembling a release of a Linux distro.

Yes, there should be this! However, you will still have some stragglers and outliers -- this is what the historical reality shows us. The point of such tooling is precisely to minimize the pool of stragglers, not to maximize them!

Re: The Rust Libs Blitz

#44

Post by dmix »

This is something Haskell could really benefit from. Largely just through writing documentation for common libraries.

A post was recently on the frontpage of HN about using Haskell in production [1] that divided the common documentation experience between "hard" and "soft" docs. Far too often with Haskell you only get the 'hard' docs where you get descriptions of functionality and functions but it lacks why (and cohesively how) you would want to use the various functionality.

This makes a strong assumption you are already deeply familiar with the usecase and implementation concept.

This may apply to Rust as well. Rust will likely attract experienced developers, much like Haskell, where in most cases a decent level of code quality would be anticipated. But one of the hardest things to get right as an OSS developer is documentation. You're often so busy with the burden of maintenance that the explanatory side gets sidelined. Especially as a library and the underlying language evolves. So I hope this is a priority focus during their reviews.

[1] https://news.ycombinator.com/item?id=14266462

Re: The Rust Libs Blitz

#46

Post by comex »

Manishearth wrote: post #35
Animats wrote: post #31

Earlier quoted context omitted.

Right. That's the road to buffer overflow exploits. The most recent CERT advisory reporting a buffer overflow exploit was April 17th, 2017.[1] About one per week is reported, year after year. Others not reported are probably being exploited. Rust can stop that, but one "unsafe" declaration can break Rust's safety. Hash maps need to be 100% safe code. They're complicated, and involve elaborate calculations that output…

It's pretty easy to audit a couple lines of unsafe in a library. It is not easy to do the same with a C library where it's basically prone to overflow issues everywhere. These are vastly different issues. Yes, we should totally be strict on unsafe code. No, it is not the end of the world when the stdlib hashmap uses unsafe code. Unsafe is designed exactly for this purpose, dealing with the innards of safe abstraction…

The main reason it requires unsafe is for memory layout optimizations. Logically, a hash table is an array of buckets, each of which is either a (hash, key, value) triplet or empty. The naive representation would be Vec>, but using Option would significantly increase the size of each bucket: it's only a one-byte tag to differentiate between Some and None, but alignment constraints would round the overhead up to at least the alignment of usize, typically 8. Instead, HashMap stores just (usize, K, V), and considers a bucket empty whenever 0 is stored in the hash slot; if 0 ever comes up as an actual hash value, it's changed to a different value before being stored.

To the extent I've described it, that might be possible to accomplish in Rust without unsafe code, using a safe abstraction around NonZero (an unsafe type which the compiler assumes will never be zero, allowing it to automatically use a zero value as a marker if the type is found in an enum, such as Option). However, HashMap goes a step further: it actually stores all the hashes contiguously in one array, and the (K, V) pairs in another. The nth index in the hashes array and in the (K, V) array together represent a single bucket, but storing them separately makes HashMap's typical access patterns somewhat nicer on the CPU's cache. However, this means that slots in the (K, V) array can contain either valid data (of arbitrary types, which might contain pointers etc.) or garbage, depending on a value stored somewhere completely different. There's no good 'automatic' way to make that safe.

I think the use of unsafe code here is fine - it's not that hard to audit, and if you don't audit you're in trouble anyway (hash table DoS is also a security risk) - but in the long term, it would be very interesting if Rust could integrate an optional theorem prover, with the ability to prove arbitrarily complex code safe, given sufficient annotations...

Re: The Rust Libs Blitz

#47

Post by moosingin3space »

Manishearth wrote: post #37
ronjouch wrote: post #27

Earlier quoted context omitted.

Beginner question: what would make a crate non -statically-checkable?

A guideline like "your tests should test all major functionalities of an API" can't be statically checked for example. It's not about the crate being non checkable, it's about the check being something that needs a human to look at.

I realize that statically checking for tests is reducible to the halting problem, but would it be possible to have some sort of code-coverage checker that would make sure all externally-facing functions have non-zero coverage? That could be used to construct a checklist for a library developer.

Re: The Rust Libs Blitz

#48

Post by agwa »

Manishearth wrote: post #35
Animats wrote: post #31

Earlier quoted context omitted.

Right. That's the road to buffer overflow exploits. The most recent CERT advisory reporting a buffer overflow exploit was April 17th, 2017.[1] About one per week is reported, year after year. Others not reported are probably being exploited. Rust can stop that, but one "unsafe" declaration can break Rust's safety. Hash maps need to be 100% safe code. They're complicated, and involve elaborate calculations that output…

It's pretty easy to audit a couple lines of unsafe in a library. It is not easy to do the same with a C library where it's basically prone to overflow issues everywhere. These are vastly different issues. Yes, we should totally be strict on unsafe code. No, it is not the end of the world when the stdlib hashmap uses unsafe code. Unsafe is designed exactly for this purpose, dealing with the innards of safe abstraction…

> It's pretty easy to audit a couple lines of unsafe in a library.

You have to audit more than just the lines within the unsafe block. For example, I discovered a buffer overflow in a Rust library this week that was caused by an integer overflow outside the unsafe block[1]. The unsafe code itself was written correctly.

[1] https://github.com/RustSec/advisory-db/blob/master/crates/ba...

Re: The Rust Libs Blitz

#49

Post by moosingin3space »

Manishearth wrote: post #19
jhasse wrote: post #12

My biggest gripe with the crate situation is that some of them require nightly. E.g. everything coroutines AFAIK.

The coroutine libraries are the codegen ones, yes? Coroutines are kinda more like a language feature that people have hacked libraries to do codegen for instead, so IMO it's pretty understandable that it needs nightly. Many of these libraries are prototyping designs that will eventually be proposed as part of the language.

Since Rust 1.15, most libraries that require nightly seem to be this kind of prototyping, at least to my eye.

Re: The Rust Libs Blitz

#50

Post by yazaddaruvala »

Manishearth wrote: post #24
Animats wrote: post #21

Earlier quoted context omitted.

Let's see the benchmarks justifying the use of "unsafe" for maps. Maybe there's a better way to do it without much of a performance penalty. It's a good way to find out what optimizations the compiler is missing. It may even turn out that unsafe code written early no longer is a performance win, since the Rust compiler is getting better at optimizing out redundant subscript checks. When you start looking through Rust…

(The hashmap that is used in the winning rust benchmarksgame entry is 100% safe, fwiw) > When you start looking through Rust libraries, "unsafe" turns up way too often. You keep making this claim without substantiation. Yes, there is some level of unnecessary unsafe, but certainly not "way too often". I recall going through all the crates in my .cargo and finding very little unnecessary unsafe, and showing you the au…

Do you think there is some room for education here? Would it be useful for Crates.io to show some "coverage" metric? eg. "unsafe lines = 5%" or "unsafe instructions = 0.01%"

I think that would be a pretty easy way to put this convo to bed yea?

Have there been talks about this before? Is it worth me opening an issue/RFC on Cargo?

Post reply on HN