Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bugs You'll Probably Only Have in Rust

gankro.github.io

Post reply on HN
Print view View original

81–90 of 91 posts

Re: Bugs You'll Probably Only Have in Rust

#81

Post by vitalyd »

burntsushi wrote: post #79
vitalyd wrote: post #75

Earlier quoted context omitted.

> I guess I don't see the difference At a high and general level, yeah, it's all "unsafe". But, most conversations about unsafe don't talk about this aspect. So while what you say is true, I'm merely pointing out that this concurrency aspect doesn't seem to be mentioned much. And while it's implied at a high level, I think it's worth mentioning. Basically, there's no issue - I just think this should be called out mor…

Could you say more about why you see concurrency as a special thing to call out? I'm genuinely curious so that I can adapt how I explain this stuff in the future. It would help to understand the distinction you are seeing that I am not. :-)

Only because I've had several people ask me how Rust handles memory ordering after they've learned of Sync. Sync documentation focuses very narrowly on there being no data races against the type that implements it. There's also obviously a difference between a struct consisting of a single AtomicUInt (for example) and a struct consisting of a plain uint. The former can be published unsafely because it internally already provides the appropriate fences by virtue of the atomic. In the latter, it requires the publishing to provide the necessary fences. Again, I don't mean to say that doesn't make sense or is unexpected (after one thinks about it), but I'd urge a bit more focus on that part as well.

Given there's no official memory model, similar to say Java Memory Model, there's not much to go by (correct me if I'm wrong). The JMM, for instance, spells out what needs to happen for happens-before edges to be created. It also talks about racy publication of values. Granted there's no close analog to Rust's unsafe in Java. But saying "T: Sync means it's free of data races in safe code", while correct, is a slightly vacuous statement since that T interacts with other components. And yes, those components will likely involve unsafe code, and unsafe code has its own caveats, but still, I don't think it hurts to make this more pronounced. Concurrency is a special beast, rife with its own hard-to-debug hazards. Being a bit more verbose and possibly repetitive about the hazards won't hurt :).

Re: Bugs You'll Probably Only Have in Rust

#82

Post by vitalyd »

Manishearth wrote: post #80
vitalyd wrote: post #75

Earlier quoted context omitted.

> I guess I don't see the difference At a high and general level, yeah, it's all "unsafe". But, most conversations about unsafe don't talk about this aspect. So while what you say is true, I'm merely pointing out that this concurrency aspect doesn't seem to be mentioned much. And while it's implied at a high level, I think it's worth mentioning. Basically, there's no issue - I just think this should be called out mor…

You need raw pointer ops (or, well, dealing with UnsafeCell, which does raw pointer stuff), or syscalls for these too. Concurrency is not special here. There are all kinds of invariants unsafe code might be required to uphold. So yeah, we could mention concurrency, but then we could also mention UTF8, noalias, initialization, the vector length invariant, the HashMap robin hood invariant, various BTreeMap invariants,…

> Concurrency is not special here I beg to differ. Concurrency comes with its own bag of hazards, as I mentioned in my reply to burntsushi. Comparing its invariants with Vec's length invariant/HashMap's RH invariant, and any other single threaded/internal invariants misses the point.

Unsafe discussions that I've seen rarely talk about fences - they tend to focus on raw pointer ops, ffi, transmutes, unbound lifetimes, and in general, are single thread focused.

Re: Bugs You'll Probably Only Have in Rust

#83

Post by Manishearth »

vitalyd wrote: post #82
Manishearth wrote: post #80

Earlier quoted context omitted.

You need raw pointer ops (or, well, dealing with UnsafeCell, which does raw pointer stuff), or syscalls for these too. Concurrency is not special here. There are all kinds of invariants unsafe code might be required to uphold. So yeah, we could mention concurrency, but then we could also mention UTF8, noalias, initialization, the vector length invariant, the HashMap robin hood invariant, various BTreeMap invariants,…

> Concurrency is not special here I beg to differ. Concurrency comes with its own bag of hazards, as I mentioned in my reply to burntsushi. Comparing its invariants with Vec's length invariant/HashMap's RH invariant, and any other single threaded/internal invariants misses the point. Unsafe discussions that I've seen rarely talk about fences - they tend to focus on raw pointer ops, ffi, transmutes, unbound lifetimes,…

Right, but I can make the same point about other invariants. Each comes with its own bag of hazards. You can write pages about the robin hood invariants.

Concurrency is particularly complex, perhaps. I think one of the reasons you don't see that much discussion of this is that in general folks in Rust don't write that many internally-unsafe concurrent abstractions. There are a bunch of great safe building blocks out there (stdlib ones, rayon, crossbeam) which folks use for concurrency; it's very rare to build your own. So that might be it.

At least with the stuff I work on like 50% of the unsafe Rust discussions have been around thread safety and ordering and fences, but we're in that relatively rare situation where we need to build those abstractions, so perhaps it's just me who sees these discussions happening.

-----

It's also probably just that discussions introducing unsafe will deal with problems people are used to -- and memory safety is a far more "normal" problem than thread safety.

Re: Bugs You'll Probably Only Have in Rust

#84

Post by Manishearth »

vitalyd wrote: post #81
burntsushi wrote: post #79

Earlier quoted context omitted.

Could you say more about why you see concurrency as a special thing to call out? I'm genuinely curious so that I can adapt how I explain this stuff in the future. It would help to understand the distinction you are seeing that I am not. :-)

Only because I've had several people ask me how Rust handles memory ordering after they've learned of Sync. Sync documentation focuses very narrowly on there being no data races against the type that implements it. There's also obviously a difference between a struct consisting of a single AtomicUInt (for example) and a struct consisting of a plain uint. The former can be published unsafely because it internally alre…

I don't think it's vacuous. It's very important for compartmentalizing unsafe. It interacts with other components, and that statement tells you the responsibilities of both T implementing Sync, and abstractions accepting Sync types.

It is vacuous at a big picture level where you're trying to understand the complete thread safety story, but that was never what that statement was trying to convey.

----

I think documentation here can be improved, though. When I get more time one of the things I want to do is do a major revamp of the concurrency docs, including paragraphs on how the memory ordering stuff works. I'll include filling out the Sync docs in this, thanks for the feedback!

Re: Bugs You'll Probably Only Have in Rust

#85

Post by dbaupp »

vitalyd wrote: post #81
burntsushi wrote: post #79

Earlier quoted context omitted.

Could you say more about why you see concurrency as a special thing to call out? I'm genuinely curious so that I can adapt how I explain this stuff in the future. It would help to understand the distinction you are seeing that I am not. :-)

Only because I've had several people ask me how Rust handles memory ordering after they've learned of Sync. Sync documentation focuses very narrowly on there being no data races against the type that implements it. There's also obviously a difference between a struct consisting of a single AtomicUInt (for example) and a struct consisting of a plain uint. The former can be published unsafely because it internally alre…

The atomic type doesn't provide the necessary fences automatically. Operations on it have various sorts of fencing, but these do not necessarily connect in the way you think they do. The partial initialisation problem of creating a struct and publishing a pointer happens for the atomic types too, as they don't (and should not!) do everything with sequential consistency.

Re: Bugs You'll Probably Only Have in Rust

#86

Post by dbaupp »

jononor wrote: post #70
dbaupp wrote: post #59

Earlier quoted context omitted.

Could you be more specific about what you mean? The relevant (i.e. when it isn't premature optimisation: benchmarks have been run) alternatives here are 1. always risk memory unsafety, fast 2. crash on conditions that would trigger memory unsafety, slow 3. use 1 in release builds and 2 in debug/testing builds. This is effectively having an in-code sanitiser. You can definitely argue that using an actual sanitizer mig…

Another alternative: 4. Provide two codepaths, foo() without unsafe and foo_unsafe() for using the unsafe/fast paths. Then the consumer/caller can decide on the tradeoffs. Documentation should maybe prefer the safe, a benchmark can show the difference. Can be used in combination with your current approach of checking in debug builds. Some test cases should exist to exercise the unsafe checks, and show problems to avo…

That is another alternative although I am not sure it is a great one in the context of code where the unsafe operations expose a safe interface. That is, there should be no memory unsafety no matter how the API is used and the safety checks are just there to double check the library's internal invariants that the scoped `unsafe` is relying upon to be correct.

The downsides are that it is much more complicated to maintain: either ~2 times as much code, some ugly macros encompassing a lot of code or something along the lines of passing around SafeOps/UnsafeOps types everywhere.

Your suggestion definitely makes sense if foo_unsafe API isn't trying to expose a safe interface (as in, some combinations of parameters may result in memory unsafety), but that's not the case for the specific cases in URL and regex that are under consideration here.

Re: Bugs You'll Probably Only Have in Rust

#87

Post by vitalyd »

dbaupp wrote: post #85
vitalyd wrote: post #81

Earlier quoted context omitted.

Only because I've had several people ask me how Rust handles memory ordering after they've learned of Sync. Sync documentation focuses very narrowly on there being no data races against the type that implements it. There's also obviously a difference between a struct consisting of a single AtomicUInt (for example) and a struct consisting of a plain uint. The former can be published unsafely because it internally alre…

The atomic type doesn't provide the necessary fences automatically. Operations on it have various sorts of fencing, but these do not necessarily connect in the way you think they do. The partial initialisation problem of creating a struct and publishing a pointer happens for the atomic types too, as they don't (and should not!) do everything with sequential consistency.

Note that I didn't mention anything about it being automatic. It's merely a case of being more explicit about what may happen to the value.

Sequential consistency is irrelevant to the example I gave of a single valued struct. I wasn't talking about any ordering with respect other loads/stores.

Re: Bugs You'll Probably Only Have in Rust

#88

Post by dbaupp »

vitalyd wrote: post #87
dbaupp wrote: post #85

Earlier quoted context omitted.

The atomic type doesn't provide the necessary fences automatically. Operations on it have various sorts of fencing, but these do not necessarily connect in the way you think they do. The partial initialisation problem of creating a struct and publishing a pointer happens for the atomic types too, as they don't (and should not!) do everything with sequential consistency.

Note that I didn't mention anything about it being automatic. It's merely a case of being more explicit about what may happen to the value. Sequential consistency is irrelevant to the example I gave of a single valued struct. I wasn't talking about any ordering with respect other loads/stores.

Being explicit isn't relevant: a relaxed store to an atomic followed by publishing a pointer to it won't have the expected happens-before relationship (program order won't be respected).

In any case, the only reason to think about publishing is how loads and stores are ordered, so it isn't irrelevant. The single valued struct can still have the partial initialisation problem, it just appears as the single field being completely uninitialised.

Re: Bugs You'll Probably Only Have in Rust

#89

Post by vitalyd »

dbaupp wrote: post #88
vitalyd wrote: post #87

Earlier quoted context omitted.

Note that I didn't mention anything about it being automatic. It's merely a case of being more explicit about what may happen to the value. Sequential consistency is irrelevant to the example I gave of a single valued struct. I wasn't talking about any ordering with respect other loads/stores.

Being explicit isn't relevant: a relaxed store to an atomic followed by publishing a pointer to it won't have the expected happens-before relationship (program order won't be respected). In any case, the only reason to think about publishing is how loads and stores are ordered, so it isn't irrelevant. The single valued struct can still have the partial initialisation problem, it just appears as the single field being…

That's all true, but I think we're talking past each other a bit. My point isn't about what mechanics would be used or the type of memory order specified (that's the irrelevant part), but merely to highlight that this could use more attention in the docs. The atomic used may be viewed as a "lint" to a reader/user that something interesting may be going on with how this type is used - similar to how *mut isn't automatically Sync; it's an opportunity for people to pause and think things through. If you're looking at a type absent of any atomics or concurrency primitives, it's not apparent that it may be used like that (particularly since Sync is auto-derived and doesn't appear in the source code of the type).

I know one can look at this as being unimportant because only unsafe code that works with Sync types carries the burden of upholding the invariants. But, I still contend that memory ordering ought to be discussed, somewhere, while there's no official memory model documentation (not even sure that's in the plans, although I feel like I may have come across a github issue for that).

Anyway, I don't think I'm saying anything new in this reply over my others. If you feel existing docs are adequate in this regard, that's fine - we can agree to disagree :).

Re: Bugs You'll Probably Only Have in Rust

#90

Post by tonydanza »

sctb wrote: post #14
tonydanza wrote: post #2

are SJWs counted as a bug?

We've banned this account for posting only unsubstantively. We're happy to unban accounts if you email us at hn@ycombinator.com and we believe you'll post civilly and substantively in the future.

pls no banerino
Post reply on HN