Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A deep dive into Rust and C memory interoperability

notashes.me

Post reply on HN
Print view View original

71–80 of 84 posts

Re: A deep dive into Rust and C memory interoperability

#71

Post by Arnavion »

RGBCube wrote: post #70
Arnavion wrote: post #11

Earlier quoted context omitted.

>But with the upcoming support for passing an allocator to any data structure (in the Rust standard library anyway) I think this gets a lot easier? Yes and no. Even within libstd, some things require A=GlobalAlloc, eg `std::io::Read::read_to_end(&mut Vec )` will only accept Vec . It cannot be changed to work with Vec because that change would make it not dyn-compatible (nee "object-safe"). And as you said it will cut…

If the `A` generic parameters were changed to be ?Sized, it would still be possible to make `read_to_end` support custom allocators by changing the signature to `read_to_end(&mut dyn Vec )` Not sure if that is a breaking change though, it probably is because of a small detail, I'm not a rustc dev.

First of all, `dyn Vec` is impossible. Vec is a concrete type, not a trait. I assume you meant `Vec`.

Second, no a `&mut Vec` is not convertible to `&mut Vec`. This kind of unsized coercion cannot work because it'll require a whole different Vec to be constructed, one which has an `allocator: dyn Allocator` field (which is unsized, and thus makes the Vec unsized) instead of an `allocator: A` field. The unsized coercion you're thinking of is for converting trait object references to unsized trait object references; here we're talking about a field behind a reference.

Re: A deep dive into Rust and C memory interoperability

#72

Post by tialaramex »

duped wrote: post #66
tialaramex wrote: post #56

Earlier quoted context omitted.

> This is commonly done when using Vec as a fixed capacity buffer and you don't want to add another field to whatever owns it that's semantically equivalent to .capacity(). The documentation for Vec already explains exactly what it's offering you, but lets explore, what exactly is the problem? You've said this is "commonly done" so doubtless you can point at examples for reference. Suppose a Goose is 40 bytes in size…

The condescension isn't appropriate here. I'm talking about using `Vec` as a convenient temporary storage without additional bookkeeping on top if the capacity() is meaningful. Like you said, Rust doesn't guarantee that because `reserve_exact` is not `reserve_exact`. In C++, the pattern is to resize() and shrink_to_fit(), which is implementation defined but when it's defined to do what it says, you can rely on it. >…

The condescension you've detected is because I doubt your main premise - that what you've described is "common" and so the defined behaviour will have a significant negative outcome. It's no surprise to me that you can offer no evidence for that premise whatsoever and instead just retreat to insisting you were correct anyway.

The resize + shrink_to_fit incantation sounds to me a lot like one of those "Sprinkle the volatile keyword until it works" ritualistic C++ practices not based in any facts.

Re: A deep dive into Rust and C memory interoperability

#73

Post by CupricTea »

saati wrote: post #64
CupricTea wrote: post #41

Earlier quoted context omitted.

It's funny. When I first tried Rust in 2018 they were still statically linking jemalloc into every binary rustc compiled by default, and that alone very much put me off of the language for a while. Apparently they did away with jemalloc in favor of the system allocator that same year but nonetheless when I came back to it years later I was very happy to learn of its removal.

By 2018 it was the system malloc by default, so that does not add up.

Jemalloc was removed in November 2018 [1] and didn't land in stable where it defaulted to the system allocator until Rust 1.32 released in January 2019 [2]

[1]: https://github.com/rust-lang/rust/pull/55238

[2]: https://blog.rust-lang.org/2019/01/17/Rust-1.32.0/

At the time in 2018, however, I was using Linux but now use Windows. I just now learned that Rust used the system allocator in Windows longer than for Linux [2] where it was still using jemalloc for the majority of 2018.

Re: A deep dive into Rust and C memory interoperability

#74

Post by fc417fc802 »

duped wrote: post #66
tialaramex wrote: post #56

Earlier quoted context omitted.

> This is commonly done when using Vec as a fixed capacity buffer and you don't want to add another field to whatever owns it that's semantically equivalent to .capacity(). The documentation for Vec already explains exactly what it's offering you, but lets explore, what exactly is the problem? You've said this is "commonly done" so doubtless you can point at examples for reference. Suppose a Goose is 40 bytes in size…

The condescension isn't appropriate here. I'm talking about using `Vec` as a convenient temporary storage without additional bookkeeping on top if the capacity() is meaningful. Like you said, Rust doesn't guarantee that because `reserve_exact` is not `reserve_exact`. In C++, the pattern is to resize() and shrink_to_fit(), which is implementation defined but when it's defined to do what it says, you can rely on it. >…

> the pattern is to resize() and shrink_to_fit(),

As someone who primarily writes C++ I would not expect that to work. I mean it's great if it does I guess (I don't really see the point?) but that would honestly surprise me.

I would _always_ expect to use >= for capacity comparisons and I don't understand what the downside would be. The entire point of these data structures is that they manage the memory for you. If you need precise control over memory layout then these are the wrong tools for the job.

Re: A deep dive into Rust and C memory interoperability

#75

Post by PhilipRoman »

School-Cotton wrote: post #43
CupricTea wrote: post #41

Earlier quoted context omitted.

It's funny. When I first tried Rust in 2018 they were still statically linking jemalloc into every binary rustc compiled by default, and that alone very much put me off of the language for a while. Apparently they did away with jemalloc in favor of the system allocator that same year but nonetheless when I came back to it years later I was very happy to learn of its removal.

> that alone very much put me off of the language for a while Why?

There are embedded Linux systems with a total image size of ~10MB (not image as in docker "image" but an actual disk image). It would be quite hard to justify the additional dependency.

Re: A deep dive into Rust and C memory interoperability

#76

Post by School-Cotton »

cozzyd wrote: post #69
School-Cotton wrote: post #60

Earlier quoted context omitted.

> Jemalloc added over a megabyte to every project Right, but, so what? I can't imagine a practical reason this matters for the vast majority of situations. So either you're doing something pretty niche, or it's a purely aesthetic complaint.

Imagine you had a bunch of rust binaries on your system instead of one. And you're running on a 2 GB emmc.

Okay, that's a niche scenario. I agree Rust during the statically linked jemalloc era might not be ideal in that scenario. But just because you can concoct some possible scenario in which it's not the best language to use doesn't mean you should write it off completely.

Re: A deep dive into Rust and C memory interoperability

#77

Post by School-Cotton »

PhilipRoman wrote: post #75
School-Cotton wrote: post #43

Earlier quoted context omitted.

> that alone very much put me off of the language for a while Why?

There are embedded Linux systems with a total image size of ~10MB (not image as in docker "image" but an actual disk image). It would be quite hard to justify the additional dependency.

Okay, then don't use a language with statically-linked jemalloc for those. Why is that a reason to write off the entire language for all purposes?

Re: A deep dive into Rust and C memory interoperability

#78

Post by 7e »

rectang wrote: post #16
7e wrote: post #5

Allocating memory with C and freeing it with Rust is silly. If you want to free a C-allocated pointer in Rust, just have Rust call back in to C. Expecting that allocators work identically in both runtimes is unreasonable and borderline insane. Heck, I wouldn't expect allocators to work the same even across releases of libc from the same vendor (or across releases of Rust's std).

I don't agree with your contemptuous framing. It's incorrect, and per the post's author, "dangerous" — but depending on your background it's not "silly" or "borderline insane". It's just naive, and writing a slab allocator as an exercise or making honest explorations like in this blog post will help cure the naivete.

It’s undefined behavior. It will never be stable. Investigating every permutation of zero-utility undefined behavior in the universe is borderline insane. Will the author next investigate exactly how a 2002 Fiat becomes inoperable after a head on collision with a 2025 Volkswagen? These are all deep dives into infinite chaos.

Re: A deep dive into Rust and C memory interoperability

#79

Post by rectang »

7e wrote: post #78
rectang wrote: post #16

Earlier quoted context omitted.

I don't agree with your contemptuous framing. It's incorrect, and per the post's author, "dangerous" — but depending on your background it's not "silly" or "borderline insane". It's just naive, and writing a slab allocator as an exercise or making honest explorations like in this blog post will help cure the naivete.

It’s undefined behavior. It will never be stable. Investigating every permutation of zero-utility undefined behavior in the universe is borderline insane. Will the author next investigate exactly how a 2002 Fiat becomes inoperable after a head on collision with a 2025 Volkswagen? These are all deep dives into infinite chaos.

I agree that we shouldn't mix allocators, and so does the post's author. Can you put your technical arguments into terms which aren't so dismissive of people's honest efforts to learn? We ought to be all on the same page. You could affirm and refine the post's conclusions and bring us together, rather than ridicule someone for ever entertaining a notion you consider "insane".

Here's what I got when I asked ChatGPT to rewrite your first comment to be as constructive as possible:

"Totally agree — relying on C and Rust to interoperate at the allocator level is risky at best. Allocating in C and freeing in Rust (or vice versa) assumes a level of compatibility that just isn’t guaranteed. Even within a single ecosystem, allocator behavior can change across versions — whether it’s different versions of libc or updates to Rust’s std. So expecting consistent behavior across language boundaries is, at the very least, unreliable. If you need to free a C-allocated pointer, the safest and cleanest approach is to call back into C to do it."

That's not a drop-in substitute for your original comment and "totally agree" is over-the-top cloying in the usual ChatGPT obsequious way, but I still think it's helpful in suggesting alternative framings.

Re: A deep dive into Rust and C memory interoperability

#80

Post by RGBCube »

Arnavion wrote: post #71
RGBCube wrote: post #70

Earlier quoted context omitted.

If the `A` generic parameters were changed to be ?Sized, it would still be possible to make `read_to_end` support custom allocators by changing the signature to `read_to_end(&mut dyn Vec )` Not sure if that is a breaking change though, it probably is because of a small detail, I'm not a rustc dev.

First of all, `dyn Vec` is impossible. Vec is a concrete type, not a trait. I assume you meant `Vec `. Second, no a `&mut Vec ` is not convertible to `&mut Vec `. This kind of unsized coercion cannot work because it'll require a whole different Vec to be constructed, one which has an `allocator: dyn Allocator` field (which is unsized, and thus makes the Vec unsized) instead of an `allocator: A` field. The unsized coe…

Sorry, I meant `&Vec`.

And no, it is possible. Here is an example that does it with BufReader, which has T: ?Sized and uses it as a field: https://play.rust-lang.org/?version=stable&mode=debug&editio...

Though it comes with a caveat that you can't take self by value, which is perfectly fine for this use case & is what a normal allocator-aware language does anyway.

Post reply on HN