Live data from Hacker News

A deep dive into Rust and C memory interoperability

notashes.me

61–70 of 84 posts

Re: A deep dive into Rust and C memory interoperability

#61
Me: “If we do it via FFI then there’s a possibility the program may continue working (because the underlying structs share the same memory layout? right? …right?)”

I didn't understand what was being said here; was he suggesting that you call libc free using FFI; which would be fine? I understand the interviewer asked about using Rust dealloc though. I think the FFI bit is confusing me.

Re: A deep dive into Rust and C memory interoperability

#62

The reason you are not seeing crashes when allocating with Rust and freeing with C (or vice versa) is that by default Rust also uses the libc allocator. https://stdrs.dev/nightly/x86_64-unknown-linux-gnu/src/std/s...

Miri and Valgrind will usually catch this kind of issue. I did lots of work mixing C and Rust and that tripped me as well at the beginning. I wrote about it if someone is interested: https://gaultier.github.io/blog/perhaps_rust_needs_defer.htm...

I’m a bit confused by this writeup. It seems to me that you’re experiencing two issues:

1. An object that’s really a Rust Vec under the hood is awkward when you try to pass ownership to C. ISTM you end up solving it by creating what is, in effect, a handle, and freeing that using a custom free function. Is this really a problem with Rust FFI? This style of API is very common in native C code, and it’s a lot more flexible and future-proof than having a library user call free() directly.

2. You’re consuming the API in Rust and being sad that there’s no native defer statement. But Rust can do RAII — could you build a little wrapper struct that implements Drop?

Re: A deep dive into Rust and C memory interoperability

#63
post #39
post #37

Earlier quoted context omitted.

While I usually hate all the accusations of writings being LLM generated, I find your example a bit odd as that phasing is very typical of ChatGPT, especially when it was glazing everyone after that one update they had to reverent. “It’s not just _________. It’s _________________.” This was in almost every response doubling down on the users ideas and blowing things out of proportion. Stuff like… “It’s not just a goo…

huh, interesting, I guess I haven't read enough of it to pick up on the patterns

Atomic Shrimp has an aside in a recent video about how to identify AI writing. It's worth a look https://youtu.be/VeD9dUUFl-E?t=668

He's not the only one to point out these things that LLMs (currently) tend to output, but this is one of the shorter overviews of the tells you can spot.

Re: A deep dive into Rust and C memory interoperability

#64

The reason you are not seeing crashes when allocating with Rust and freeing with C (or vice versa) is that by default Rust also uses the libc allocator. https://stdrs.dev/nightly/x86_64-unknown-linux-gnu/src/std/s...

It's funny. When I first tried Rust in 2018 they were still statically linking jemalloc into every binary rustc compiled by default, and that alone very much put me off of the language for a while. Apparently they did away with jemalloc in favor of the system allocator that same year but nonetheless when I came back to it years later I was very happy to learn of its removal.

By 2018 it was the system malloc by default, so that does not add up.

Re: A deep dive into Rust and C memory interoperability

#65
post #39
post #37

Earlier quoted context omitted.

While I usually hate all the accusations of writings being LLM generated, I find your example a bit odd as that phasing is very typical of ChatGPT, especially when it was glazing everyone after that one update they had to reverent. “It’s not just _________. It’s _________________.” This was in almost every response doubling down on the users ideas and blowing things out of proportion. Stuff like… “It’s not just a goo…

huh, interesting, I guess I haven't read enough of it to pick up on the patterns

E.g. the not x but why slop leader board

Re: A deep dive into Rust and C memory interoperability

#66
post #55

Earlier quoted context omitted.

That seems suspect to me. If I call reserve_exact I do actually mean reserve_exact and I want .capacity() to return with the argument I passed to reserve_exact(). This is commonly done when using Vec as a fixed capacity buffer and you don't want to add another field to whatever owns it that's semantically equivalent to .capacity(). I don't really care if the memory region is past capacity * size of:: (), but I do wan…

> This is commonly done when using Vec as a fixed capacity buffer and you don't want to add another field to whatever owns it that's semantically equivalent to .capacity(). The documentation for Vec already explains exactly what it's offering you, but lets explore, what exactly is the problem? You've said this is "commonly done" so doubtless you can point at examples for reference. Suppose a Goose is 40 bytes in size…

The condescension isn't appropriate here. I'm talking about using `Vec` as a convenient temporary storage without additional bookkeeping on top if the capacity() is meaningful. Like you said, Rust doesn't guarantee that because `reserve_exact` is not `reserve_exact`. In C++, the pattern is to resize() and shrink_to_fit(), which is implementation defined but when it's defined to do what it says, you can rely on it.

> Now, what disaster awaits in the common code you're talking about? Capacity is 6 and... there's capacity for 6 entries instead of 4

The capacity was expected to be 4 and not 6, which may be a logical error in code that requires it to be. If this wasn't a problem the docs wouldn't call it out as a potential problem.

Re: A deep dive into Rust and C memory interoperability

#67

Earlier quoted context omitted.

Jemalloc added over a megabyte to every project for only questionable gains, and it was awkward and unwieldy to remove it. While there are good reasons to use a different allocator depending on the project, Rust defaulting to this type of behavior failed a certain personal litmus test on what it wanted to be as a language in that it felt like it was fighting the system rather than integrating with it. It also does no…

> Jemalloc added over a megabyte to every project Right, but, so what? I can't imagine a practical reason this matters for the vast majority of situations. So either you're doing something pretty niche, or it's a purely aesthetic complaint.

So it's wasted space if you don't need it. Being a small amount of space to waste doesn't make it reasonable to do that.

Re: A deep dive into Rust and C memory interoperability

#68

Earlier quoted context omitted.

Jemalloc added over a megabyte to every project for only questionable gains, and it was awkward and unwieldy to remove it. While there are good reasons to use a different allocator depending on the project, Rust defaulting to this type of behavior failed a certain personal litmus test on what it wanted to be as a language in that it felt like it was fighting the system rather than integrating with it. It also does no…

> Jemalloc added over a megabyte to every project Right, but, so what? I can't imagine a practical reason this matters for the vast majority of situations. So either you're doing something pretty niche, or it's a purely aesthetic complaint.

> or it's a purely aesthetic complaint

Those are certainly valid. There are entire Linux distros whose mere existence is as the answer to what is only an aesthetic complaint.

Re: A deep dive into Rust and C memory interoperability

#69

Earlier quoted context omitted.

Jemalloc added over a megabyte to every project for only questionable gains, and it was awkward and unwieldy to remove it. While there are good reasons to use a different allocator depending on the project, Rust defaulting to this type of behavior failed a certain personal litmus test on what it wanted to be as a language in that it felt like it was fighting the system rather than integrating with it. It also does no…

> Jemalloc added over a megabyte to every project Right, but, so what? I can't imagine a practical reason this matters for the vast majority of situations. So either you're doing something pretty niche, or it's a purely aesthetic complaint.

Imagine you had a bunch of rust binaries on your system instead of one. And you're running on a 2 GB emmc.

Re: A deep dive into Rust and C memory interoperability

#70

One of the areas I wonder about this a lot is when integrating Rust code into Postgres which has its own allocator system. Mostly right now when we need to have complex data structures (non-Postgres data structures) that must live outside of the lexical scope we put them somewhere global and return a handle to the C code to reference the object. But with the upcoming support for passing an allocator to any data struc…

>But with the upcoming support for passing an allocator to any data structure (in the Rust standard library anyway) I think this gets a lot easier? Yes and no. Even within libstd, some things require A=GlobalAlloc, eg `std::io::Read::read_to_end(&mut Vec )` will only accept Vec . It cannot be changed to work with Vec because that change would make it not dyn-compatible (nee "object-safe"). And as you said it will cut…

If the `A` generic parameters were changed to be ?Sized, it would still be possible to make `read_to_end` support custom allocators by changing the signature to `read_to_end(&mut dyn Vec)`

Not sure if that is a breaking change though, it probably is because of a small detail, I'm not a rustc dev.

Post reply on HN