Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A deep dive into Rust and C memory interoperability

notashes.me

Post reply on HN
Print view View original

51–60 of 84 posts

Re: A deep dive into Rust and C memory interoperability

#51

Post by eviks »

rectang wrote: post #26
ryanf wrote: post #12

This article looked interesting, but I bounced off it because the author appears to have made heavy use of an LLM to generate the text. How can I trust that the content is worth reading if a person didn't care enough to write it themselves?

I find it hard to believe that an LLM would have come up with this quote to start the article: > “Memory oppresses me.” - Severian, The Book of the New Sun That sort of artistic/humourous flourish isn't in character for an LLM.

But it's easy to believe that this is one of the few things the author added. Doesn't have to be 0% or 100%

Re: A deep dive into Rust and C memory interoperability

#52

Post by IshKebab »

phkahler wrote: post #3

Something I'd like to know for mixing Rust and C. I know it's possible to access a struct from both C and Rust code and have seen examples. But those all use accessor functions on the Rust side rather than accessing the members directly. Is it possible to define a structure in one of the languages and then via some wrapper or definitions be able to access it idiomatically in the other language? Can you point to some…

It's idiomatic to access struct fields directly in both languages. What more do you want?

Re: A deep dive into Rust and C memory interoperability

#53

Post by IshKebab »

tialaramex wrote: post #20
eatonphil wrote: post #6

One of the areas I wonder about this a lot is when integrating Rust code into Postgres which has its own allocator system. Mostly right now when we need to have complex data structures (non-Postgres data structures) that must live outside of the lexical scope we put them somewhere global and return a handle to the C code to reference the object. But with the upcoming support for passing an allocator to any data struc…

For me the most interesting thing in Allocator is that it's allowed to say OK, you wanted 185 bytes but I only have a 256 byte allocation here, so, here is 256 bytes. This means that e.g. a growable container type doesn't have to guess that your allocator probably loves powers of 2 and so it should try growing to 256 bytes not 185 bytes, it can ask for 185 bytes, get 256 and then pass that on to the user . Significan…

There's so much more information that code could give allocators but doesn't due to being stuck with ancient C APIs. Is the allocation likely to be short lived? Is speed or efficiency more important? Is it going to be accessed by multiple threads? Is it likely to grow in future?

Re: A deep dive into Rust and C memory interoperability

#54

Post by fc417fc802 »

School-Cotton wrote: post #43
CupricTea wrote: post #41

Earlier quoted context omitted.

It's funny. When I first tried Rust in 2018 they were still statically linking jemalloc into every binary rustc compiled by default, and that alone very much put me off of the language for a while. Apparently they did away with jemalloc in favor of the system allocator that same year but nonetheless when I came back to it years later I was very happy to learn of its removal.

> that alone very much put me off of the language for a while Why?

Seems like overstepping bounds. I expect a language runtime to make use of system provided facilities wherever feasible. I also expect it to provide hooks for me to link in alternatives.

If nothing else doing otherwise is likely to cause compatibility issues for someone at some point. For example see all the problems Go had with DIY syscalls pretty much everywhere except for Linux.

There's a legitimate question of whether the kernel ABI or the libc API qualifies as the system provided facilities on a linux box. But that uncertainty only furthers my latter point about compatibility.

Re: A deep dive into Rust and C memory interoperability

#55

Post by duped »

tialaramex wrote: post #20
eatonphil wrote: post #6

One of the areas I wonder about this a lot is when integrating Rust code into Postgres which has its own allocator system. Mostly right now when we need to have complex data structures (non-Postgres data structures) that must live outside of the lexical scope we put them somewhere global and return a handle to the C code to reference the object. But with the upcoming support for passing an allocator to any data struc…

For me the most interesting thing in Allocator is that it's allowed to say OK, you wanted 185 bytes but I only have a 256 byte allocation here, so, here is 256 bytes. This means that e.g. a growable container type doesn't have to guess that your allocator probably loves powers of 2 and so it should try growing to 256 bytes not 185 bytes, it can ask for 185 bytes, get 256 and then pass that on to the user . Significan…

That seems suspect to me. If I call reserve_exact I do actually mean reserve_exact and I want .capacity() to return with the argument I passed to reserve_exact(). This is commonly done when using Vec as a fixed capacity buffer and you don't want to add another field to whatever owns it that's semantically equivalent to .capacity().

I don't really care if the memory region is past capacity * size of::(), but I do want to be able to check if .len() == .capacity() and not be surprised

Re: A deep dive into Rust and C memory interoperability

#56

Post by tialaramex »

duped wrote: post #55
tialaramex wrote: post #20

Earlier quoted context omitted.

For me the most interesting thing in Allocator is that it's allowed to say OK, you wanted 185 bytes but I only have a 256 byte allocation here, so, here is 256 bytes. This means that e.g. a growable container type doesn't have to guess that your allocator probably loves powers of 2 and so it should try growing to 256 bytes not 185 bytes, it can ask for 185 bytes, get 256 and then pass that on to the user . Significan…

That seems suspect to me. If I call reserve_exact I do actually mean reserve_exact and I want .capacity() to return with the argument I passed to reserve_exact(). This is commonly done when using Vec as a fixed capacity buffer and you don't want to add another field to whatever owns it that's semantically equivalent to .capacity(). I don't really care if the memory region is past capacity * size of:: (), but I do wan…

> This is commonly done when using Vec as a fixed capacity buffer and you don't want to add another field to whatever owns it that's semantically equivalent to .capacity().

The documentation for Vec already explains exactly what it's offering you, but lets explore, what exactly is the problem? You've said this is "commonly done" so doubtless you can point at examples for reference.

Suppose a Goose is 40 bytes in size, and we aim to store say 4 of them, for some reason we decide to Vec::new() and then Vec::reserve_exact(..., 4) rather than more naturally (but with the same effect) asking Vec::with_capacity(4) but alas the allocator underpinning our system has 128 or 256 byte blocks to give, 4x40 = 160, too big for 128, so a 256 byte block is allocated and (a hypothetical future) Vec sets capacity to 6 anyway.

Now, what disaster awaits in the common code you're talking about? Capacity is 6 and... there's capacity for 6 entries instead of 4

Re: A deep dive into Rust and C memory interoperability

#57

Post by [deleted] »

CupricTea wrote: post #50
School-Cotton wrote: post #43

Earlier quoted context omitted.

> that alone very much put me off of the language for a while Why?

Jemalloc added over a megabyte to every project for only questionable gains, and it was awkward and unwieldy to remove it. While there are good reasons to use a different allocator depending on the project, Rust defaulting to this type of behavior failed a certain personal litmus test on what it wanted to be as a language in that it felt like it was fighting the system rather than integrating with it. It also does no…

[deleted]

Re: A deep dive into Rust and C memory interoperability

#58

Post by gblargg »

jml7c5 wrote: post #47
rectang wrote: post #26

Earlier quoted context omitted.

I find it hard to believe that an LLM would have come up with this quote to start the article: > “Memory oppresses me.” - Severian, The Book of the New Sun That sort of artistic/humourous flourish isn't in character for an LLM.

It looks like a mix of LLM and human-written content. The (human) author would have been the one who chose to put that quote there.

Which is even worse. It's like mixing broken glass with food. Why even waste food if it's going to be inedible anyway?

Re: A deep dive into Rust and C memory interoperability

#59

Post by Joel_Mckay »

broken_broken_ wrote: post #49
veber-alex wrote: post #14

The reason you are not seeing crashes when allocating with Rust and freeing with C (or vice versa) is that by default Rust also uses the libc allocator. https://stdrs.dev/nightly/x86_64-unknown-linux-gnu/src/std/s...

Miri and Valgrind will usually catch this kind of issue. I did lots of work mixing C and Rust and that tripped me as well at the beginning. I wrote about it if someone is interested: https://gaultier.github.io/blog/perhaps_rust_needs_defer.htm...

Valgrind sometimes has problems with encapsulated C libraries slow anonymous memory leaks. For example, the session caching in modern libcurl is obfuscated fairly well from novices.

Valgrind is good to run as a sanity check, but I prefer a month under a regression test hammer to validate something isn't going to slowly frag itself before scheduled maintenance. =3

Re: A deep dive into Rust and C memory interoperability

#60

Post by School-Cotton »

CupricTea wrote: post #50
School-Cotton wrote: post #43

Earlier quoted context omitted.

> that alone very much put me off of the language for a while Why?

Jemalloc added over a megabyte to every project for only questionable gains, and it was awkward and unwieldy to remove it. While there are good reasons to use a different allocator depending on the project, Rust defaulting to this type of behavior failed a certain personal litmus test on what it wanted to be as a language in that it felt like it was fighting the system rather than integrating with it. It also does no…

> Jemalloc added over a megabyte to every project

Right, but, so what? I can't imagine a practical reason this matters for the vast majority of situations. So either you're doing something pretty niche, or it's a purely aesthetic complaint.

Post reply on HN