Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neat Rust Tricks: Passing Closures to C

blog.seantheprogrammer.com

Post reply on HN
Print view View original

41–50 of 69 posts

Re: Neat Rust Tricks: Passing Closures to C

#41

Post by KenanSulayman »

Interestingly this is very similar to how I implemented passing closures into JavaScriptCore as hooks for JS class invocations ("function calls"). [0]

Essentially it's taking advantage of the fact that closures are static methods with "implicit" data pointers. It should be fairly obvious that this is a massive violation of safety and undefined behavior, and most likely to break when debugging symbols etc. are inserted.

The safest way to do this until Rust has figured out a stable-enough-ABI for closure passing would be a thread-local trampoline, I guess. Not very nice..

[0] https://github.com/psychonautwiki/rust-ul/blob/master/src/he...

Re: Neat Rust Tricks: Passing Closures to C

#42

Post by The_rationalist »

steveklabnik wrote: post #34
cellularmitosis wrote: post #31

Earlier quoted context omitted.

If Rust is intended to replace C, wouldn't you expect lots of this sort of literature? i.e. isn't this actually a sign of its _success_?

Also, being able to add Rust to an existing C or C++ codebase was an important design consideration. Big projects like Firefox aren’t just going to re-write millions of lines of code all at once.

And this is why rust is not going to succeed. I has not a great compatibility with c++.

Re: Neat Rust Tricks: Passing Closures to C

#43

Post by MaulingMonkey »

jonny383 wrote: post #37
cellularmitosis wrote: post #31

Earlier quoted context omitted.

If Rust is intended to replace C, wouldn't you expect lots of this sort of literature? i.e. isn't this actually a sign of its _success_?

No. It shows that people are still struggling with changing the way in which they write software to the "rust" way. The attitude of falling back to C or using unsafe rust just undermines the premise of the argument of why you should use rust. This is just like the node.js craze a few years ago - people will rant on trying to justify why you should use rust and write the "rust" way before realising that what they alre…

> It shows that people are still struggling with changing the way in which they write software to the "rust" way.

It shows no such thing.

I generally work on relatively small ~1MLOC C++ codebases. There are codebases out there measured in the hundreds of MLOC. These are not the comparatively tiny javascript codebases you find React used in - where additional milliseconds of download / parse / evaluation time has a measurable effect on your user retention statistics. There is no "near instant" at 100M+ LOC scales. There is only incremental rewrites, and incremental rewrites means making your new code talk to your old code, and to other people's existing code.

This means interop with existing C ABIs. There is no such thing as a C "replacement" that can't talk to an existing C ABI, or expose an existing C ABI.

Of course, a C ABI doesn't mean C. It's more frequently C++ in my ecosystem, for example. But it could just as easily be Rust, or any number of other languages capable of exposing a C ABI.

Re: Neat Rust Tricks: Passing Closures to C

#44

Post by wahern »

dfox wrote: post #35
tedunangst wrote: post #14

Now call qsort with a closure.

qsort(3) or even ftw(3) is the simple case. You can either dynamically generate trampoline code with exactly bounded dynamic scope (and even do the gcc-style executable stack hack) or simply stash the whole context in some TLS region and completely sidestep the whole issue. Side point: ftw(3) is much more interesting unix API to call from some FFI layer than qsort(3). And I spent about a year pestering people from Su…

And it appears that you succeeded. Awesome!

It seems Solaris has been adding many BSD and, especially, Linux compatibility APIs lately. It seems too little, too late; or perhaps the initiative is part of their effort to EoL Solaris, providing an upgrade path to Linux.

Re: Neat Rust Tricks: Passing Closures to C

#45

Post by jonny383 »

MaulingMonkey wrote: post #43
jonny383 wrote: post #37

Earlier quoted context omitted.

No. It shows that people are still struggling with changing the way in which they write software to the "rust" way. The attitude of falling back to C or using unsafe rust just undermines the premise of the argument of why you should use rust. This is just like the node.js craze a few years ago - people will rant on trying to justify why you should use rust and write the "rust" way before realising that what they alre…

> It shows that people are still struggling with changing the way in which they write software to the "rust" way. It shows no such thing. I generally work on relatively small ~1MLOC C++ codebases. There are codebases out there measured in the hundreds of MLOC. These are not the comparatively tiny javascript codebases you find React used in - where additional milliseconds of download / parse / evaluation time has a me…

>There are codebases out there measured in the hundreds of MLOC.

I agree with your argument, but I think in practice trying to "port" something with hundreds of MLOC is a losing battle (especially away from C). By the time you finished porting to rust (or your other language of the week), rust will likely have come and gone and will be been replaced by something either better or "better".

IMO people should spend less time trying to re-invent the wheel in rust and more time either improving C or the tooling / static analysis for C. It would avoid _so many_ of these issues.

Re: Neat Rust Tricks: Passing Closures to C

#46

Post by comex »

kazinator wrote: post #24
rabidferret wrote: post #21

Earlier quoted context omitted.

`void ( )(void , everything_else)` is in my personal experience a much more common API than that of `qsort` (probably for exactly the point you're pedantically trying to make), so I chose to focus on that API for this article. There's really no reason to pass a rust closure to `qsort` instead of sorting in Rust. That said, if there's demand for real world use cases that require passing Rust closures to C APIs that ta…

In any decent language, all functions are first-class, so if you want to use them as callbacks, you need that to work. That's still true even if the API takes a separate context pointer that is given to your function as an argument. There is still a function pointer there, and what you'd like to use as a function pointer is a function in your local language, and that's an object with an environment. Even if some inst…

Generating native function pointers on the fly:

- is inherently slow, because CPUs have separate data and instruction caches;

- is extra slow in practice because you need a separate allocation for executable memory (unless your stacks and heap are RWX, which is a terrible idea);

- is not portable, requiring architecture- and OS-specific code; and

- is not supported at all in many environments (of varying levels of braindeadness).

For a statically compiled language like Rust, it makes much more sense to use the context pointer.

Re: Neat Rust Tricks: Passing Closures to C

#49

Post by MaulingMonkey »

jonny383 wrote: post #45
MaulingMonkey wrote: post #43

Earlier quoted context omitted.

> It shows that people are still struggling with changing the way in which they write software to the "rust" way. It shows no such thing. I generally work on relatively small ~1MLOC C++ codebases. There are codebases out there measured in the hundreds of MLOC. These are not the comparatively tiny javascript codebases you find React used in - where additional milliseconds of download / parse / evaluation time has a me…

>There are codebases out there measured in the hundreds of MLOC. I agree with your argument, but I think in practice trying to "port" something with hundreds of MLOC is a losing battle (especially away from C). By the time you finished porting to rust (or your other language of the week), rust will likely have come and gone and will be been replaced by something either better or "better". IMO people should spend less…

I agree that porting 100MLOC is a losing battle no matter the language. And I'll concede I'm not certain Rust will have the staying power - although I hope it might.

And I'm all for more C tooling. Static analysis, fuzzing, sanitizers, valgrind, clang thread safety annotations, etc. are all wonderful tools I lean on heavily. But these are opt in, patchwork, platform specific, rife with false positives, false negatives, inconsistent, slow, painful to configure and use... I've wanted far more out of my C and C++ tooling than it's been able to give me for many years now. I'll frequently try out new attributes and annotations, only to curse when they fail to handle really trivial edge cases.

Meanwhile, Rust? It already catches things I didn't even realize I wanted to catch. Static checks opt-out into fast dynamic checks opt-out into heisenbugs in auditable unsafe blocks. The defaults are great.

I doubt C or C++'s tooling will reach the state of Rust, as frozen today, within the next decade. Smart people have tried long and hard to improve things, with quite middling results, convincing me it's a hard problem. I'm a bit more optimistic that it might catch up within the next century, but if I'm not long dead by then, I'll almost certainly be long retired. If it eventually does catch up, I suspect it'll have taken more than a few notes from Rust's approach.

I share your wariness of re-inventing the wheel, but the C & C++ static analysis ecosystem has left enough to be desired that I think it's warranted in this case. It's to rust's credit that they aren't re-inventing everything, and e.g. leverage LLVM for codegen, optimizations, debug info generation, etc.

Re: Neat Rust Tricks: Passing Closures to C

#50

Post by a1369209993 »

rabidferret wrote: post #27
mmastrac wrote: post #13

Earlier quoted context omitted.

Definitely. Technically this can also be done via static code trampolines that are mmap'd as well [1]. That approach has been used on iOS in the past to turn blocks into raw function pointers. If you have a platform that allows W+X on code (yikes!), you can do [2] as well. [1] https://github.com/plausiblelabs/plblockimp/blob/master/Sour... [2] https://www.mikeash.com/pyblog/friday-qa-2010-02-12-trampoli...

Anything that doesn't require W+X would need an entire page allocated per closure, wouldn't it?

Nope! You'd do something to the effect of:

  clo_code:
  4C8B1501100000  mov r10 [rel clo_code+0x1008]
  FF25F30F0000    jmp [rel clo_code+0x1000]
  0F1F00          nop3
  # one page away...
  struct clo_slot {
    void (*func)(void* _R10,...);
    void* data;
    };
Edit: to use r10 rather than rotating all the argument registers.
Post reply on HN