Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Emacs internals: Tagged pointers vs. C++ std:variant and LLVM (Part 3)

thecloudlet.github.io

Post reply on HN
Print view View original

11–20 of 39 posts

Re: Emacs internals: Tagged pointers vs. C++ std:variant and LLVM (Part 3)

#11

Post by tialaramex »

shadowgovt wrote: post #7
thecloudlet wrote: post #6

Earlier quoted context omitted.

Doing bitwise operations directly on raw pointers is a fast track to Undefined Behavior in standard C/C++. Emacs gets away with it largely due to its age, its heavy reliance on specific GCC behaviors/extensions, and how its build system configures compiler optimizations. In modern C++, the technically "correct" and safe way to spell this trick is exactly as you suggested: using uintptr_t (or intptr_t).

Is there a similar solution to doing this in Rust? I suppose inside `unsafe` you can do basically anything.

Unlike C++ all of Rust's primitive types get the same first class treatment as your user defined types and so the appropriate API is provided as methods on pointer types. For this you want ptr::map_addr which takes a callable (such as your own function for this mapping or a lambda) to fiddle with the pointer.

https://doc.rust-lang.org/std/primitive.pointer.html#method....

Rust's MIRI is able to run code which uses this (a strict provenance API) because although MIRI's pointers are some mysterious internal type, it can track that we mapped them to hide our tags, and then later mapped back from the tagged pointer to recover our "real" pointer and see that's fine.

This isn't an unsafe operation. Dereferencing a pointer is unsafe, but twiddling the bits is fine, it just means whoever writes the unsafe dereferencing part of your codebase needs to be very careful about these pointers e.g. making sure the ones you've smuggled a tag in aren't dereferenced 'cos that's Undefined Behaviour.

It's clear to me how this works in Rust, it's just unclear still in C++

Re: Emacs internals: Tagged pointers vs. C++ std:variant and LLVM (Part 3)

#12

Post by tialaramex »

simonask wrote: post #10
shadowgovt wrote: post #7

Earlier quoted context omitted.

Is there a similar solution to doing this in Rust? I suppose inside `unsafe` you can do basically anything.

Rust is basically in the same place as C++, i.e. provenance rules are currently ad-hoc/conventional, meaning that pointer tagging is a grey area.

Nope. Rust stabilized strict provenance over a year ago. Some details about aliasing aren't tied down, but so long as you can obey the strict provenance rules you're golden today in Rust to hide flags in pointers etc.

https://blog.rust-lang.org/2025/01/09/Rust-1.84.0/#strict-pr...

Re: Emacs internals: Tagged pointers vs. C++ std:variant and LLVM (Part 3)

#13

Post by tialaramex »

trws wrote: post #8
thecloudlet wrote: post #6

Earlier quoted context omitted.

Doing bitwise operations directly on raw pointers is a fast track to Undefined Behavior in standard C/C++. Emacs gets away with it largely due to its age, its heavy reliance on specific GCC behaviors/extensions, and how its build system configures compiler optimizations. In modern C++, the technically "correct" and safe way to spell this trick is exactly as you suggested: using uintptr_t (or intptr_t).

There’s a paper in flight to add a stdlib type to handle pointer tagging as well while preserving pointer provenance and so-forth. It’s currently best to use the intptr types, but the goal is to make it so that an implementation can provide specializations based on what bits of a pointer are insignificant, or even ignored, on a given target without user code having to be specialized. Not sure where it has landed sinc…

Given you aren't sure since SG1 this might be useless but... do you have a paper number? Or, more likely, know an author's name ?

Re: Emacs internals: Tagged pointers vs. C++ std:variant and LLVM (Part 3)

#14

Post by legobmw99 »

tialaramex wrote: post #13
trws wrote: post #8

Earlier quoted context omitted.

There’s a paper in flight to add a stdlib type to handle pointer tagging as well while preserving pointer provenance and so-forth. It’s currently best to use the intptr types, but the goal is to make it so that an implementation can provide specializations based on what bits of a pointer are insignificant, or even ignored, on a given target without user code having to be specialized. Not sure where it has landed sinc…

Given you aren't sure since SG1 this might be useless but... do you have a paper number? Or, more likely, know an author's name ?

Seems like its p3125r0

Re: Emacs internals: Tagged pointers vs. C++ std:variant and LLVM (Part 3)

#15

Post by trws »

tialaramex wrote: post #13
trws wrote: post #8

Earlier quoted context omitted.

There’s a paper in flight to add a stdlib type to handle pointer tagging as well while preserving pointer provenance and so-forth. It’s currently best to use the intptr types, but the goal is to make it so that an implementation can provide specializations based on what bits of a pointer are insignificant, or even ignored, on a given target without user code having to be specialized. Not sure where it has landed sinc…

Given you aren't sure since SG1 this might be useless but... do you have a paper number? Or, more likely, know an author's name ?

It’s Hana Dusikova’s paper IIRC.

Re: Emacs internals: Tagged pointers vs. C++ std:variant and LLVM (Part 3)

#16

Post by trws »

shadowgovt wrote: post #7
thecloudlet wrote: post #6

Earlier quoted context omitted.

Doing bitwise operations directly on raw pointers is a fast track to Undefined Behavior in standard C/C++. Emacs gets away with it largely due to its age, its heavy reliance on specific GCC behaviors/extensions, and how its build system configures compiler optimizations. In modern C++, the technically "correct" and safe way to spell this trick is exactly as you suggested: using uintptr_t (or intptr_t).

Is there a similar solution to doing this in Rust? I suppose inside `unsafe` you can do basically anything.

Everything else in the siblings is true, but remember that the language and std types in rust all do this already. Most of the time it’s better to use a native enum or optional/result because they do this in the compiler/lib. It’s only really worth it if you need more than a few types or need precise control of the representation for C interop or something.

Re: Emacs internals: Tagged pointers vs. C++ std:variant and LLVM (Part 3)

#17

Post by tialaramex »

legobmw99 wrote: post #14
tialaramex wrote: post #13

Earlier quoted context omitted.

Given you aren't sure since SG1 this might be useless but... do you have a paper number? Or, more likely, know an author's name ?

Seems like its p3125r0

Thanks! https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2025/p31...

(is the current version of that paper, the tracking ticket insisted there's a P3125R5 and that LEWG had seen it in 2025, but it isn't listed in a mailing so it might be a mirage)

You know it's a Hana paper because it wants this to be allowed at compile time (C++ constrexpr) but joking aside this seems like a nice approach for C++ which stays agnostic about future implementation details.

Re: Emacs internals: Tagged pointers vs. C++ std:variant and LLVM (Part 3)

#18

Post by jandrewrogers »

tialaramex wrote: post #3

It's not clear to me (and as an unsafe language it's not called out by your compiler if you do something illegal) what the correct way to spell this kind of trick is in C++ I had thought you need the pointer-sized integer types and mustn't do this directly to an actual pointer, but maybe I was wrong (in theory, obviously practice doesn't follow but that's a dangerous game)

The idiomatic way to safely do pointer tagging in C++ works through uintptr_t.

If you don't care about portability or using every theoretically available bit then it is trivial. A maximalist implementation must be architecture aware and isn't entirely knowable at compile-time. This makes standardization more complicated since the lowest common denominator is unnecessarily limited.

In C++ this really should be implemented through a tagged pointer wrapper class that abstracts the architectural assumptions and limitations.

Re: Emacs internals: Tagged pointers vs. C++ std:variant and LLVM (Part 3)

#19

Post by tialaramex »

trws wrote: post #16
shadowgovt wrote: post #7

Earlier quoted context omitted.

Is there a similar solution to doing this in Rust? I suppose inside `unsafe` you can do basically anything.

Everything else in the siblings is true, but remember that the language and std types in rust all do this already. Most of the time it’s better to use a native enum or optional/result because they do this in the compiler/lib. It’s only really worth it if you need more than a few types or need precise control of the representation for C interop or something.

I mean, kinda, sorta? Rust's guaranteed niche optimisation means Option [which might be Some(&T) or just None] is promised to be the same size in memory as &T the reference to a T

So that's one tiny use of this sort of idea which is guaranteed unnecessary in Rust, and indeed although it isn't guaranteed the optimiser will typically spot less obvious opportunities so that Option> which might be None, or Some(None) or Some(Some(true)) or Some(Some(false)) is the same size (one byte) as bool.

But hiding stuff in a pointer is applicable in places your Rust compiler won't try to take advantage unless you do something like this. A novel tiny String-like type I saw recently does this, https://crates.io/crates/cold-string ColdString is 8 bytes, if your text is 8 or fewer bytes of UTF-8 then you're done, that'll fit, but, if you have more text ColdString allocates on the heap to store not only your text but also its length and so it needs to actually "be" in some sense a raw pointer to that structure, but if the string is shorter that pointer is nonsense, we've hidden our text in the pointer itself.

Implementation requires knowing how pointers work, and how UTF-8 encoding works. I actually really like one of the other Rust tiny strings, CompactString but if you have a lot of very small strings (e.g. UK postcodes fit great) then ColdString might be as much as three times smaller than your existing Rust or C++ approach and it's really hard to beat that for such use cases.

Edited: To remove suggestion ColdString has a distinct storage capacity, this isn't intended as a conventional string buffer, it can't grow after creation

Re: Emacs internals: Tagged pointers vs. C++ std:variant and LLVM (Part 3)

#20

Post by VorpalWay »

thecloudlet wrote: post #6
tialaramex wrote: post #3

It's not clear to me (and as an unsafe language it's not called out by your compiler if you do something illegal) what the correct way to spell this kind of trick is in C++ I had thought you need the pointer-sized integer types and mustn't do this directly to an actual pointer, but maybe I was wrong (in theory, obviously practice doesn't follow but that's a dangerous game)

Doing bitwise operations directly on raw pointers is a fast track to Undefined Behavior in standard C/C++. Emacs gets away with it largely due to its age, its heavy reliance on specific GCC behaviors/extensions, and how its build system configures compiler optimizations. In modern C++, the technically "correct" and safe way to spell this trick is exactly as you suggested: using uintptr_t (or intptr_t).

Do (u)intptr_t preserve provenance? Or does this count as exposed provenance when you convert back and forth?

Maybe that is not the correct C++ terminology, I'm more familiar with how provenance works in Rust, where large parts of it got stabilised a little over a year ago. (What was stabilised was "strict provenance", which is a set of rules that if you abide them will definitely be correct, but it is possible the rules might be loosened in the future to be more lenient.)

https://doc.rust-lang.org/std/ptr/index.html#provenance

Post reply on HN