The Lost Art of Structure Packing (2018) • Hacker News Board

Re: The Lost Art of Structure Packing (2018)

#91

Post by RealityVoid » Mon, Apr 27, 2020, 9:27 PM UTC

bregma wrote: post #41

gambiting wrote: post #10

Earlier quoted context omitted.

I work in video games, and very recently we had a sneaky bug in one of our AAA titles(that was already out!), where(in huge simplification) we had a struct that looked like: struct Obj { int foo; bool bar; } then we were storing those in a custom hashmap using these as keys, where the hashing function was basically hashing bits of each stored object, without any awareness of what's in the object. The bug was found wh…

If you rely on code containing undefined behaviour you're in for a world of butthurt sooner rather than later. There is no way in C++ to get at the padding bytes unless you're using undefined behaviour. How does the hash function work? Pointer aliasing using reinterpret_cast? Pointer aliasing using C-style casts? Typing punning through the old union switcheroo?

Ok, this will sound stupid then, but is pointer aliasing using casts undefined?

Re: The Lost Art of Structure Packing (2018)

#92

Post by SlowRobotAhead » Mon, Apr 27, 2020, 9:33 PM UTC

elteto wrote: post #5

adrianN wrote: post #2

It would be nice if there was an annotation that just lets the compiler do all the optimization for me for the cases where I don't care about the memory layout of the struct. Just like the Rust compiler can do without repr(C)

Wait, forgive my ignorance, isn’t this the default? If you don’t care about the memory layout then won’t the compiler reorg / pad structure members to fulfill alignment?

A practical reason you can’t just allow the compiler to do it is if you are doing goofy things like overlaying two structures, void pointer manipulation or casting one object to the format of another because you know the first n number of elements line up anyhow, or if you’re changing specific bytes as memory manipulation and not by the object itself.

Typically if I define a 8bit as the first element, I need to be certain those 8bits are first even if that wastes three more bytes to align on the next variable.

Re: The Lost Art of Structure Packing (2018)

#93

Post by samatman » Mon, Apr 27, 2020, 9:44 PM UTC

0xDEEPFAC wrote: post #68

Looks like what he really wants is to use Ada which has had much better support for low-level programming than C. Example: Word : constant := 4; -- storage element is byte, 4 bytes per word type State is (A,M,W,P); type Mode is (Fix, Dec, Exp, Signif); type Byte_Mask is array (0..7) of Boolean; type State_Mask is array (State) of Boolean; type Mode_Mask is array (Mode) of Boolean; type Program_Status_Word is record S…

For all its somewhat fussy verbosity, Ada really impresses me every time this sort of thing comes up.

I keep hoping the Zig developer will do a deep dive on Ada and bring over more of this kind of precise control. A language where I have this kind of control over layout, but can still spell `end record;` as `}`, is ideal for some projects I have in mind.

Re: The Lost Art of Structure Packing (2018)

#94

Post by icedchai » Mon, Apr 27, 2020, 10:26 PM UTC

saagarjha wrote: post #58

shawxe wrote: post #32

Earlier quoted context omitted.

sizeof(struct foo) is required to return the actual size, including padding, so struct foo foo_inst; memset(&foo_inst, 0, sizeof(foo_inst)); will certainly result in all padding bytes being set to zero.

sizeof will include padding, but there is no need for memset to actually perform a write that cannot be consistently observed.

Are you saying memset does not have to write the entire struct? I understand it can be optimized out, but thought it if it was actually executed it had to set all of the specified bytes.

Re: The Lost Art of Structure Packing (2018)

#95

Post by nly » Mon, Apr 27, 2020, 11:47 PM UTC

gambiting wrote: post #48

bregma wrote: post #41

Earlier quoted context omitted.

If you rely on code containing undefined behaviour you're in for a world of butthurt sooner rather than later. There is no way in C++ to get at the padding bytes unless you're using undefined behaviour. How does the hash function work? Pointer aliasing using reinterpret_cast? Pointer aliasing using C-style casts? Typing punning through the old union switcheroo?

I don't have the code in front of me, but something like int hash=0; for(int i=0;i (&obj)+i); return hash; Basically hashing each byte of the memory containing the object, regardless of what the object itself represents. We can argue whether that's a smart thing to do or not, but I wasn't in charge of implementing it - it's a relic from a codebase that's more than a decade old at this point. It's a simple hashing met…

This doesn't help you, because the contents of the padding bytes is not guaranteed to be anything in particular. Two structs containing identical field values can have different padding bytes. Reading the padding bytes is UB.

Just pack the struct and be done with it.

Re: The Lost Art of Structure Packing (2018)

#96

Post by nly » Mon, Apr 27, 2020, 11:50 PM UTC

adrianN wrote: post #2

It would be nice if there was an annotation that just lets the compiler do all the optimization for me for the cases where I don't care about the memory layout of the struct. Just like the Rust compiler can do without repr(C)

Great idea, but you have to define these optimizations and do them deterministically, even for debug builds, all the time - otherwise you'll never have a stable ABI.

Re: The Lost Art of Structure Packing (2018)

#97

Post by gpderetta » Tue, Apr 28, 2020, 12:04 AM UTC

saagarjha wrote: post #80

gpderetta wrote: post #79

Earlier quoted context omitted.

Well, the contents of the padding bits is legally observable (with a memcpy for example) so the compiler can't assume they are unovservable.

As far as I understand, padding bits are indeterminate until you observe them via a memcpy into a buffer, at which point they will collapse (only in the buffer) to some arbitrary but now-constant value. Is this correct?

I think you are right in the general case. Definitely padding bits are not preserved during copies, but I think that at least in some cases they might be preserved, at least I think that if you memset+memcpy, the memcpy is guaranteed to see the zeros (what if you want to end the lifetime of the object and simply reuse the storage as an array of chars?). But there is divergence between C and C++ and also an area quite in flux (the notion of lifetimes of objects and memory locations gets tweaked every standard in C++, not sure where we are at now).

edit: interestingly, Biriba is yet another game in the canasta family, my theory is that different groups were originally playing these differents variants, but when they gained knowledge of the more popular variant (burraco), they started playing it but kept calling it with the original name. I guess that up until the internet era, these games were mostly passed via oral knowledge in casual groups.

edit2: macchiavelli [1] is another very fun game of the same family, but a lot more puzzle solving oriented.

[1] https://en.wikipedia.org/wiki/Machiavelli_(Italian_card_game...

Re: The Lost Art of Structure Packing (2018)

#98

Post by kazinator » Tue, Apr 28, 2020, 12:11 AM UTC

scatters wrote: post #87

kazinator wrote: post #84

Earlier quoted context omitted.

I don't think so, unless we're mistaking memset for __builtin_memset.

GCC treats the two identically. See https://github.com/gcc-mirror/gcc/blob/229c0ef777161ec5adfb7...

I see; because the sixth argument (BOTH_P) in that macro call is true.

Re: The Lost Art of Structure Packing (2018)

#99

Post by saagarjha » Tue, Apr 28, 2020, 1:19 AM UTC

Asooka wrote: post #67

bregma wrote: post #41

Earlier quoted context omitted.

If you rely on code containing undefined behaviour you're in for a world of butthurt sooner rather than later. There is no way in C++ to get at the padding bytes unless you're using undefined behaviour. How does the hash function work? Pointer aliasing using reinterpret_cast? Pointer aliasing using C-style casts? Typing punning through the old union switcheroo?

You can trivially get at the padding bytes, just cast to char pointer. Their contents are undefined and there is absolutely no guarantee that they will remain constant, but they're there. Generally you have several possibilities for how to escape this problem, but the simplest is to just add the padding and a static assert that the sizeof the struct is what you expect.

> Their contents are undefined

No, their contents has an unspecified value.

Re: The Lost Art of Structure Packing (2018)

#100

Post by saagarjha » Tue, Apr 28, 2020, 1:29 AM UTC

RealityVoid wrote: post #91

bregma wrote: post #41

Earlier quoted context omitted.

If you rely on code containing undefined behaviour you're in for a world of butthurt sooner rather than later. There is no way in C++ to get at the padding bytes unless you're using undefined behaviour. How does the hash function work? Pointer aliasing using reinterpret_cast? Pointer aliasing using C-style casts? Typing punning through the old union switcheroo?

Ok, this will sound stupid then, but is pointer aliasing using casts undefined?

Not stupid–pointer aliasing using a cast is undefined if the types do not match, unless the type you are converting to is a void * , char * , or unsigned char *.