Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Lost Art of Structure Packing (2018)

catb.org

Post reply on HN
Print view View original

1–10 of 120 posts

Re: The Lost Art of Structure Packing (2018)

#3

Post by munchbunny »

If you do any development that requires touching Win32, structure packing and memory alignment is still very real.

I was recently doing that in the context of improving a language’s library support for something that had to go through the OS SDK. I don’t miss the days when that stuff was the norm.

Re: The Lost Art of Structure Packing (2018)

#4

Post by BubRoss »

munchbunny wrote: post #3

If you do any development that requires touching Win32, structure packing and memory alignment is still very real. I was recently doing that in the context of improving a language’s library support for something that had to go through the OS SDK. I don’t miss the days when that stuff was the norm.

I have done some win32 programming but haven't encountered struct packing or alignment being an issue, where does that pop up?

Re: The Lost Art of Structure Packing (2018)

#5

Post by elteto »

adrianN wrote: post #2

It would be nice if there was an annotation that just lets the compiler do all the optimization for me for the cases where I don't care about the memory layout of the struct. Just like the Rust compiler can do without repr(C)

Wait, forgive my ignorance, isn’t this the default? If you don’t care about the memory layout then won’t the compiler reorg / pad structure members to fulfill alignment?

Re: The Lost Art of Structure Packing (2018)

#6

Post by htfy96 »

adrianN wrote: post #2

It would be nice if there was an annotation that just lets the compiler do all the optimization for me for the cases where I don't care about the memory layout of the struct. Just like the Rust compiler can do without repr(C)

Automatic reordering of fields is great, but sometimes people know more about how they will be used (like frequently accessed groups of fields, or leave some fields in the first 256 bytes so a u8 relative index could be used to save space), so manual reordering still exists for a reason beyond packing.

Re: The Lost Art of Structure Packing (2018)

#7

Post by virgilp »

elteto wrote: post #5
adrianN wrote: post #2

It would be nice if there was an annotation that just lets the compiler do all the optimization for me for the cases where I don't care about the memory layout of the struct. Just like the Rust compiler can do without repr(C)

Wait, forgive my ignorance, isn’t this the default? If you don’t care about the memory layout then won’t the compiler reorg / pad structure members to fulfill alignment?

Pad - yes, typically. Reorg - seldom.

Re: The Lost Art of Structure Packing (2018)

#8

Post by dan-robertson »

adrianN wrote: post #2

It would be nice if there was an annotation that just lets the compiler do all the optimization for me for the cases where I don't care about the memory layout of the struct. Just like the Rust compiler can do without repr(C)

This is hard. If your struct definition is in a header file then the compiler needs to generate correct field offsets for anything that includes that header file, so it would need to always optimally pack structure the same way based only on the definition and not on the usage.

People will also do silly things like casting between types in ways that rely on similarly written structures having similar memory layouts. So C is probably a bad language to turn this on by default in

Re: The Lost Art of Structure Packing (2018)

#9

Post by Chabsff »

elteto wrote: post #5
adrianN wrote: post #2

It would be nice if there was an annotation that just lets the compiler do all the optimization for me for the cases where I don't care about the memory layout of the struct. Just like the Rust compiler can do without repr(C)

Wait, forgive my ignorance, isn’t this the default? If you don’t care about the memory layout then won’t the compiler reorg / pad structure members to fulfill alignment?

The rule needs to be deterministic accross compilers, since a library can be compiled using a compiler and linked in an executable using the same header.

Re: The Lost Art of Structure Packing (2018)

#10

Post by gambiting »

BubRoss wrote: post #4
munchbunny wrote: post #3

If you do any development that requires touching Win32, structure packing and memory alignment is still very real. I was recently doing that in the context of improving a language’s library support for something that had to go through the OS SDK. I don’t miss the days when that stuff was the norm.

I have done some win32 programming but haven't encountered struct packing or alignment being an issue, where does that pop up?

I work in video games, and very recently we had a sneaky bug in one of our AAA titles(that was already out!), where(in huge simplification) we had a struct that looked like:

  struct Obj
  {
    int foo;
    bool bar;
  }
then we were storing those in a custom hashmap using these as keys, where the hashing function was basically hashing bits of each stored object, without any awareness of what's in the object.

The bug was found when someone did something like:

  void DoSomething(Obj obj)
  {
    if(!map.has(obj))
      map.add(obj, new Whatever());
  
    map[obj].blah(); //CRASH null pointer exception
  }
I was like.....well, if there is no key "obj" in the map, we insert one....and yet literally one line after it doesn't have a value for that key??? How can this be?

Well, it can be because even though the struct looks like it takes 5 bytes, in reality it's 8 bytes because it's getting padded. So a naive hashing method that just looks at bits is hashing your 5 bytes of actual data + 3 bytes of garbage, which means that two "identical" objects are very unlikely to actually produce the same hash.

C++20 now has a "hashable" concept to help with this, but it still requires the programmer to be aware of structure packing.

Post reply on HN