Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Lost Art of C Structure Packing (2014)

catb.org

Post reply on HN
Print view View original

71–80 of 116 posts

Re: The Lost Art of C Structure Packing (2014)

#71

Post by vonmoltke »

blaisio wrote: post #61
vonmoltke wrote: post #38

Earlier quoted context omitted.

> The worst that you will get is padding at the end of the structure so that if two or more of them are arrayed, the first member is correctly aligned at all the array indices. This is not guaranteed. If you want to ensure all array member starting addresses are aligned for all compilers and platforms you need to add explicit dead space. You should also be using the fixed-width numeric types.

Can you give an example of a compiler and platform where this isn't true? I thought it was part of the standard, and I've never seen C code that didn't make the assumption that this is true.

IIRC, icc does not normally pad structures, since x86-64 supports unaligned loads fairly well. xlc on POWER generally tries to do what kazinator stated above because unaligned loads are expensive, if even possible, on those platforms.

Re: The Lost Art of C Structure Packing (2014)

#72

Post by Animats »

The bit-field feature of C structs is underutilized. People are still writing hex constants and using AND and OR to clear and set bits. Let the compiler do that; it's more readable. As of C99, there's named structure initialization, which makes bitfield constants more readable.

Re: The Lost Art of C Structure Packing (2014)

#74

Post by vonmoltke »

david-given wrote: post #66
kazinator wrote: post #54

Earlier quoted context omitted.

It's not guaranteed (as in "money back" or whatever). It's merely required by ISO C for conformance. If a struct is declared like this: struct foo { whatever_type_t first_member; // ... }; then an array of this type can also be declared: struct foo farray[42]; A conforming ISO C implementation has to ensure that farray[1].first_member, and farray[2].first_member, and so on, are all allocated such that they meet the a…

I agree with you, but it'd be nicer if it were easier to figure this out from the standard --- either I'm missing something or else you have to infer it from reading between the lines: 6.7.2.1 para 15 of the C11 standard: Within a structure object, the non-bit-field members and the units in which bit-fields reside have addresses that increase in the order in which they are declared. A pointer to a structure object, s…

> I agree with you, but it'd be nicer if it were easier to figure this out from the standard --- either I'm missing something or else you have to infer it from reading between the lines:

> So, on a platform with strict alignment, in order to achieve contiguous packing of structures (required by the last) and to achieve each structure member having a valid pointer (required by the first) the implementation must add enough optional padding (allowed by the middle) to make the size of the structure aligned.

It's not in the standard per se. It's a requirement for machine code to function on certain platforms. The standard basically just says that in all cases compilers have to generate machine code that works properly. It is imposed by the designers of the processor, not the C standard.

> I believe the parent is referring to the int8_t, int16_t types rather than the implementation-defined int, short, long types here.

This is indeed what I was talking about.

Re: The Lost Art of C Structure Packing (2014)

#75

Post by stephencanon »

Animats wrote: post #72

The bit-field feature of C structs is underutilized. People are still writing hex constants and using AND and OR to clear and set bits. Let the compiler do that; it's more readable. As of C99, there's named structure initialization, which makes bitfield constants more readable.

C11 §6.7.2.1: "The order of allocation of bit-fields within a unit (high-order to low-order or low-order to high-order) is implementation-defined. The alignment of the addressable storage unit is unspecified."

This makes bitfields quite difficult to use correctly when portability is of interest. It's often easier to write a couple inline get/set functions and use bit indices. If portability isn't a concern, then sure, have at it.

Re: The Lost Art of C Structure Packing (2014)

#77

Post by jlarocco »

dysoco wrote: post #41
kazinator wrote: post #31

TL;DR: to get smaller structures, don't do stuff like this: struct foo { char a; int b; char c; int d; }; but this: struct foo { int b; // or int b, c; int c; char a; char c; }; Basically if you sort the types by size in reverse descending order, you get optimal packing without messing with compiler-specific packing extensions that skew alignment and possibly bloat code. The worst that you will get is padding at the…

Alright this might be a stupid question, but if packing is about the order in which the fields are arranged... shouldn't the compiler optimize that? I mean, it does much more complex optimizations already doesn't it?

No.

A huge and important use of structs is to communicate with other things, like hardware, libraries, and other languages, and in in those cases it's very important that the struct matches what the other side expects. You don't want the compiler to rearrange your TCP packet, for example.

It might make sense to have a compiler extension to optimize packing for internal-only data structures, but I doubt any compilers bother. It's just another entry on the long list of things that C assumes the programmer will do themselves if it's important to them.

Re: The Lost Art of C Structure Packing (2014)

#78

Post by Animats »

stephencanon wrote: post #75
Animats wrote: post #72

The bit-field feature of C structs is underutilized. People are still writing hex constants and using AND and OR to clear and set bits. Let the compiler do that; it's more readable. As of C99, there's named structure initialization, which makes bitfield constants more readable.

C11 §6.7.2.1: "The order of allocation of bit-fields within a unit (high-order to low-order or low-order to high-order) is implementation-defined. The alignment of the addressable storage unit is unspecified." This makes bitfields quite difficult to use correctly when portability is of interest. It's often easier to write a couple inline get/set functions and use bit indices. If portability isn't a concern, then sure…

That's an escape clause for endian issues. If you write hex constants, you still have endian issues.

Re: The Lost Art of C Structure Packing (2014)

#79

Post by cheez »

dmit wrote: post #70
cheez wrote: post #37

Earlier quoted context omitted.

Most programmers would have known this a few years ago. Programmers today don't. So, it is a lost art.

There are more people alive who know this in the year 2016 than at any prior point in history.

I guess

Re: The Lost Art of C Structure Packing (2014)

#80

Post by khedoros »

Uptrenda wrote: post #4

Very nice guide. I remember seeing extremely similar code when I was working with raw TCP packets and I always wondered what all the esoteric rules were for working with raw struct memory. There was no where that seemed to explain things happening on this level so it kind of just never made any sense to me. Guess now I can finally figure out what all that code meant, awesome.

A few years ago, I ran into some situations where packed structs would be useful. I ended up writing a lot of little test programs to verify struct sizes based on different element orders, packing pragmas, etc. It was certainly illuminating, but it would've been helpful having a write-up like this to work from.

It got even more complicated when I threw unions into the mix. Knowing how the data is stored at the bit level becomes important in some of those cases.

Post reply on HN