Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Lost Art of C Structure Packing (2014)

catb.org

Post reply on HN
Print view View original

101–110 of 116 posts

Re: The Lost Art of C Structure Packing (2014)

#101

Post by valarauca1 »

dbaupp wrote: post #87
valarauca1 wrote: post #53

Earlier quoted context omitted.

>The optimization is present in other languages that did not start with this restriction, but it's too late for C now. Which? .NET is the only one I could find that appears to do some from of smart packing [1] The new hotness languages behave like: Go doesn't sort by size. It packs to the byte, but guarantees structures within structures will start on the 32/64bit boundary (depending on 32/64bit system) so it'll pad…

Rust doesn't ignore field size nor does it align to any particular boundary. E.g. struct Foo { x: u8 } is one byte and has alignment 1, and similarly struct Bar { x: u8, y: i16 } is 4 bytes and has alignment 2. The compiler doesn't give guarantees about structure layout so that it is able to reduce the size by reordering fields.

So the nomicon is out of date?

It says Struct fields are aligned to to the 32/64bit boundary based on system architecture. Then you have #[repr(u8/u16/u32/u64)] which will align to 8/16/32/64 boundary.

Re: The Lost Art of C Structure Packing (2014)

#102

Post by maccard »

kazinator wrote: post #90
superuser2 wrote: post #68

Earlier quoted context omitted.

Structs get written directly to disk and network. Copies of the same program would produce different output depending on the optimization strategy they were compiled with. That's vigorously not allowed.

Also, the translation unit compilation model of C would be troublesome here, if optimization settings could influence struct layout. You can't have different translation units interpreting the same struct type declaration differently.

> You can't have different translation units interpreting the same struct type declaration differently.

Sure you can!

    // MyHeader.h  
    struct MyStruct {  
    int A;  
    char b;  
    int* c;  
    };

    // A.cpp  
    #include "MyHeader.h"

    

    // B.cpp  
    #pragma pack(push, 16)  
    #include "MyHeader.h"  
    #pragma pack(pop)

Re: The Lost Art of C Structure Packing (2014)

#103

Post by jstelly »

This can also be important for routines that treat structures as strings of bytes. e.g. void MD5( unsigned char *pStructure, size_t size );

struct foo { char a; int x; };

if I MD5( &foo, sizeof(foo) ); on two foo with the same a & x, they may not produce the same MD5 because the pad bytes might contain noise from the stack or heap.

The solution is to either zero memory on anything you will MD5 or explicitly declare all of the padding (easy if you have internalized the rules) so your code can handle it.

Re: The Lost Art of C Structure Packing (2014)

#104

Post by Gankro »

valarauca1 wrote: post #101
dbaupp wrote: post #87

Earlier quoted context omitted.

Rust doesn't ignore field size nor does it align to any particular boundary. E.g. struct Foo { x: u8 } is one byte and has alignment 1, and similarly struct Bar { x: u8, y: i16 } is 4 bytes and has alignment 2. The compiler doesn't give guarantees about structure layout so that it is able to reduce the size by reordering fields.

So the nomicon is out of date? It says Struct fields are aligned to to the 32/64bit boundary based on system architecture. Then you have #[repr(u8/u16/u32/u64)] which will align to 8/16/32/64 boundary.

No you're just making stuff up.

https://doc.rust-lang.org/nomicon/repr-rust.html

The bulk of this section is describing how Rust has reserved the right to reorder and pack stuff. The first example happens to be 32-bit aligned because it contains a value that is.

Re: The Lost Art of C Structure Packing (2014)

#105

Post by Gankro »

Crespyl wrote: post #48
ThatGeoGuy wrote: post #40

Although I somewhat tire of constant comparisons between Rust and C, in this case it's interesting so I'll ask: Does Rust do struct packing any differently than C and what kinds of tradeoffs are associated with that? Most people don't even think about struct packing in the context of C++ (which is in many ways closer to Rust) because of vtables / inheritance / etc, but in this case I'd like to know if Rust does anyth…

IIRC, Rust struct layout is undefined/unstable, unless you explicitly use the #repr(C) tag on the struct definition, in which case it behaves pretty much exactly like C. You can also specify that the struct be packed, with no padding between items if necessary. I'm not sure if unmarked Rust structs actually get layout optimized or not right now, but they've worked to keep that option available.

Unless someone's recenrly bothered to implement it, I believe Rust doesn't currently bother to optimize your struct layout. It can and it surely will but not yet (this is a dangerous state of affairs since people may start to rely on it).

Re: The Lost Art of C Structure Packing (2014)

#106

Post by [deleted] »

blaisio wrote: post #57

Is this really a lost art? I feel like this is one of the main reasons people who use C still use C: you have a lot more control over how memory is handled. Honestly, if someone doesn't know about this, then they really just don't know C, because it's a pretty fundamental piece of information.

[deleted]

Re: The Lost Art of C Structure Packing (2014)

#107

Post by justinlardinois »

mikestew wrote: post #94
Sacho wrote: post #84

Earlier quoted context omitted.

Here are ESR's words on the subject: http://esr.ibiblio.org/?p=6907

However, ignore the way it's worded and the overall message is good advice whether there's a conspiracy or not (which I agree, there is likely not). If you're a male whose name I'd likely recognize, the only woman with whom you should ever be in a room alone is your SO. Someone might still accuse you of sexual assault, but it'll be pretty darned hard to prove if it can never be demonstrated that the two of you were a…

I really don't see how that's good advice. It's paranoia, and paranoia about something (being falsely accused of sexual assault) that very rarely happens.

It also strikes me as sexist. Every woman is a potential threat to you?

Re: The Lost Art of C Structure Packing (2014)

#108

Post by minipci1321 »

jcranmer wrote: post #24
minipci1321 wrote: post #5

Sigh... apparently this is now enough of rocket science to be considered for posting on HN... People using packing pragma of GCC should also beware -- an access to a field of a packed structure will be done bytewise, whether a variable happens to be actually aligned or not (I guess the compiler simplified its life by assuming no variable is ever aligned in packet structs), so memory size would go down but CPU use mig…

On x86, most memory can be accessed unaligned (and, unless you cross a cache line boundary, there's effectively no penalty. If you're on Haswell or newer, even if you cross a cache line boundary, there's no latency hit for an L1 cache hit). The code for a struct { char a; double b; } load reference on x86-64 after -O3 is: movsd 1(%rdi), %xmm0 So it looks like gcc is able to condense the load on platforms with unalign…

> So it looks like gcc is able to condense the load on platforms with unaligned accesses.

It seems there are still some leftovers -- try: struct { int i:31; }; with and without __attribute__((__packed__)).

Re: The Lost Art of C Structure Packing (2014)

#109

Post by Too »

Animats wrote: post #72

The bit-field feature of C structs is underutilized. People are still writing hex constants and using AND and OR to clear and set bits. Let the compiler do that; it's more readable. As of C99, there's named structure initialization, which makes bitfield constants more readable.

If your struct is mapped to a register then writing two bit fields in sequence will trigger two separate writes on the hardware register when you intended to write both the bits in one shot.

Re: The Lost Art of C Structure Packing (2014)

#110

Post by Too »

In chapter 4, padding outside structure. Isn't the compiler free to do whatever it wants there? There might not even be a need to reserve memory for some variables if the optimizer concludes they are irrelevant for the program?
Post reply on HN