Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Lost Art of Structure Packing (2018)

catb.org

Post reply on HN
Print view View original

81–90 of 120 posts

Re: The Lost Art of Structure Packing (2018)

#81

Post by gpderetta »

klodolph wrote: post #16
virgilp wrote: post #7

Earlier quoted context omitted.

Pad - yes, typically. Reorg - seldom.

Reorg is prohibited by C standard.

It is complicated. Because of the as-if rule, a compiler can do anything it wants as long as a conforming program can't tell the difference. So for example if a compiler can prove that a program doesn't compare fiel addresses, doesn't cast pointer to a struct to its first member, etc it can reorder as it pleases. Turns out it is very hard to prove, often requires whole program optimization and the gains are questionable, so it is seldom done. Both clang and gcc had such an optimization pass in the past but it got dropped.

Re: The Lost Art of Structure Packing (2018)

#82

Post by banachtarski »

shawxe wrote: post #31
banachtarski wrote: post #29

Earlier quoted context omitted.

Wrong answer. This is brittle and relies on the memory always being initialized correctly and will be prone to all sorts of issues in the wild. Better is just either templatize on the key type or store the size on the map.

To be clear, there are obviously different/better ways to handle this in C++. We're talking about K&R C here.

Right I mentioned "store size of the key on the map" as the second option for K&R/Ansi C.

Re: The Lost Art of Structure Packing (2018)

#83

Post by kazinator »

saagarjha wrote: post #56
kazinator wrote: post #38

Earlier quoted context omitted.

memset is absolutely required to zero out padding bytes. Padding is part of the structure size, and memset hast to zero out exactly as many bytes as it is told. memseting a structure to zero is commonly done in programs that send structure outside of the process (like passing it to communication or storage-related system calls) because the padding can leak sensitive information. That better work! If the size of a str…

memset is not required to do anything if the side effects are not observable; this is the entire reason why memset_s needed to be added (and also why __attribute__((packed)) is recommended for anything that is being sent directly over the wire, if this kind of construction cannot be avoided). Compilers often inline and unroll small, constant-size memsets anyways, and since it is not possible to observe a consistent p…

memset_s is a pointless misinvention. I can't think of any situation in which I would use memset, and not want it to be what memset_s is claimed to be.

Which is to say that memset_s should just be called memset; there is no need for memset to be doing stupid things so that people must use memset_s.

I have no plans to use memset_s (ever), or to upstream any fix that involves using it unless the author provides a repro test case, and a proof that the problem can't be fixed with compiler options that make the problem go away with memset.

In the worst imaginable scenario, I will #define memset memset_s everywhere (after the inclusion of of course, not before, and an #undef memset).

(The #undef memset may be enough, in fact, if the only problem is that memset is #define'd to some compiler built-in that doesn't properly implement classic C90 memset in all cases.)

Re: The Lost Art of Structure Packing (2018)

#84

Post by kazinator »

saagarjha wrote: post #59
kazinator wrote: post #40

Earlier quoted context omitted.

A compiler that optimizes away memset is not one that could be used for targeting anything that sits on a network.

That immediately rules out GCC and Clang…

I don't think so, unless we're mistaking memset for __builtin_memset.

Re: The Lost Art of Structure Packing (2018)

#85

Post by saagarjha »

kazinator wrote: post #83
saagarjha wrote: post #56

Earlier quoted context omitted.

memset is not required to do anything if the side effects are not observable; this is the entire reason why memset_s needed to be added (and also why __attribute__((packed)) is recommended for anything that is being sent directly over the wire, if this kind of construction cannot be avoided). Compilers often inline and unroll small, constant-size memsets anyways, and since it is not possible to observe a consistent p…

memset_s is a pointless misinvention. I can't think of any situation in which I would use memset, and not want it to be what memset_s is claimed to be. Which is to say that memset_s should just be called memset; there is no need for memset to be doing stupid things so that people must use memset_s. I have no plans to use memset_s (ever), or to upstream any fix that involves using it unless the author provides a repro…

Most of the Annex K * _s functions are pretty stupid, to be honest, but this is one that is actually useful. It's not that memset is doing stupid things; rather it's being smart and skipping work that it's not required to do.

Re: The Lost Art of Structure Packing (2018)

#86

Post by not2b »

kazinator wrote: post #22

I've documented how GCC does bitfield packing in the TXR reference manual. Or rather, the abstract algorithm used in the FFI to replicate it. https://www.nongnu.org/txr/txr-manpage.html#N-027D075C This is the result of empirical investigation. The description also covers allocation of non-bitfields (paragraph 3) and the padding of the structure (paragraph 9) which require few words. I felt that the bitfield handling…

You could have saved time and gone to the ABI spec.

https://itanium-cxx-abi.github.io/cxx-abi/abi.html

This was originally developed as a joint effort to make compilers ABI-compatible on Itanium, but it's also used (by GCC, clang, Intel's proprietary compiler and others) on x86-64.

An old Hacker News comment said that it's from Intel; it's not, it was a joint effort with lots of work from CodeSourcery and Red Hat folks.

Re: The Lost Art of Structure Packing (2018)

#87

Post by scatters »

kazinator wrote: post #84
saagarjha wrote: post #59

Earlier quoted context omitted.

That immediately rules out GCC and Clang…

I don't think so, unless we're mistaking memset for __builtin_memset.

GCC treats the two identically. See https://github.com/gcc-mirror/gcc/blob/229c0ef777161ec5adfb7...

Re: The Lost Art of Structure Packing (2018)

#88

Post by rurban »

kazinator wrote: post #83
saagarjha wrote: post #56

Earlier quoted context omitted.

memset is not required to do anything if the side effects are not observable; this is the entire reason why memset_s needed to be added (and also why __attribute__((packed)) is recommended for anything that is being sent directly over the wire, if this kind of construction cannot be avoided). Compilers often inline and unroll small, constant-size memsets anyways, and since it is not possible to observe a consistent p…

memset_s is a pointless misinvention. I can't think of any situation in which I would use memset, and not want it to be what memset_s is claimed to be. Which is to say that memset_s should just be called memset; there is no need for memset to be doing stupid things so that people must use memset_s. I have no plans to use memset_s (ever), or to upstream any fix that involves using it unless the author provides a repro…

Agree. memset should be secure by default. That means it should not be optimized away by the compiler. And second the secure variant should flush the content, so that cache attacks can not read secrets which were memset'ed, but still in the cache on broken CPU's (Intel).

Re: The Lost Art of Structure Packing (2018)

#89

Post by [deleted] »

saagarjha wrote: post #58
shawxe wrote: post #32

Earlier quoted context omitted.

sizeof(struct foo) is required to return the actual size, including padding, so struct foo foo_inst; memset(&foo_inst, 0, sizeof(foo_inst)); will certainly result in all padding bytes being set to zero.

sizeof will include padding, but there is no need for memset to actually perform a write that cannot be consistently observed.

[deleted]

Re: The Lost Art of Structure Packing (2018)

#90

Post by kazinator »

saagarjha wrote: post #85
kazinator wrote: post #83

Earlier quoted context omitted.

memset_s is a pointless misinvention. I can't think of any situation in which I would use memset, and not want it to be what memset_s is claimed to be. Which is to say that memset_s should just be called memset; there is no need for memset to be doing stupid things so that people must use memset_s. I have no plans to use memset_s (ever), or to upstream any fix that involves using it unless the author provides a repro…

Most of the Annex K * _s functions are pretty stupid, to be honest, but this is one that is actually useful. It's not that memset is doing stupid things; rather it's being smart and skipping work that it's not required to do.

memset has always been required to set N bytes starting at a given address to zero. That's the requirement. Been that way since it appeared in AT&T Unix and beat BSD's bzero to the ANSI C punch.

There is no need for a broken memset that fails to set some of the bytes, so that a fixed one under a different name has to be used in its place.

If you're using memset such that it's okay for memset not to set some of the bytes, and you'd like them not to be set if that makes things faster, then you shouldn't be using memset. You're using a hammer to drive a screw: wrong tool.

C has perfectly good initialization and assignment for structures.

End of story; I'm going to walk away pretend I never read this subthread, re-joining the hordes of C programmers using memset in the normal way, adding to the countless lines of code that do it that way and are never going to be changed.

This bullshit will be backpedaled out of the standard eventually, you just wait.

Post reply on HN