Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Lost Art of Structure Packing (2018)

catb.org

Post reply on HN
Print view View original

101–110 of 120 posts

Re: The Lost Art of Structure Packing (2018)

#101

Post by saagarjha »

icedchai wrote: post #94
saagarjha wrote: post #58

Earlier quoted context omitted.

sizeof will include padding, but there is no need for memset to actually perform a write that cannot be consistently observed.

Are you saying memset does not have to write the entire struct? I understand it can be optimized out, but thought it if it was actually executed it had to set all of the specified bytes.

memset must write to the bytes that underly the members of the struct. What it does to padding bytes is not of consequence to you, as you cannot observe it as far as I understand. (That is, if you tried to read it out it would not necessarily be the value you had memset, or even be consistent across reads.)

Re: The Lost Art of Structure Packing (2018)

#102

Post by saagarjha »

gpderetta wrote: post #97
saagarjha wrote: post #80

Earlier quoted context omitted.

As far as I understand, padding bits are indeterminate until you observe them via a memcpy into a buffer, at which point they will collapse (only in the buffer) to some arbitrary but now-constant value. Is this correct?

I think you are right in the general case. Definitely padding bits are not preserved during copies, but I think that at least in some cases they might be preserved, at least I think that if you memset+memcpy, the memcpy is guaranteed to see the zeros (what if you want to end the lifetime of the object and simply reuse the storage as an array of chars?). But there is divergence between C and C++ and also an area quite…

I looked it up for C, and as far as I can tell this has not standardized. (The Defect Report I read, which seems to have the latest opinion, literally calls such things "wobbly values". I'm not joking.) The TL;DR of Defect Report #451 seems to be that this is needs some work from the standard. In the case of an automatic struct or whatever (so you can't pull the bytes out from under it and reuse it) I think the consensus seems to be tending towards Heisenburg-ish values that have values that can change between reads, and certain library functions will be legal to call on them so that they "collapse" the state in the view that you've just created. For your case, I would have the question of whether two objects (e.g. a char array and a struct carved from that array) can share the same memory; in this case I would intuitively tend to agree with you that the padding bits are visible from the longer-lived object, kind of like an ad-hoc union. But I am not a licensed language lawyer, so I am not quite sure on that point.

I appreciate your edits, by the way ;)

Re: The Lost Art of Structure Packing (2018)

#103

Post by saagarjha »

rurban wrote: post #88
kazinator wrote: post #83

Earlier quoted context omitted.

memset_s is a pointless misinvention. I can't think of any situation in which I would use memset, and not want it to be what memset_s is claimed to be. Which is to say that memset_s should just be called memset; there is no need for memset to be doing stupid things so that people must use memset_s. I have no plans to use memset_s (ever), or to upstream any fix that involves using it unless the author provides a repro…

Agree. memset should be secure by default. That means it should not be optimized away by the compiler. And second the secure variant should flush the content, so that cache attacks can not read secrets which were memset'ed, but still in the cache on broken CPU's (Intel).

sloooooooooooooooow. No, really, do you want a full flush every time you do a memset? That would be horrendous for performance.

Re: The Lost Art of Structure Packing (2018)

#104

Post by icedchai »

saagarjha wrote: post #101
icedchai wrote: post #94

Earlier quoted context omitted.

Are you saying memset does not have to write the entire struct? I understand it can be optimized out, but thought it if it was actually executed it had to set all of the specified bytes.

memset must write to the bytes that underly the members of the struct. What it does to padding bytes is not of consequence to you, as you cannot observe it as far as I understand. (That is, if you tried to read it out it would not necessarily be the value you had memset, or even be consistent across reads.)

memset definitely writes to padding bytes, as noted by several other folks here. you observe these padding bytes during serialization and deserialization (disk, network, memory mapped buffer, etc.)

Note this is important enough that compiler folks consider it a bug if it doesn't work correctly. see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92486 for a recent example involving memset and memcpy.

Re: The Lost Art of Structure Packing (2018)

#105

Post by saagarjha »

kazinator wrote: post #90
saagarjha wrote: post #85

Earlier quoted context omitted.

Most of the Annex K * _s functions are pretty stupid, to be honest, but this is one that is actually useful. It's not that memset is doing stupid things; rather it's being smart and skipping work that it's not required to do.

memset has always been required to set N bytes starting at a given address to zero. That's the requirement. Been that way since it appeared in AT&T Unix and beat BSD's bzero to the ANSI C punch. There is no need for a broken memset that fails to set some of the bytes, so that a fixed one under a different name has to be used in its place. If you're using memset such that it's okay for memset not to set some of the by…

…and assigning to a variable is supposed to put some bytes at a certain address, yes. Except compilers will skip doing so if you never read from the variable again, and the same thing happens here: the compiler can "know" that it has no need to actually do the assignment, as writing to padding bytes is not required to actually do anything and trying to observe the value is currently the subject of debate but somewhere between "undefined" and "whatever you get out of it has nothing to do with what you think went in".

Re: The Lost Art of Structure Packing (2018)

#106

Post by saagarjha »

virgilp wrote: post #66
klodolph wrote: post #16

Earlier quoted context omitted.

Reorg is prohibited by C standard.

I know, it wasn't obvious to me that elteto talked exclusively about C/C++. AFAIK few -if any- compilers/VMs take it upon themselves to reorder struct members (even outside C/C++ world).

Both Swift and Rust have an unspecified structure ordering by default, which almost always boils down to "the compiler will lay them out in the order specified, except if you leave enough padding to fit a member, in which case it'll reorder the elements for you".

Re: The Lost Art of Structure Packing (2018)

#107

Post by saagarjha »

icedchai wrote: post #104
saagarjha wrote: post #101

Earlier quoted context omitted.

memset must write to the bytes that underly the members of the struct. What it does to padding bytes is not of consequence to you, as you cannot observe it as far as I understand. (That is, if you tried to read it out it would not necessarily be the value you had memset, or even be consistent across reads.)

memset definitely writes to padding bytes, as noted by several other folks here. you observe these padding bytes during serialization and deserialization (disk, network, memory mapped buffer, etc.) Note this is important enough that compiler folks consider it a bug if it doesn't work correctly. see https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92486 for a recent example involving memset and memcpy.

As mentioned in the bug report, the value of padding bytes is unspecified by the C standard. Actually, according to the linked Defect Report mentioned in the bug, there is a bit of discussion on whether the functions you mentioned should be entirely undefined to call or if some exceptions should be carved out. If the functions are legal to call, it is very likely that the results will be some arbitrary reification of the unspecified value.

Re: The Lost Art of Structure Packing (2018)

#108

Post by adrianN »

shawxe wrote: post #32
saagarjha wrote: post #27

Earlier quoted context omitted.

I don’t think memset is required to zero out padding bytes. The correct way to do this is to only access non-padding bytes.

sizeof(struct foo) is required to return the actual size, including padding, so struct foo foo_inst; memset(&foo_inst, 0, sizeof(foo_inst)); will certainly result in all padding bytes being set to zero.

Reading padding bytes is UB, so the compiler could choose to optimize that memset call so that the padding bytes aren't touched.

Re: The Lost Art of Structure Packing (2018)

#109

Post by Natsu »

SlowRobotAhead wrote: post #76
Natsu wrote: post #46

I'm sort of surprised there are no tools for this. I understand why having the compiler reorder things could be bad, though it seems like there should be room to tell the compiler it's okay to repack it for minimum space, but I don't even see any mention of a source-level tool that would just sort the items in a struct for you. It seems like something like that could be useful rather than making programmers try to or…

If you just order top down in structures from pointers, 32s, 16s, arrays, 8s, it’s almost entirely done without thinking. There is almost never a difference to the user what order things are structured. Although to be fair this does get tricky with unions of structure over structure.

True, but that's the kind of drudgery that's best farmed off to computers. I see that the comment above mentions there is a tool for this that I simply wasn't aware of.

Re: The Lost Art of Structure Packing (2018)

#110

Post by gpderetta »

saagarjha wrote: post #102
gpderetta wrote: post #97

Earlier quoted context omitted.

I think you are right in the general case. Definitely padding bits are not preserved during copies, but I think that at least in some cases they might be preserved, at least I think that if you memset+memcpy, the memcpy is guaranteed to see the zeros (what if you want to end the lifetime of the object and simply reuse the storage as an array of chars?). But there is divergence between C and C++ and also an area quite…

I looked it up for C, and as far as I can tell this has not standardized. (The Defect Report I read, which seems to have the latest opinion, literally calls such things "wobbly values". I'm not joking.) The TL;DR of Defect Report #451 seems to be that this is needs some work from the standard. In the case of an automatic struct or whatever (so you can't pull the bytes out from under it and reuse it) I think the conse…

Thanks for hunting down the details.

And it seem that I should double check which comment I'm editing before submitting :)

Post reply on HN