Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Memsetting Structures

anmolsarma.in

Post reply on HN
Print view View original

261–270 of 277 posts

Re: Stop Memsetting Structures

#261

Post by saagarjha »

olliej wrote: post #260
saagarjha wrote: post #195

Earlier quoted context omitted.

> You have to assume that data is malicious, and will attack any weakness. The point is that the standard cannot help you here because malicious data attacks things outside the scope of the standard.

The standard gives compilers leeway to handle “undefined” behaviour however they want. The original purpose of “undefined behaviour” was to deal with the myriad different platform behaviours of the era (remember at that point in history 1 byte was not necessary 8 bits). At some point the compiler writers decided to extend the effect of that to “if this behaviour is undefined in the spec” then we are allowed to treat…

Believe me, I know how compilers exploit undefined behavior and how this can lead to security issues. While you may disagree, there are certain useful optimizations that all but require the compiler assuming that undefined behavior cannot occur. Yes, occasionally compilers end up doing somewhat stupid and annoying things in the process or go "overboard" (Clang is notorious for this–I think some old version would just stop writing out code if it hit undefined behavior, so control would just run off the bottom of the procedure into the next one) but in general there are good reasons behind this choice being made.

> The original purpose of “undefined behaviour” was to deal with the myriad different platform behaviours of the era (remember at that point in history 1 byte was not necessary 8 bits).

FWIW, CHAR_BIT is implementation defined (not undefined) to be at least 8.

> Therefore the memcmp is undefined, and the compiler can assume undefined behavior cannot occur, therefore that branch can be removed.

I am fairly certain that this memcmp is "implementation-defined" (namely, reading the value of padding bytes is valid but not guaranteed to produce anything useful), which allows the compiler to pick whether it wants to emit that branch (but it must still return some value). If the memcmp was actually undefined the compiler would be "justified" in producing code that wiped your hard drive.

Re: Stop Memsetting Structures

#262

Post by nocman »

teej wrote: post #79
nocman wrote: post #44

Earlier quoted context omitted.

No, you are not the only one. The author of that code is trying to make it explicit -- "yes, I want to allow reuse of local addressses". Some will prefer this, some won't. I personally like the use of the variable, though I think it's not worth getting all bent out of shape about it one way or another. But I guess we all have our pet peeves, and I know I'm guilty of getting all bent out of shape about things that ult…

Code is read many times more than it is written. If someone scrutinizes code for clear intent, I don’t think that should be considered a “pet peeve” or some quirk to be explained away.

I wasn't suggesting that any scrutinizing of the code for clear intent should be considered a "pet peeve". I was saying I see both sides in this particular case, and that in my opinion devlopers (myself included) often make a way bigger deal out of things than they really are -- when the larger part of their offense toward a particular choice is often more about personal taste than it is anything else (thus the reference to pet peeves).

Again, I like the use of the variable, because in my opinion it makes the code a bit clearer, but I also don't think it is the end of the world if someone chooses the option without it.

Re: Stop Memsetting Structures

#263

Post by BeeOnRope »

paavoova wrote: post #80
loeg wrote: post #71

Earlier quoted context omitted.

The compiler knows the representation of NULL on the targeted platform and if it is safe to use memset. It is perfectly valid for the compiler to make this transformation if the platform represents NULL as zero.

My post is sarcasm, I just find it humorous that one's efforts to eliminate explicit memset doesn't stop the compiler from going one step backwards and embedding it just the same. If you replace the initialization with memset, it still generates identical code for both Clang and GCC.

I don't think the point was to eliminate "code that looks like memset at the assembly level", but to eliminate memset at the source level, which the author considers cleaner.

Under the covers, one would hope a good optimizing compiler could choose the same assembly for both, which may very well look "memset like" since fewer, wide writes are generally more efficient.

Re: Stop Memsetting Structures

#264

Post by NikkiA »

hazeii wrote: post #217
swiley wrote: post #199

Ugh, what do you do when the structure definition is updated and you now have uninitialized values? If there's no explicit initializer I think memset,calloc or ={0} is just fine thanks.

Unclear to me why you've been downvoted; I've seen many cases where a structure has had elements added which would be end up uninitialised. A memset(&struct,0,sizeof(struct)) seems pretty reasonable to me.

Probably because the article specifically addresses this issue and C99 specifically initializes any members who are not named in the initializer form to either zero or NULL.

Re: Stop Memsetting Structures

#265

Post by jcr1488 »

Dylan16807 wrote: post #198
boulos wrote: post #159

Earlier quoted context omitted.

Do you mean in the sense that a compiler can say "I know what memset is supposed to do" then decides that even though sizeof returns a particular size, that it might ignore that and set fewer bytes when it does a substitution for the memcpy call?

Yes. As long as the behavior observable by a conforming C program is the same, the compiler is allowed to change anything else. Even if memset in a vacuum is guaranteed, look at the quote by anyfoo. "When a value is stored in an object of structure or union type, including in a member object, the bytes of the object representation that correspond to any padding bytes take unspecified values." If the compiler knows yo…

As soon as a pointer to said memory is passed to an extern function in another translation unit, the compiler can't prove anything about how it's used, which is the case in pretty much all of the examples mentioned in this thread.

Also, type-punning is a thing. memset is byte-oriented/memory-oriented. Just because you're using it to zero a struct of a particular kind doesn't mean that's the only way the memory will accessed. Just because reading the padding of some struct is undefined behaviour doesn't mean accessing those bits by some other means is also undefined.

A compiler usually can't eliminate a call to memset() in most practical cases of initialization (where memory leakage is also a concern) because they almost always pass a reference to a routine in another translation unit. Something like memzero_explicit() can be used anyway -- but you're massively overstating the relevance of compilers eliminating dead stores done via memset(). It's much more of an issue for post-destruction memory sanitizing (which is the primary use case for memzero_explicit) than it is for compiler f*ckery when memset() is used for initialization.

Re: Stop Memsetting Structures

#266

Post by Dylan16807 »

jcr1488 wrote: post #265
Dylan16807 wrote: post #198

Earlier quoted context omitted.

Yes. As long as the behavior observable by a conforming C program is the same, the compiler is allowed to change anything else. Even if memset in a vacuum is guaranteed, look at the quote by anyfoo. "When a value is stored in an object of structure or union type, including in a member object, the bytes of the object representation that correspond to any padding bytes take unspecified values." If the compiler knows yo…

As soon as a pointer to said memory is passed to an extern function in another translation unit, the compiler can't prove anything about how it's used, which is the case in pretty much all of the examples mentioned in this thread. Also, type-punning is a thing. memset is byte-oriented/memory-oriented. Just because you're using it to zero a struct of a particular kind doesn't mean that's the only way the memory will a…

> As soon as a pointer to said memory is passed to an extern function in another translation unit, the compiler can't prove anything about how it's used, which is the case in pretty much all of the examples mentioned in this thread.

You would have to call such a function between the memset and the first time you write to a member. Otherwise the compiler is allowed to say "I put the padding back, and you can't prove otherwise".

> Just because reading the padding of some struct is undefined behaviour doesn't mean accessing those bits by some other means is also undefined.

It's not always undefined, but it says very clearly that the value of padding becomes unspecified.

> they almost always pass a reference to a routine in another translation unit

> you're massively overstating the relevance of compilers eliminating dead stores done via memset

Unless inlining happened, or link-time optimization, or, or...

If the compiler zeroes the memory most of the time, that makes it even scarier. Because all your tests come back clean and safe, then four years later a macro changes and suddenly you're leaking data all over the place.

I don't think I'm overstating the relevance at all. Any security feature that could disappear because of a reasonable, trying-to-help optimization is one that should have a bright red warning label. And this is such a feature. It doesn't require a "sufficiently smart" compiler, and it doesn't require a malicious compiler. This is the kind of thing that can break by accident and ruin everyone's month.

Re: Stop Memsetting Structures

#267

Post by nybble41 »

andrewshadura wrote: post #148
bjourne wrote: post #113

Earlier quoted context omitted.

I'm joking because endless flame wars have already been spent debating this issue. But I realize not everyone has heard them yet, so let's strap on the football shoes and bring out the horse carcass one more time! There are platforms on which NULL, integer 0 and floating point 0 is not the same. Hence memsetting won't do the right thing. On modern platforms they are, but still, you don't want to get the Standard C We…

By the standard, the null pointer must be equivalent to 0.

The memory representation of the null pointer is implementation-defined. The C99 standard only guarantees that a integer constant expression with the value 0, implicitly or explicitly cast to a pointer type, will give a null pointer. It does not guarantee that a null pointer cast to an integer type (other than _Bool) will give the value 0; all pointer-to-integer casts are implementation-defined. It also does not guarantee that a non-constant expression which evaluates to 0 will produce a null pointer if cast to a pointer type.

In short, there may exist counter-intuitive but nonetheless standards-compliant platforms where tests such as "assert((uintptr_t)(void * )0 == 0);" or "int x = 0; assert((void * )x == (void * )0);" would fail.

Re: Stop Memsetting Structures

#268

Post by nybble41 »

Sharlin wrote: post #43
ThrowawayR2 wrote: post #23

Earlier quoted context omitted.

> it's possible for the assertion to fail, since NULL it not guaranteed to be represented in memory by a zero bytes. Technically correct (which is of course the best kind of correct) but you'd be hard pressed to find a system in 2019 where NULL != (void*)0. The most recent machines with non-zero NULL in the C FAQ entry on the matter ( http://c-faq.com/null/machexamp.html ) date back to the mid '90s.

The literal `0` is guaranteed to be the null pointer value when used as a pointer, so by definition NULL == (void*)0. But given void *p = 0; intptr_t i = 0; it is not guaranteed that `memcmp(&i, &p, sizeof(p)) == 0`.

> it is not guaranteed that `memcmp(&i, &p, sizeof(p)) == 0`.

Even less intuitively, it is not guaranteed that `(void*)i == p` since `i` is not an integer constant expression, even if the value is known to be 0.

Re: Stop Memsetting Structures

#269

Post by jcr1488 »

Dylan16807 wrote: post #266
jcr1488 wrote: post #265

Earlier quoted context omitted.

As soon as a pointer to said memory is passed to an extern function in another translation unit, the compiler can't prove anything about how it's used, which is the case in pretty much all of the examples mentioned in this thread. Also, type-punning is a thing. memset is byte-oriented/memory-oriented. Just because you're using it to zero a struct of a particular kind doesn't mean that's the only way the memory will a…

> As soon as a pointer to said memory is passed to an extern function in another translation unit, the compiler can't prove anything about how it's used, which is the case in pretty much all of the examples mentioned in this thread. You would have to call such a function between the memset and the first time you write to a member. Otherwise the compiler is allowed to say "I put the padding back, and you can't prove o…

> This is the kind of thing that can break by accident and ruin everyone's month.

Not in practice. Compilers make use of undefined behaviour to optimize things that are widely applicable and profitable. No real compiler does what you're saying and no future compiler is likely to without explicitly being asked to.

I agree that, by the letter of the spec, you're right, but you're still most certainly overstating the relevance.

I'm not arguing that this isn't a real problem or that people shouldn't use memzero_explicit() (or similar) where security is on the line, as I already said several times in another sub-thread -- I'm just saying that this kind of thing is extreme language-lawyering beyond the realms of probability. It's still not an excuse to be lax, but let's be realistic about the actual likelihood of it happening.

Re: Stop Memsetting Structures

#270

Post by Matheus28 »

olliej wrote: post #260
saagarjha wrote: post #195

Earlier quoted context omitted.

> You have to assume that data is malicious, and will attack any weakness. The point is that the standard cannot help you here because malicious data attacks things outside the scope of the standard.

The standard gives compilers leeway to handle “undefined” behaviour however they want. The original purpose of “undefined behaviour” was to deal with the myriad different platform behaviours of the era (remember at that point in history 1 byte was not necessary 8 bits). At some point the compiler writers decided to extend the effect of that to “if this behaviour is undefined in the spec” then we are allowed to treat…

Undefined behavior != implementation defined.

Also, is it that much harder to do this (C++ for example):

    struct S {
        char c = 0;
        int i = 0;
        bool operator==(const S &other) const {
            return c == other.c && i == other.i;
        }
    };
    
    S a, b;
    if(something) return !(a == b);
    return 3;
Bam, no undefined behavior and you don't have to remember to manually zero out the structure everywhere.

If you're using memcmp on an object, you should be asserting that std::has_unique_object_representations is true for it (unfortunately this is only available after C++17).

Also, if you are okay with anonymous field names:

    std::tuple a, b;
    if(something) return a != b;
    return 3;
Post reply on HN