Undefined behavior in C is a reading error • Hacker News Board

Re: Undefined behavior in C is a reading error

#241

Post by vyodaiken » Thu, May 20, 2021, 9:48 PM UTC

aw1621107 wrote: post #169

vyodaiken wrote: post #146

Earlier quoted context omitted.

e.g. removing a check for for overflow is definitely NOT ignoring the behavior. Deleting write because it would be undefined behavior for a pointer to point at some location is also NOT ignoring the behavior. Ignoring the behavior is exactly what the rationale is describing when it says UB allows compilers to not detect certain kinds of errors. Returning a pointer is certainly a use. In any event, the prevailing inte…

> e.g. removing a check for for overflow is definitely NOT ignoring the behavior. Deleting write because it would be undefined behavior for a pointer to point at some location is also NOT ignoring the behavior. Depending on how you look at it, this is ignoring the behavior. For example, say you have this: int f(int a) { if (a + 1 You have 2 situations: 1. a + 1 overflows 2. a + 1 does not overflow Situation 1 contain…

More than slightly convoluted. The obvious intention is that the compiler ignores overflow and lets the processor architecture make the decision. Assuming that overflow doesn't happen is assuming something false. There's no excuse for that and it doesn't "optimize" anything.

Re: Undefined behavior in C is a reading error

#242

Post by MauranKilom » Thu, May 20, 2021, 9:49 PM UTC

gpderetta wrote: post #199

ynik wrote: post #163

Earlier quoted context omitted.

What do you mean by "these UB optimizations"? C is a low-level language; it's basically impossible for a compiler to reason about the code unless it makes certain assumptions. It needs to assume the code is not self-modifying to do pretty much any code-generation more intelligent than a macro assembler. It needs to assume the code isn't messing with the stack frames/return addresses in order to inline functions. It n…

Exactly. For example there were programs 30-40 years ago that relied on exact stack layouts. These days everybody would agree they are completely broken. The issue of course is that it is extremely hard to write programs that have no UB. It would be nice for compilers to have an option to automatically introduce assetions whenever they rely on some UB-derived axiom, basically as a sort of lightweight sanitizer. In fa…

> It would be nice for compilers to have an option to automatically introduce assetions whenever they rely on some UB-derived axiom

Modifying a value from a different thread without synchronization is UB. The compiler assumes this does not happen in order to e.g. move things into registers. Could you elaborate how (and how often) you would like to have this kind of UB-derived axiom ("this value remains the same from here to there") checked with assertions?

Re: Undefined behavior in C is a reading error

#243

Post by vyodaiken » Thu, May 20, 2021, 9:51 PM UTC

gpderetta wrote: post #199

ynik wrote: post #163

Earlier quoted context omitted.

What do you mean by "these UB optimizations"? C is a low-level language; it's basically impossible for a compiler to reason about the code unless it makes certain assumptions. It needs to assume the code is not self-modifying to do pretty much any code-generation more intelligent than a macro assembler. It needs to assume the code isn't messing with the stack frames/return addresses in order to inline functions. It n…

Exactly. For example there were programs 30-40 years ago that relied on exact stack layouts. These days everybody would agree they are completely broken. The issue of course is that it is extremely hard to write programs that have no UB. It would be nice for compilers to have an option to automatically introduce assetions whenever they rely on some UB-derived axiom, basically as a sort of lightweight sanitizer. In fa…

That's good example, because nobody would complain if stack layouts changed and those programs failed. But if the compiler chooses to "optimize away" checks on stack layout, that's a different thing altogether. Also note that if you use pthreads or Linux clone or you are writing an operating system you can need to rely on exact stack layouts even today.

Re: Undefined behavior in C is a reading error

#244

Post by pklausler » Thu, May 20, 2021, 9:53 PM UTC

bonzini wrote: post #192

pklausler wrote: post #157

Earlier quoted context omitted.

I think the compiler that you were using is broken. One can't infer "index use of "flag", and that inference can't override the predicates that dominate that use.

No, the initializer already counts as a use of the undefined expression.

You're right. Thanks!

Re: Undefined behavior in C is a reading error

#245

Post by Gaelan » Thu, May 20, 2021, 10:03 PM UTC

tsimionescu wrote: post #206

layer8 wrote: post #149

Because there seems to be some confusion in this thread: - "Implementation-defined behavior" means that the C standard specifies the allowable behaviors that a C implementation must choose from, and the implementation must document its particular choice. - "Unspecified behavior" means that the C standard places no particular restrictions on the behavior, but a C implementation must pick a behavior and document its ch…

I would note that the article is explicitly contesting the definition of UB that you are giving here (though you are absolutely right that this is the de facto definition used by all major compilers, and the commtitee). Basically the article is arguing that UB should be similar to Unspecified behavior - behavior that the implementation leaves up to the hardware and/or OS. I'm not sure where I fall to this issue, thou…

> and the commtitee

I'd argue that for a document like the C standard, if there's a well-known intended meaning, that is the meaning of the document - any other interpretation is purely academic.

Re: Undefined behavior in C is a reading error

#246

Post by klodolph » Thu, May 20, 2021, 10:08 PM UTC

MauranKilom wrote: post #229

klodolph wrote: post #60

I think one of the best attempts to solve this is the attempt to classify undefined behavior into "bounded" and "critical" UB, which is a distinction that hasn't gained as much traction as I'd like. Something like 1 Critical UB is stuff like writing to a const object, calling a function pointer after casting it to an incompatible type, dereferencing an invalid pointer, etc. Basically, anything goes. This is part of t…

> Something like 1 That is what implementation-defined behavior is for. Feel free to advocate for changing the standard accordingly.

> Feel free to advocate for changing the standard accordingly.

I am, in fact, describing the existing C standard, not advocating for changes. Please refer to Annex L "Analyzability", which defines the terms I used: "bounded undefined behavior" and "critical undefined behavior". Note that this section is conditional... it is not widely adopted. I am advocating for increased adoption of this part of the standard, or barring that, revisions that would make adoption more palatable.

And in general, "If you don't like it, advocate changing the standard" is not a useful or insightful response. It is completely reasonable and normal to complain about something without trying to fix it.

Re: Undefined behavior in C is a reading error

#247

Post by gpderetta » Thu, May 20, 2021, 10:15 PM UTC

MauranKilom wrote: post #242

gpderetta wrote: post #199

Earlier quoted context omitted.

Exactly. For example there were programs 30-40 years ago that relied on exact stack layouts. These days everybody would agree they are completely broken. The issue of course is that it is extremely hard to write programs that have no UB. It would be nice for compilers to have an option to automatically introduce assetions whenever they rely on some UB-derived axiom, basically as a sort of lightweight sanitizer. In fa…

> It would be nice for compilers to have an option to automatically introduce assetions whenever they rely on some UB-derived axiom Modifying a value from a different thread without synchronization is UB. The compiler assumes this does not happen in order to e.g. move things into registers. Could you elaborate how (and how often) you would like to have this kind of UB-derived axiom ("this value remains the same from…

Obviously you wouldn't be able to catch many, or even most cases. Use-after-free is another case that would be very expensive to detect.

Re: Undefined behavior in C is a reading error

#248

Post by _Nat_ » Thu, May 20, 2021, 10:18 PM UTC

fiter wrote: post #214

bregma wrote: post #211

Earlier quoted context omitted.

from ISO/IEC 9899:2011 "Programming Languages -- C" 3.4.3 1 undefined behavior behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements 2 NOTE Possible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner charac…

This quote is the topic of the original article and the article goes into detail about how it believes the quote should be interpreted.

The original article's interpretation seemed untenable.

While the difference between "Permissible" and "Possible" could be quite significant, in this case, it was qualifying:

> [Permissible/Possible] undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message).

The previously-"Permissible" behaviors were so broad that they basically allowed anything, including translating the source-code in any documented manner.. which basically means that, as long as a compiler says how it'll treat undefined-behavior, it can do it that way, because it's free to completely reinterpret the source-code in any (documented) manner.

Re: Undefined behavior in C is a reading error

#249

Post by nwallin » Thu, May 20, 2021, 10:20 PM UTC

layer8 wrote: post #149

Because there seems to be some confusion in this thread: - "Implementation-defined behavior" means that the C standard specifies the allowable behaviors that a C implementation must choose from, and the implementation must document its particular choice. - "Unspecified behavior" means that the C standard places no particular restrictions on the behavior, but a C implementation must pick a behavior and document its ch…

> - "Undefined behavior" means that C implementations are allowed to assume that the respective runtime condition does not ever occur, and for example can generate optimized code based on that assumption.

Please note that the article is making the specific argument that this interpretation of UB is an incorrect interpretation. The author is arguing that you, me, the llvm and gcc teams are wrong to interpret UB that way.

Linux had a bug in it a few years ago; the code would dereference a pointer, then check if it was null, then returned an error state if it was null, or continued performing the important part of the function. The compiler deduced that if the pointer had been null when it was dereferenced, that's UB, so the null check was unnecessary, and optimized the null check out. The trouble was that in that context, a null pointer dereference didn't trap, (because it was kernel code? not sure.) so the bug was not detected. It ended up being an exploitable security vulnerability in the kernel, I think a local privilege escalation.

The article is making the argument that the compiler should not be free to optimize out the null check before subsequent dereferences. The compiler is permitted to summon nasal demons where the pointer is dereferenced the first time, but should not be free to summon nasal demons at later lines of code, after the no-nasal-demons-please check.

(The linux kernel now uses -fno-delete-null-pointer-checks to ensure that doesn't happen again. The idea is that even though it was a bug that UB was invoked, the failure behavior should be safe instead of granting root privileges to an unprivileged user.)

Fun with NULL pointers part 1 https://lwn.net/Articles/342330/

Fun with NULL pointers part 2 https://lwn.net/Articles/342420/

Re: Undefined behavior in C is a reading error

#250

Post by steveklabnik » Thu, May 20, 2021, 10:24 PM UTC

bhk wrote: post #144

bhk wrote: post #112

Earlier quoted context omitted.

It isn't clear to me precisely what example you have in mind. If you are saying that deleting array bounds checks might have performance benefits that outweigh the security concerns, then I disagree. If you are saying that the compiler would have to insert bounds checks, I don't see how you arrive at that. I have seen claims that gratuitous UB is important for enabling meaningful optimizations, but in every such case…

For some reason I couldn't 'Reply' to compiler-guy's reply directly, so I'll try here. I'm familiar with the Chris Lattner article. Most of it (especially the second installment) shows bad outcomes from UB optimization. When it comes to signed integer overflow UB, I see two examples where performance is cited as a motivation. One mentions unrolling, and gives an example similar to one elsewhere in this thread: https:…

> For some reason I couldn't 'Reply' to compiler-guy's reply directly, so I'll try here.

Hacker News has a timeout where if you try to reply to someone too quickly, it will hide the reply button. This timeout increases the deeper the comment tree gets.