Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Undefined behavior in C is a reading error

yodaiken.com

Post reply on HN
Print view View original

281–290 of 503 posts

Re: Undefined behavior in C is a reading error

#281

Post by PeterisP »

tsimionescu wrote: post #206
layer8 wrote: post #149

Because there seems to be some confusion in this thread: - "Implementation-defined behavior" means that the C standard specifies the allowable behaviors that a C implementation must choose from, and the implementation must document its particular choice. - "Unspecified behavior" means that the C standard places no particular restrictions on the behavior, but a C implementation must pick a behavior and document its ch…

I would note that the article is explicitly contesting the definition of UB that you are giving here (though you are absolutely right that this is the de facto definition used by all major compilers, and the commtitee). Basically the article is arguing that UB should be similar to Unspecified behavior - behavior that the implementation leaves up to the hardware and/or OS. I'm not sure where I fall to this issue, thou…

For me, the canonical UB example is a buffer overflow. No matter how you define UB, in practice a buffer overflow can result in for example, a system crash or - given appropriate very specific input data - encrypting all the files on your hard drive for a ransom.

Requiring compilers to restrict UB to something similar to unspecified behavior (where the behavior is not specified by the standard, but a C implementation must pick a behavior and document its choice) would require C compilers to prevent arbitrary code execution given a buffer overflow in C code, i.e. ensure that a buffer overflow in C code does not ever result in actual buffer overflow on the physical machine. This seems implausible to achieve - given the tradeoffs chosen for C, once UB hits, it really is undefined as it can (not in all, but in some) scenarios result in executing absolutely arbitrary machine code depending on the data provided to the program.

On the other hand, if you'd just want to say that the compiler can't assume that UB won't happen for optimizations, then many optimizations are impossible because theoretical UB is literally everywhere where basic arithmetic happens because of the possibility of an integer overflow - if UB would be required to do "whatever the hardware does" then it means that you can't even reorder basic chains of arithmetic instructions and have to execute every operation in order as-written to have the appropriate behavior in case of overflow.

Re: Undefined behavior in C is a reading error

#282

Post by btilly »

Jweb_Guru wrote: post #212
btilly wrote: post #153

Earlier quoted context omitted.

The author suggests that the text following the definition of "undefined behavior", listing the permitted or possible range of undefined behavior, should be read to restrict the consequences. But the first possibility listed is "ignoring the situation completely with unpredictable results". Surely that covers any possible consequences. Absolutely not. In the C89 standard, undefined behavior becomes undefined *UPON US…

This is actually desired though, at least by some programs. For example, say you have a function with a very expensive loop that repeatedly performs a null check and then executes some extra code if it's null, but never sets the value. This is called from another function which uses the checked value without a null check (proving it's not null) before and after the loop ends. The first function is inlined. You want t…

There may be programs that desire such behavior. But I've never intentionally written one. Which is why I personally avoid C, and wish that I didn't have to work in environments coded in C.

I seriously would accept everything running at half speed for the certainty of not being subject to the problems of C level bugs. But as Rust grows in popularity, it looks like I won't need to worry about that.

Re: Undefined behavior in C is a reading error

#283

Post by minitech »

vyodaiken wrote: post #150
anarazel wrote: post #76

First: I do dislike how hard it is to avoid some UB / how impractical some of the rules are. But I also think a lot of discussions of this topic caricaturize compiler writers to a ridiculous degree. Almost describing them to write optimization passes looking for UB so they can over-optimize something, while cackling loudly in glee about all the programs they can break. The set of people doing so overlaps with the set…

The current situation is not good for compiler writers either. But nobody has ever shown that either C programmers want to sacrifice safety for "optimizations", or that these UB optimizations actually improve performance of anything.

> But nobody has ever shown … that these UB optimizations actually improve performance of anything.

That’s just not true. Already examples in this thread: https://news.ycombinator.com/item?id=27223870

Re: Undefined behavior in C is a reading error

#284

Post by olliej »

MauranKilom wrote: post #258
olliej wrote: post #107

I dislike how UB is used where unspecified would work. For instance overflowing arithmetic is well specified on every architecture - the behaviour may be different on say sparc vs an hc12 vs x86, but on any of those architectures it will always be the same. Yet compiler devs have instead decided that it is undefined and so can be treated however they like. There are so many of these UB that could be unspecified that…

The C++ committee recently (for the C++20 standard) voted against making signed overflow defined, opting to keep it undefined. Primarily for performance reasons (and because it would usually be a bug anyway). www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r4.html#r0r1 You're of course free to disagree with the reasoning. But you'd probably agree that a new standard revision that forces the average application…

I’m not willing to accept a “say” for performance here.

I’d also want performance impact on a non-benchmark perf sensitive code base (browser, database, etc rather than spec).

The problem with the claim that the code is incorrect is that it is only* incorrect due to the imo bogus interpretation of what should be UB.

If I write

    If (a+b
On any modern architecture that has a well defined and understood meaning. But compilers have repeatedly overridden I well understood behavior in the name of perf benefits that only turn up in Spec-style benchmarks

Re: Undefined behavior in C is a reading error

#285

Post by bigcheesegs »

layer8 wrote: post #275
layer8 wrote: post #149

Because there seems to be some confusion in this thread: - "Implementation-defined behavior" means that the C standard specifies the allowable behaviors that a C implementation must choose from, and the implementation must document its particular choice. - "Unspecified behavior" means that the C standard places no particular restrictions on the behavior, but a C implementation must pick a behavior and document its ch…

I noticed that I mixed up "implementation-defined behavior" and "unspecified behavior" a bit. Here are the actual definitions: implementation-defined behavior : unspecified behavior where each implementation documents how the choice is made unspecified behavior : use of an unspecified value, or other behavior where this International Standard provides two or more possibilities and imposes no further requirements on w…

Yeah, I was about to post about this. Note that implementation-defined behavior and unspecified behavior are not _only_ used in places where a choice is given. Quite a few uses of implementation-defined behavior do not actually select between options. It just says the behavior is implementation-defined. So I suppose it selects between infinite options.

For this kind of behavior there is actually no limit on what the implementation is allowed to do, including assuming it never happens and optimizing accordingly. Implementations just need to document what they do.

I don't think these attempts to change the definition of UB are useful. As a compiler vendor it doesn't help to just say "it has a behavior", because that doesn't stop me from doing exactly what I do today. If people want some specific behavior, or to limit behaviors, then they need to actually say that in the spec.

To take left shift for example. Instead of saying it's implementation-defined behavior, they should say that it produces a implementation-defined non-trap value in the range of the resulting type that is consistent for the same inputs.

This would be pretty short to write in standardees, doesn't allow UB based optimizations, and allows the required implementation divergence.

Re: Undefined behavior in C is a reading error

#286

Post by barrkel »

PeterisP wrote: post #281
tsimionescu wrote: post #206

Earlier quoted context omitted.

I would note that the article is explicitly contesting the definition of UB that you are giving here (though you are absolutely right that this is the de facto definition used by all major compilers, and the commtitee). Basically the article is arguing that UB should be similar to Unspecified behavior - behavior that the implementation leaves up to the hardware and/or OS. I'm not sure where I fall to this issue, thou…

For me, the canonical UB example is a buffer overflow. No matter how you define UB, in practice a buffer overflow can result in for example, a system crash or - given appropriate very specific input data - encrypting all the files on your hard drive for a ransom. Requiring compilers to restrict UB to something similar to unspecified behavior (where the behavior is not specified by the standard, but a C implementation…

The trouble here is porting the maximalist disaster of one scenario to another scenario simply because the same phrase is used to describe them.

Because a buffer overflow may lead to arbitrary control flow escape via ROP or other hijacking of the instruction pointer, it does not follow that e.g. signed overflow is similarly dangerous.

Signed overflow, if not guarded against, may enable buffer overflow, but the deeper irony is that detecting when signed overflow has already occurred is close to impossible if the compiler is also aware that the check implies that it has occurred.

Thus, code designed for safety, guarding against buffer overflows by detecting signed overflow, ends up not protecting against buffer overflow, simply because signed overflow is described using the same name as buffer overflow.

Re: Undefined behavior in C is a reading error

#287

Post by layer8 »

PeterisP wrote: post #281
tsimionescu wrote: post #206

Earlier quoted context omitted.

I would note that the article is explicitly contesting the definition of UB that you are giving here (though you are absolutely right that this is the de facto definition used by all major compilers, and the commtitee). Basically the article is arguing that UB should be similar to Unspecified behavior - behavior that the implementation leaves up to the hardware and/or OS. I'm not sure where I fall to this issue, thou…

For me, the canonical UB example is a buffer overflow. No matter how you define UB, in practice a buffer overflow can result in for example, a system crash or - given appropriate very specific input data - encrypting all the files on your hard drive for a ransom. Requiring compilers to restrict UB to something similar to unspecified behavior (where the behavior is not specified by the standard, but a C implementation…

It goes further than that. The debates around UB a more about whether the compiler can assume that there are no buffer overflows and perform optimizations based on that. For example, if you have a local variable `char buffer[16]`, and there is an access `buffer[i]`, should the compiler be allowed to derive that 0 = 16, why shouldn't it? But some argue that the compiler shouldn't, exactly because such automated formal reasoning can lead to arbitrarily wide reaching effects. The problem is that it's hard to see where a middle ground could be.

Re: Undefined behavior in C is a reading error

#288

Post by barrkel »

jcranmer wrote: post #120
rectang wrote: post #102

Earlier quoted context omitted.

> But I also think a lot of discussions of this topic caricaturize compiler writers to a ridiculous degree. The inflamed backlash should tell you just how damaging it is to impose silent failure on meticulously written, previously fine programs.

> meticulously written, previously fine programs With relatively few exceptions, if your program hits undefined behavior, then your program was already doing something pretty wrong to begin with. Signed overflow is a poignant example: in how many contexts is INT_MAX + 1 overflowing to INT_MIN actually sane semantics? Unless you're immediately attempting to check the result to see if it overflowed (which is extremely…

Triggering signed overflow and testing whether it has occurred was how I protected the Delphi RTL memory allocation routines from signed overflow attacks, attacks which if not prevented, lead fairly directly to buffer overruns.

The code was written in Pascal and assembly, though, so it was safe from a C compiler.

Re: Undefined behavior in C is a reading error

#289

Post by PeterisP »

Animats wrote: post #221

Most of this discussion revolves around integer overflow. Part of the problem is that most of the computer hardware is now twos-complement arithmetic. Programmers think of that as part of the language. It's not, for C. C has run, in the past, on - 36 bit ones complement machines (DEC and UNIVAC) - Machines with 7-bit "char" (DEC) - Machines with 9-bit "char" (UNIVAC, DEC) - Machines where integer overflow yields a pr…

Explicit bounds everywhere mean that those bounds need to be automatically checked every time instead of only where explicitly specified by the programmer. This leads to safer code at a nontrivial performance tradeoff, one which would not be acceptable for C.

Re: Undefined behavior in C is a reading error

#290

Post by bigcheesegs »

dooglius wrote: post #122
anarazel wrote: post #76

First: I do dislike how hard it is to avoid some UB / how impractical some of the rules are. But I also think a lot of discussions of this topic caricaturize compiler writers to a ridiculous degree. Almost describing them to write optimization passes looking for UB so they can over-optimize something, while cackling loudly in glee about all the programs they can break. The set of people doing so overlaps with the set…

The caricatures are somewhat accurate though, optimizations that look at UB adversarially are never anywhere close to justified. > The set of people doing so overlaps with the set of people complaining that the compiler doesn't optimize their code sufficiently to a significant degree. There's no contradiction here, and the overlap is generally just "people who care". The optimizations that are not safe shouldn't exis…

Compiler dev here. Do you really think we just come up with code transformations and add them to the compiler just because?

All code transformations have a compile time cost and runtime perf impact. We don't add transformations unless the runtime perf impact greatly outweighs the compile time cost.

These optimizations are added because they measurably improve the performance of real code. This comes up in every review for new or updated optimization passes. This claim that they aren't justified is actually rather insulting to the effort put in to improve perf without taking days to compile.

Post reply on HN