Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clang emits memcpy for std::swap, which can introduce undefined behavior

llvm.org

Post reply on HN
Print view View original

51–60 of 83 posts

Re: Clang emits memcpy for std::swap, which can introduce undefined behavior

#51

Post by xorblurb »

chandlerc1024 wrote: post #34
haberman wrote: post #10

Given how aggressively compilers (including Clang) are exploiting undefined behavior these days, it is surprising to see Clang developers (I presume) on the bug being so blasé about introducing undefined behavior themselves.

Hi, LLVM and Clang developer here. I don't see anyone being blasé about this. The issue comes down to something really simple: - The compiler can emit substantially simpler code if it has a guarantee from the platform's implementation of memcpy. - The compiler authors thought they had such a guarantee and so chose to leverage it. Now, maybe they don't have that guarantee on all platforms. If that's the case, its flat…

Can't you see the pattern here? Even LLVM devs are incapable of doing what they are requiring from random regular developers. If that does not motivate to address that kind of craziness more in depth in a systematic way to restore safety and defense in depth, I don't know what will.

- Programmers can and do emit substantially simpler code if they have a guarantee from the platform's implementation of .

- Programmers think they have such a guarantee. (I don't know a single programmer around me who knows the complete list of undefined behavior, myself included)

- Compiler injects bug -- and LLVM/CLang seek excuse in the "technically not a bug" completely useless answer.

- Sane projects actually disable all that kind of crazyness (adding -fno- options every now and then) so all that kind of cute optims are useless in the real world (either dangerous, or completely disabled)

It's actually reasonable to have a platform that e.g. gives you 2's complement arithmetic or defines the result of shifts, or allows unaligned access, because mainstream computer hardware does provide those guarantee. And at least half of C programmers are not even aware those kind of stuff are technically forbidden, (historically often because of other architectures, some obscure, and not compiler optimizations, btw). More knowledgeable people who know about the spirit of the standard (1) can also be bitten by compiler revelation of bugs because compilers devs are doing literal and insane interpretation. Bug ensues, including security holes.

(1): Let me remind you about the definition of undefined behavior: "behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements" One keyword is nonportable and it is clearly not considered enough. So yes, there are numerous constructs that are reasonable to expect on a given architecture when we do not learn the compiler manual by heart, but some people don't give a fuck about the principle of least astonishment when they can gain 1% on a synthetic benchmark.

Re: Clang emits memcpy for std::swap, which can introduce undefined behavior

#52

Post by xorblurb »

chandlerc1024 wrote: post #35
electrograv wrote: post #28

Earlier quoted context omitted.

> it is surprising to see Clang developers (I presume) on the bug being so blasé about introducing undefined behavior themselves. I have suddenly lost a great deal of respect and confidence for LLVM/Clang after reading this bug thread; I value software reliability above all else, and being able to write reliable software without the compiler negating that effort. It seems I can no longer be assured this is even a pri…

Understand that some of the folks who don't think this is a bug own one of the libc implementations being used . So for their platform, it really is 100% correct and valid. The folks working on other platforms seem universally unhappy with that bug.

It is not the particular situation people are upset about (well, at least me). It is the reasoning, especially in the context of "you shall not write undefined behavior according to the standard even if it should makes sens given the computer you are actually programming" that is usually sent on us regular programmer by compiler authors... :/

Yet they do exactly the same thing, and pretty much casually. And I actually don't really blame them for doing it. Everybody makes mistakes. But I blame them for the double standard, the risks, and the attitude. The problem in the first place: engineering is also about mitigating the fact that mistakes will happen, often mitigating even the mistake of others, and the modern approach in regard with undefined behavior does not help and is even ultra-hurtful for that. I don't see how stability will emerge with that kind of thought process.

Re: Clang emits memcpy for std::swap, which can introduce undefined behavior

#53

Post by chandlerc1024 »

Dylan16807 wrote: post #50
chandlerc1024 wrote: post #44

Earlier quoted context omitted.

I want to assure you that the compiler does not exclusively use the ISO language standard to define the behavior of internal components of the implementation.

I'm talking about a specific component named memcpy. Obviously this does not apply to other components. And even that is only if the bug report is correct.

The compiler is not producing C code that calls memcpy. It is leveraging a specific implementation of a memcpy symbol in a particular system library. The contract between Clang and that system library can include guarantees not provided by any ISO standard.

Re: Clang emits memcpy for std::swap, which can introduce undefined behavior

#54

Post by chandlerc1024 »

haberman wrote: post #48
chandlerc1024 wrote: post #34

Earlier quoted context omitted.

Hi, LLVM and Clang developer here. I don't see anyone being blasé about this. The issue comes down to something really simple: - The compiler can emit substantially simpler code if it has a guarantee from the platform's implementation of memcpy. - The compiler authors thought they had such a guarantee and so chose to leverage it. Now, maybe they don't have that guarantee on all platforms. If that's the case, its flat…

Hi Chandler, I should have clarified that the comments I was referring to were on this bug: https://llvm.org/bugs/show_bug.cgi?id=11763 I was surprised to see: "I don't think it's really worth warning about, considering I don't know of any real-world implementation where it doesn't work." That sounds a lot like someone knowingly writing UB in a C program because no compiler they know of breaks the intended behavior.…

That comment was actually a considered comment: when we look at what warnings we should produce, we consider cases where everyone has written code a particular way and the fact that it isn't defined is actually a language bug.

We (the Clang community members, and I suspect the GCC community members as well) have also worked to change language standards to provide guarantees relied on consistently and where the lack of guarantee has no reasonable basis. Also that comment is very old, and predates us learning about all of why this is probably something actually worth warning on. That discussion ended up in a different forum, and we do actually warn on stuff like self assignment.

Understand that it is often precisely the people working on a compiler that do deal with the cases of when we discover widespread UB in real code and whether or not the code is incorrect and must be changed or the standard is incorrect and must be fixed.

But all of that is about a warning, and not about the actual question of how the compiler and the system library interact.

Note that both Clang and GCC have at least some variants of this bug, and so I don't actually expect glibc to break this in the foreseeable future. But we're still actively working to defend against this on platforms for which we don't have a guarantee.

Re: Clang emits memcpy for std::swap, which can introduce undefined behavior

#55

Post by chandlerc1024 »

xorblurb wrote: post #52
chandlerc1024 wrote: post #35

Earlier quoted context omitted.

Understand that some of the folks who don't think this is a bug own one of the libc implementations being used . So for their platform, it really is 100% correct and valid. The folks working on other platforms seem universally unhappy with that bug.

It is not the particular situation people are upset about (well, at least me). It is the reasoning, especially in the context of "you shall not write undefined behavior according to the standard even if it should makes sens given the computer you are actually programming" that is usually sent on us regular programmer by compiler authors... :/ Yet they do exactly the same thing, and pretty much casually. And I actuall…

You are tarring all compiler authors with a very broad brush here.

Myself and other compiler authors I work with are much more specific in their requests. We do actually define some behavior that the standard doesn't require, and we do document this and tell our users its fine to rely on it.

Unfortunately, we also have a lot of really clear experience with this stuff that lead us to two conclusions:

1) When we identify UB in code and go look at that code, for very large bodies of code what we find are overwhelmingly bugs. Bad, complete bugs. Defining the behavior wouldn't fix the bug, it was just one of the many symptoms.

2) We have real hardware differences for a lot of simple things that make them hard or impossible to give defined behavior for. As an example, shifting left by more bits than the size of the value gives different results on several different platforms.

None of this means that we don't need to help programmers cope with this. We do indeed also make mistakes, and in fact it was these compiler authors that built a tool to help programmers find UB in their code: http://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html

Re: Clang emits memcpy for std::swap, which can introduce undefined behavior

#56

Post by Dylan16807 »

chandlerc1024 wrote: post #53
Dylan16807 wrote: post #50

Earlier quoted context omitted.

I'm talking about a specific component named memcpy. Obviously this does not apply to other components. And even that is only if the bug report is correct.

The compiler is not producing C code that calls memcpy. It is leveraging a specific implementation of a memcpy symbol in a particular system library. The contract between Clang and that system library can include guarantees not provided by any ISO standard.

Then the bug report is wrong, and disregard the second and third posts I made. They depend on the bug report not being wrong.

My first post is generic and not about this particular issue; it's still fine.

Re: Clang emits memcpy for std::swap, which can introduce undefined behavior

#57

Post by whitequark_ »

Dylan16807 wrote: post #56
chandlerc1024 wrote: post #53

Earlier quoted context omitted.

The compiler is not producing C code that calls memcpy. It is leveraging a specific implementation of a memcpy symbol in a particular system library. The contract between Clang and that system library can include guarantees not provided by any ISO standard.

Then the bug report is wrong, and disregard the second and third posts I made. They depend on the bug report not being wrong. My first post is generic and not about this particular issue; it's still fine.

The bug report is not wrong. Rather, whether the current behavior is correct depends on the platform.

For example, Darwin's libc offers this contract for memcpy and clang is perfectly within its rights to generate such code (note that the IR it generates is still violating LLVM's contract on the memcpy intrinsic); glibc offers no such contract for memcpy, and so clang's code is nonconformant.

Re: Clang emits memcpy for std::swap, which can introduce undefined behavior

#58

Post by haberman »

chandlerc1024 wrote: post #54
haberman wrote: post #48

Earlier quoted context omitted.

Hi Chandler, I should have clarified that the comments I was referring to were on this bug: https://llvm.org/bugs/show_bug.cgi?id=11763 I was surprised to see: "I don't think it's really worth warning about, considering I don't know of any real-world implementation where it doesn't work." That sounds a lot like someone knowingly writing UB in a C program because no compiler they know of breaks the intended behavior.…

That comment was actually a considered comment: when we look at what warnings we should produce, we consider cases where everyone has written code a particular way and the fact that it isn't defined is actually a language bug. We (the Clang community members, and I suspect the GCC community members as well) have also worked to change language standards to provide guarantees relied on consistently and where the lack o…

I can't see an argument here for why the standard should be changed. memcpy() does one thing and memmove() does another. Changing the standard so they are the same just because memcpy() gets misused in real-world programs doesn't make sense to me. If people misuse memcpy(), then have a compiler flag akin to -fstrict-aliasing where memcpy() gets rewritten to memmove() unless people compile with -fstrict-memcpy.

That seems a separate issue though from what calls the compiler should emit. I am not following why LLVM would emit memcpy() when the semantics it actually requires from libc are memmove(). Am I missing something?

Re: Clang emits memcpy for std::swap, which can introduce undefined behavior

#59

Post by Joky »

haberman wrote: post #58
chandlerc1024 wrote: post #54

Earlier quoted context omitted.

That comment was actually a considered comment: when we look at what warnings we should produce, we consider cases where everyone has written code a particular way and the fact that it isn't defined is actually a language bug. We (the Clang community members, and I suspect the GCC community members as well) have also worked to change language standards to provide guarantees relied on consistently and where the lack o…

I can't see an argument here for why the standard should be changed. memcpy() does one thing and memmove() does another. Changing the standard so they are the same just because memcpy() gets misused in real-world programs doesn't make sense to me. If people misuse memcpy(), then have a compiler flag akin to -fstrict-aliasing where memcpy() gets rewritten to memmove() unless people compile with -fstrict-memcpy. That s…

It is not about making memcpy behave the same as memmove: in this case the difference would still be between exact identity (src == dest) and partial overlap.

Re: Clang emits memcpy for std::swap, which can introduce undefined behavior

#60

Post by colin_mccabe »

chandlerc1024 wrote: post #54
haberman wrote: post #48

Earlier quoted context omitted.

Hi Chandler, I should have clarified that the comments I was referring to were on this bug: https://llvm.org/bugs/show_bug.cgi?id=11763 I was surprised to see: "I don't think it's really worth warning about, considering I don't know of any real-world implementation where it doesn't work." That sounds a lot like someone knowingly writing UB in a C program because no compiler they know of breaks the intended behavior.…

That comment was actually a considered comment: when we look at what warnings we should produce, we consider cases where everyone has written code a particular way and the fact that it isn't defined is actually a language bug. We (the Clang community members, and I suspect the GCC community members as well) have also worked to change language standards to provide guarantees relied on consistently and where the lack o…

LLVM is a very impressive compiler. I'm really impressed with the warning messages it's able to produce (especially compared with what was available even 2 or 3 years ago).

What is more frustrating about modern compilers (and this is not limited to LLVM) is the desire to squeeze out a tiny bit more performance by exploiting undefined behavior. I'm thinking of things like eliminating calls to memset that "don't seem to do anything user-visible," eliminating code based on signed overflow being undefined, and so on.

Most of the the undefined behavior that exists in C is there because different machines used to do different things. For example, there used to be non-two's-complement machines. It wasn't put there because C was somehow designed to be hyper-optimized. People who relied on the undefined behavior could legitimately make the same argument that you are now making-- that it worked fine on their machine, which they knew very well.

I am curious why nobody has brought up the idea of making memmove as efficiently handled as memset, rather than using undefined behavior here. UB feels like the wrong way to go.

Post reply on HN