Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Into the Depths of C: Elaborating the De Facto Standards [pdf]

cl.cam.ac.uk

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Into the Depths of C: Elaborating the De Facto Standards [pdf]

#21

Post by pizlonator »

pcwalton wrote: post #19
Gibbon1 wrote: post #17

Earlier quoted context omitted.

If the compiler writers union decides that the spec allows for 'format hard drive off on signed overflow' we can always change the spec to 'don't format hard drive on signed overflow'

It's not that simple. The compiler writers have caused security issues before in the Linux kernel. And the compiler writers are right : the undefined behavior that they exploit exists for important performance-related reasons.

To say they are right is an overstatement, I think. There is insufficient discussion of the empirically measured performance benefits of specific forms of UB.

Some kinds of UB could be turned into something stricter, like reading a bad pointer. This either traps or returns some value. It won't format your hard drive unless you install a trap that formats your hard drive, but that's none of the spec's business. Traps can happen due to timers, so if arbitrary traps mean UB then every instruction is UB. Even if the spec punts on defining what a trap is, that's a progression over saying it's UB.

Same thing goes for division and modulo. In corner cases, it will either return some value or it will trap. It won't format your hard drive.

The most profitable "true" UB is stuff like TBAA, but smart people turn that off.

Do you know what the performance benefits are of other kinds of UB? Do you know how many of those perf benefits (like being maybe being able to take some shortcuts in SROA) can't be solved by changing the compiler (i.e. you'll get the same perf, but the compiler follows slightly different rules)? Maybe I'm not so well read, but I hardly ever hear of empirical evidence that proves the need for UB, only demos that show the existence of an optimisation in some compiler that would fail to kick in if the behaviour was defined.

Also, if there were perf benefits of the really gnarly kinds of UB, I would probably be happy to absorb the loss in most of the code I write. If I added up all of the time I've wasted fixing signed-unsigned comparison bugs and used that time to make WebKit faster, then I'd probably have made WebKit faster by a larger amount than the speed-up that WebKit gets from whatever corner-case optimisation the compiler can do by playing fast and loose with signed ints.

I suspect that UB is the way that it is because of politics - you can't get everyone to agree what will happen, nobody wants to lose some optimisation that they spent time writing, and so we punt on good semantics.

Re: Into the Depths of C: Elaborating the De Facto Standards [pdf]

#22

Post by pizlonator »

caf wrote: post #13
pcwalton wrote: post #10

Earlier quoted context omitted.

Undefined behavior in the form of memory safety issues is a problem in day-to-day work.

But that's not what this study is about: everyone agrees that stomping wildly off the end of an array in C is not going to end well. This is about far more subtle issues than that - issues where there is some disagreement about whether it's OK to do or not. And the GP is right - these are often not such a problem in practice, if only because these are the kinds of issues where experienced C programmers know that they…

Signed versus unsigned comparison, signed wrap-around, undefined division behaviour, undefined behaviour of some casts, and strict aliasing optimisations have all caused hard bugs that I've had to spend a lot of time fixing.

I had to deal with some of these before I was a compiler writer, and I would end up just kind of kicking my code repeatedly until stuff worked again.

Now that I'm a compiler writer, I know how to recognize what is happening, but I'm still not smart enough to avoid the bugs in general and I still spend time fixing bugs that result from these issues.

So, I'm with pcwalton: it is a problem. Maybe I don't see it every day, but I see it probably at least once a month.

Re: Into the Depths of C: Elaborating the De Facto Standards [pdf]

#23

Post by pjmlp »

TickleSteve wrote: post #8

I've been using C for over 20 years and I'm sure I would be caught out by these... ...but... The fact these curiosities are not an issue in day-to-day work and C is (one of) the most popular languages around today mean that they aren't too serious. When you have a knowledge of the hardware and are working at that level day-in day-out then issues like this really don't bother you that much. (I do realise that this is…

> The fact these curiosities are not an issue in day-to-day work and C is (one of) the most popular languages around today mean that they aren't too serious.

C became one of the most used languages, not necessary popular, thanks to the adoption of UNIX and the rise of FOSS/C culture (UNIX based) in the late 90's.

Back then it was just yet another systems programming language.

I only started to care about it when I moved from MS-DOS into Windows / UNIX, and even by then I was into C++ after a short (1 year) encounter with C.

As for not being an issue, the CVE list shows daily the cost of any programming language that "enjoys" copy-paste compatibility with C semantics.

Or the business opportunity for those that sell tools that help both developers (static analyzers) and users (anti-virus/firewalls) to overcome those shortcomings.

Re: Into the Depths of C: Elaborating the De Facto Standards [pdf]

#24

Post by MaulingMonkey »

pizlonator wrote: post #22
caf wrote: post #13

Earlier quoted context omitted.

But that's not what this study is about: everyone agrees that stomping wildly off the end of an array in C is not going to end well. This is about far more subtle issues than that - issues where there is some disagreement about whether it's OK to do or not. And the GP is right - these are often not such a problem in practice, if only because these are the kinds of issues where experienced C programmers know that they…

Signed versus unsigned comparison, signed wrap-around, undefined division behaviour, undefined behaviour of some casts, and strict aliasing optimisations have all caused hard bugs that I've had to spend a lot of time fixing. I had to deal with some of these before I was a compiler writer, and I would end up just kind of kicking my code repeatedly until stuff worked again. Now that I'm a compiler writer, I know how to…

There are fewer issues that scare me more than undefined behavior that usually works. "Works" enough for programmers to add it, knowingly or unknowingly, even to defend it as "working in practice"... only to leave a needle in a haystack that they'll never find.

Re: Into the Depths of C: Elaborating the De Facto Standards [pdf]

#25

Post by pavlov »

pjmlp wrote: post #23
TickleSteve wrote: post #8

I've been using C for over 20 years and I'm sure I would be caught out by these... ...but... The fact these curiosities are not an issue in day-to-day work and C is (one of) the most popular languages around today mean that they aren't too serious. When you have a knowledge of the hardware and are working at that level day-in day-out then issues like this really don't bother you that much. (I do realise that this is…

> The fact these curiosities are not an issue in day-to-day work and C is (one of) the most popular languages around today mean that they aren't too serious. C became one of the most used languages, not necessary popular, thanks to the adoption of UNIX and the rise of FOSS/C culture (UNIX based) in the late 90's. Back then it was just yet another systems programming language. I only started to care about it when I mo…

Late '90s is about a decade off, IMO. C was very much the only systems language by then.

I think the last time a popular OS was built on something else than C was the original Mac OS, which had a Pascal API and used Pascal calling conventions.

On the desktop, C was chosen as the API language for Windows and OS/2 around 1986. That meant both Microsoft and IBM agreed that PC software is going to be in written in C.

Re: Into the Depths of C: Elaborating the De Facto Standards [pdf]

#26

Post by pjmlp »

pavlov wrote: post #25
pjmlp wrote: post #23

Earlier quoted context omitted.

> The fact these curiosities are not an issue in day-to-day work and C is (one of) the most popular languages around today mean that they aren't too serious. C became one of the most used languages, not necessary popular, thanks to the adoption of UNIX and the rise of FOSS/C culture (UNIX based) in the late 90's. Back then it was just yet another systems programming language. I only started to care about it when I mo…

Late '90s is about a decade off, IMO. C was very much the only systems language by then. I think the last time a popular OS was built on something else than C was the original Mac OS, which had a Pascal API and used Pascal calling conventions. On the desktop, C was chosen as the API language for Windows and OS/2 around 1986. That meant both Microsoft and IBM agreed that PC software is going to be in written in C.

On MS-DOS, Acorn, Amiga and Atari it was just yet another one.

On Mac OS (after the C transition), Windows and OS/2, they might had C as main implementation, but most of us that couldn't carry on using Turbo/Quick/HiSoft Pascal, Modula-2 or Basic compilers, moved to C++ instead.

We could still make us of improved safety and stronger type checking features, while being compatible with the C toolchains.

EDIT: Also IBM and Microsoft eventually had very good C++ support in the form of SOM, COM, C Set++ and MFC, with Borland providing the very good OWL and VCL.

Also any Windows 3.x old timer remembers the message and event handling macros alongside #define STRICT, that Microsoft used to bring some sanity to Windows programming with straight C.

Re: Into the Depths of C: Elaborating the De Facto Standards [pdf]

#27

Post by ricksplat »

pavlov wrote: post #25
pjmlp wrote: post #23

Earlier quoted context omitted.

> The fact these curiosities are not an issue in day-to-day work and C is (one of) the most popular languages around today mean that they aren't too serious. C became one of the most used languages, not necessary popular, thanks to the adoption of UNIX and the rise of FOSS/C culture (UNIX based) in the late 90's. Back then it was just yet another systems programming language. I only started to care about it when I mo…

Late '90s is about a decade off, IMO. C was very much the only systems language by then. I think the last time a popular OS was built on something else than C was the original Mac OS, which had a Pascal API and used Pascal calling conventions. On the desktop, C was chosen as the API language for Windows and OS/2 around 1986. That meant both Microsoft and IBM agreed that PC software is going to be in written in C.

> Microsoft and IBM agreed

My understanding of the narrative surrounding this is that Microsoft started working for IBM on OS/2 before switching to Windows, which - struggling here - was intended to have a certain amount of binary compatibility (?). Basically IBM decided, Microsoft went along with them, and the rest was history. Would be interesting to understand who made the decision to use C and why ... was OS/2 intended to be "unix-like"?

Re: Into the Depths of C: Elaborating the De Facto Standards [pdf]

#28

Post by anexprogrammer »

pjmlp wrote: post #26
pavlov wrote: post #25

Earlier quoted context omitted.

Late '90s is about a decade off, IMO. C was very much the only systems language by then. I think the last time a popular OS was built on something else than C was the original Mac OS, which had a Pascal API and used Pascal calling conventions. On the desktop, C was chosen as the API language for Windows and OS/2 around 1986. That meant both Microsoft and IBM agreed that PC software is going to be in written in C.

On MS-DOS, Acorn, Amiga and Atari it was just yet another one. On Mac OS (after the C transition), Windows and OS/2, they might had C as main implementation, but most of us that couldn't carry on using Turbo/Quick/HiSoft Pascal, Modula-2 or Basic compilers, moved to C++ instead. We could still make us of improved safety and stronger type checking features, while being compatible with the C toolchains. EDIT: Also IBM…

On Amiga it was the only one of significance, as the API language. Having Lattice helped of course. Pretty much everything else was a toy in the early years.

Most significant DOS development seemed to be C.

Re: Into the Depths of C: Elaborating the De Facto Standards [pdf]

#29

Post by gpderetta »

pcwalton wrote: post #10
TickleSteve wrote: post #8

I've been using C for over 20 years and I'm sure I would be caught out by these... ...but... The fact these curiosities are not an issue in day-to-day work and C is (one of) the most popular languages around today mean that they aren't too serious. When you have a knowledge of the hardware and are working at that level day-in day-out then issues like this really don't bother you that much. (I do realise that this is…

Undefined behavior in the form of memory safety issues is a problem in day-to-day work.

UB and memory safety are a big issue exactly because they are not a problem in day to day work.

I literally can't remember the last time I had spent any significant time investigating one of these issues. In my experience when that a crash happen (usually in a unit test or the first time you start the app) because of these issues, the backtrace points you to the exact problem.

The pain start when the program and tests work correctly for all reasonable inputs and the underlying issue never manifests in during normal execution and can be potentially exploited by a malicious attacker with a carefully crafted input.

What I'm trying to say is that I don't want memory safety because it would improve my daily programming experience (in fact possibly the reverse would be true), but I want it because I want security.

Re: Into the Depths of C: Elaborating the De Facto Standards [pdf]

#30

Post by gpderetta »

Too wrote: post #20
kerneis wrote: post #5

Earlier quoted context omitted.

I believe the questions in this study (I did not write it, only know the authors) were deliberately open-ended, allowing for comments on the specifics. A previous, much longer version used to contain code examples to comment, but it proved too detailed for people to complete. Moreover, the study was explicitly not about ISO C: "We were not asking what the ISO C standard permits, which is often more restrictive, or ab…

Is memset and memcmp compatible with strict aliasing? Intuitively it seems like that would be a gap into the aliasing rules. Altough void* is allowed to alias anything so maybe it works through that. Ive never seen memset, memcpy or memcmp on anything but char* in production code.

The aliasing rules only talk about dereferencing pointers. Void* can't be dereferenced so it has no interaction with the aliasing rules. You might be thinking about char, and yes, you are allowed to dereference char to inspect the bytes of an object, which is what the various mem* functions do under the hood.
Post reply on HN