Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How can C Programs be so Reliable? (2008)

tratt.net

Post reply on HN
Print view View original

141–150 of 230 posts

Re: How can C Programs be so Reliable? (2008)

#141

Post by pjmlp »

strlen wrote: post #49

There are multiple factors that contribute, but here's a few that I haven't (to the best of my recollection) seen mentioned so far: tooling (crucial), "do the simplest thing that could possibly work" attitude brought about by (lack of) a built-in collections library and simple syntax, and lack of rapidly changing requirements during development. Tools for C are powerful, mature, and available on near every platform.…

> Tools for C are powerful, mature, and available on near every platform. Every single C programmer that I know uses multiple memory debuggers, for example: e.g., valgrind for leak checks and more on smaller code segments, LLVM address sanitizer on unit test runs, jemalloc or tcmalloc/google-perftools, and more.

This is exactly the point. C developers need to make use of external tools to improve the language's flanky design in terms of type safety.

Tooling, outside the language standard thus not available in all systems that support C, instead of using a saner systems language like Ada, Modula-2, Oberon among many others.

Re: How can C Programs be so Reliable? (2008)

#142

Post by acon »

MichaelGG wrote: post #116
vidarh wrote: post #94

Earlier quoted context omitted.

I grew up using the Amiga, when having memory allocation fail was routine (a standard Amiga 500 for example, came with 512KB RAM, and was rarely expanded to more than 1MB, so you would run out of memory). What you do when malloc() fails depends entirely on your application: If a desktop application on the Amiga would shutdown just because a memory allocation failed, nobody would use it. The expection was you'd gracef…

Isn't fork the real offender, which requires Linux to overcommit by default? Disabling swap shouldn't affect that, right? Just makes your problem happen later, in a somewhat non-deterministic way. Without fork, what reason do you not disable swap? I can only think of an anonymous mmap where you want to use the OS VM as a cache system. But that's solved easily enough by providing a backing file, isn't it?

Saying that fork forces overcommit is strange. Fork is just one of the things that allocates memory. If you don't want overcommit fork should simply fail with ENOMEM if there isn't enough memory to back a copy of all the writable memory in the process.

Re: How can C Programs be so Reliable? (2008)

#143

Post by derleth »

vidarh wrote: post #96
nitrogen wrote: post #60

Earlier quoted context omitted.

If malloc() fails now, it might succeed again later. So you can just go on doing everything else you were doing, then try the memory-hungry operation again in the future. For example, this could be important in systems where you might be controlling physical hardware at the same time as reading commands from a network. It's probably a good idea to maintain control of the hardware even if you don't have enough memory…

This is a pet peeve of mine with modern applications: So many of them just throw their metaphorical hands in the air and give up. Prior to swap and excessive abuse of virtual memory this was not an option: If you gave up on running out of memory, your users gave up on your application. On the Amiga, for example, being told an operation failed due to lack of memory and given the chance to correct it by closing down so…

> If embedded systems programmers wrote code the same way modern desktop applications developers did, we'd all be dead.

If Boeing made passenger jets the way Boeing made fighters, we'd all be dead, too, but try telling a fighter pilot that they should do their job from a 777. It's two very different contexts.

Besides, some errors can't be recovered from. What do you do when your error logging code reports a failure? Keep trying to log the same error, or do you begin trying to log the error that you can't log errors anymore?

Re: How can C Programs be so Reliable? (2008)

#144

Post by daeken »

MichaelGG wrote: post #140
daeken wrote: post #132

Earlier quoted context omitted.

> Isn't fork the real offender, which requires Linux to overcommit by default? Maybe I'm missing something, but how does fork require overcommitment? When you fork, you end up with COW pages, which share underlying memory. They don't guarantee that physical memory would be available if every page were touched and required a copy; they just share underlying physical memory. Strictly speaking, very little allocation ha…

If there's no overcommit, each of those COW pages needs some way of making sure it can actually be written to. Isn't that literally the point of overcommit? Giving processes more memory than they can actually use on the assumption they probably won't use it? And Windows takes the different approach of never handing out memory unless it can be serviced (via RAM or pagefile). What am I missing? (I know you know way mor…

When you fork a process, your application's contract with the kernel is such: existing pages will be made accessible in both the parent and the child; these pages may or may not be shared between both sides -- if they are, then the first modification to a shared page will cause an attempt to allocate a copy for that process; execution flow will continue from the same point on both sides. That's pretty much the extent of it (ignoring parentage issues for the process tree). The key thing here is the 'attempt' part -- nothing is guaranteed. The kernel has never committed to giving you new pages, just the old ones.

I don't personally see this as an overcommit, since the contract for fork() on Linux doesn't ever guarantee memory in the way that you'd expect it to. But in all honesty, it's probably just a matter of terminology at the end of the day, since the behavior (write to memory -> process gets eaten) is effectively the same.

Edit: Though I should note, all of the overcommit-like behavior only happens if you are using COW pages. If you do an actual copy on fork, you can fail with ENOMEM and handle that just like a non-overcommitting alloc. So even in the non-pedantic case, fork() really doesn't require overcommit, it's just vastly less performant if you don't use COW.

Re: How can C Programs be so Reliable? (2008)

#145

Post by zvrba »

> it's reliable in the sense that it doesn't crash, and also reliable in the sense that it handles minor failures gracefully.

This is an incomplete definition of reliability. It seems that the author is only considering programs which work as advertised when used as intended.

Most software is written optimistically, i.e., the "good path" is tested most. As soon as you start feeding the program with invalid input, it is no longer reliable, as witnessed by the myriad of low-level security bugs (stemming from the nature of C) in virtually every nontrivial C program ever written.

Now, would you rather have your program crash (be "unreliable" by the author's definition) or end up running arbitrary code injected by an attacker?

(Security exploits are the most prominent example; many programs will do something weird when fed unexpected input.)

Re: How can C Programs be so Reliable? (2008)

#146

Post by MichaelGG »

acon wrote: post #142
MichaelGG wrote: post #116

Earlier quoted context omitted.

Isn't fork the real offender, which requires Linux to overcommit by default? Disabling swap shouldn't affect that, right? Just makes your problem happen later, in a somewhat non-deterministic way. Without fork, what reason do you not disable swap? I can only think of an anonymous mmap where you want to use the OS VM as a cache system. But that's solved easily enough by providing a backing file, isn't it?

Saying that fork forces overcommit is strange. Fork is just one of the things that allocates memory. If you don't want overcommit fork should simply fail with ENOMEM if there isn't enough memory to back a copy of all the writable memory in the process.

I meant the practical considerations of fork means overcommitment is needed in many cases where it otherwise wouldn't be needed. If you fork a 2GB process but the child only uses 1MB, you don't want to commit another 2GB for no reason.

Re: How can C Programs be so Reliable? (2008)

#147

Post by MichaelGG »

daeken wrote: post #144
MichaelGG wrote: post #140

Earlier quoted context omitted.

If there's no overcommit, each of those COW pages needs some way of making sure it can actually be written to. Isn't that literally the point of overcommit? Giving processes more memory than they can actually use on the assumption they probably won't use it? And Windows takes the different approach of never handing out memory unless it can be serviced (via RAM or pagefile). What am I missing? (I know you know way mor…

When you fork a process, your application's contract with the kernel is such: existing pages will be made accessible in both the parent and the child; these pages may or may not be shared between both sides -- if they are, then the first modification to a shared page will cause an attempt to allocate a copy for that process; execution flow will continue from the same point on both sides. That's pretty much the extent…

Oh. I was under the impression that if overcommit was disabled then forking a large process won't work if there's not enough RAM/swap available, regardless of usage.

Re: How can C Programs be so Reliable? (2008)

#148

Post by majika »

nitrogen wrote: post #139
majika wrote: post #135

Earlier quoted context omitted.

Doesn't it duplicate effort to put the responsibility of checking for available memory on individual applications? I think, in most computing environments, it should be the operating system's responsibility to inform the user that free memory is running out. Applications should be able to assume that they can always get memory to work with. I think the swap is an extremely sensible solution, in that executing program…

Most computing environments don't have a user to speak of, and the correct response of an application to an out of memory error could range from doing nothing to sounding an alarm. As a user, I find it incredibly frustrating when my old but indispensable music software runs out of address space (I have plenty of RAM) and, instead of canceling the current operation (e.g. processing some large segment of audio), just d…

I think most computing environments do have a user of, if you consider a "user" to be something that be notified and can act on such notifications (e.g. to close applications).

Your music software's problem seems to be a bad algorithm - not that it doesn't check the return values of the `*alloc` functions.Aas you say, it should be able to process the audio in constant space. While I assume that I can always acquire memory, I do still care about the space-efficiency of my software.

I must admit I've never seen my system depending on swap, so I don't know how bad it is. But, if you have 1GB of on-RAM memory already allocated, wouldn't it only be new processes that are slow?

Also, I'd again point out that if you don't like swap, you don't have to have one.

Re: How can C Programs be so Reliable? (2008)

#149

Post by yason »

azov wrote: post #63

Popular C software is so reliable because of enormous amount of effort spent developing, testing, and polishing it over many years - not because error handling in C is superior to that in higher-level languages. If your python script sticks around for 30 years constantly being used by millions of people - I bet it will be rock solid as well. All other things [1] being equal [2], a program written in a higher-level la…

I disagree with being polished. Newer C programs can be robust as well. In fact, I write C programs both for fun and at work and so do a lot of people I know, and these programs still end up being much more robust than I would even expect myself. And when you run them continuously, you will find some bugs in the program but the next bug will appear much later until you don't remember when was the last time a bug caused a failure.

There's something about the robustness you're forced to implement. I never trust scripting languages like that, not mine nor others', as much as solid C. I've seen it way too many times that when a C program is well-written solid code, it keeps on doing what it does and doesn't give surprises. I've had many Python scripts running for years and I always need to fix something little there every year. Maybe something has changed in some library, or I bump into a new failure mode, or something. This rarely happens with a C program once it has "settled".

Re: How can C Programs be so Reliable? (2008)

#150

Post by yason »

RyanZAG wrote: post #19
notacoward wrote: post #7

The author makes a good point about the discipline imposed by not having exceptions. Programmers tend to write code in one of two modes: the quick-and-dirty mode where you consistently don't check return values, or the built-to-last mode where you consistently do. If you start in the first mode and have to fix a bug, you often have to add error checking all up and down the call chain from where it occurs to where it'…

It depends on the problem you're trying to solve, I think. Let's consider a command line application that fetches a URL, like wget. Without exceptions, you would check the return code of all the system functions you call (dns, sockets, etc), and if any of them fail you can't really recover. You just write out an error message and exit. With exceptions, you could wrap the whole thing and if any system function throws…

C libraries don't just exit. A C executable could in some circumstances just fprintf() to stderr and exit(1) but usually even that requires sane deinitialization so before exiting you will effectively end up back in main() from the lower levels.

C libraries generally handle their errors on some level, possibly jumping directly out from the lowest levels and then doing deinit before passing back an error code to the caller of the library function, or exiting in a nested fashion by deinitializing stuff and reporting failure to the caller. Whatever works for whichever codebase but try to use any decently popular library, such as libcurl, and see if they just exit in the middle.

Post reply on HN