Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Horror in the Standard Library

zerotier.com

Post reply on HN
Print view View original

161–170 of 222 posts

Re: The Horror in the Standard Library

#161

Post by s_kilk »

Pica_soO wrote: post #159
s_kilk wrote: post #98

Earlier quoted context omitted.

The official one, I think it was in .NET 3, but it was a few years ago at an old job, so I'm a bit hazy on the details. Basically we had a bug where a whole conditional branch was being skipped, and we traced it down to the branch being omitted entirely from the compiled IR. And no, it was nothing fancy, just something like: if (customer.country == "US") { doSomething(); } else { doDifferentThing(); } The whole `else…

Sounds like optimization going haywire, deducing that the statement under question would constantly evaluate to this term. Its valid to optimize a else statement out- if it will never be reached (Dead Codepath Optimizing out). Was there something akin to this in the statement?

> Was there something akin to this in the statement?

It probably was the optimiser at fault, but there wasn't anything special about this conditional, and certainly nothing that _should_ have caused the optimiser to throw away the else branch.

If memory serves right it was comparing a string field of an object to a static string, like `someObject.foo == "some string"`.

Re: The Horror in the Standard Library

#162

Post by gpderetta »

maxlybbert wrote: post #114
CamperBob2 wrote: post #56

Earlier quoted context omitted.

Notes about deallocation. This allocator does not explicitly release memory. Because of this, memory debugging programs like valgrind or purify may notice leaks: sorry about this inconvenience. Operating systems will reclaim allocated memory at program termination anyway. Wow. This is worth a Linus Torvalds-level rant. Whoever accepted this code into the source tree needs to be put on GNU's version of a performance i…

Years ago I read in the Perl documentation ( http://perldoc.perl.org/perlfaq3.html#How-can-I-free-an-arra... ): > On most operating systems, memory allocated to a program can never be returned to the system. ... Some operating systems (notably, systems that use mmap(2) for allocating large chunks of memory) can reclaim memory that is no longer used ... Because you can't return unneeded memory to most operating system…

>I do think it's silly for operators new/delete to have a separate free list from malloc()/free().

They don't. This is a custom, simple, non-default pool allocator for standard containers (i.e nothing to do with new)

Re: The Horror in the Standard Library

#163

Post by gpderetta »

rhaps0dy wrote: post #95
saurik wrote: post #47

Earlier quoted context omitted.

This article was an extremely long rant about "something has a if but templates are hard so I have no clue what it is". The author figured out a workaround to the issue, but we still don't know what the bug was, and the strongly worded conclusion that it is a bug in libstdc++ isn't even defended well (as this is similar to concluding "there is a bug in the compiler's code optimizer" from "I compiled my code with -O0…

I'm curious, what is usually the cause of code starting to work when it is compiled with -O0?

Crap code that hit undefined behaviour every other line (like accessing dead temporaries or freed memory, assuming stack layout, out of bound accesses, etc.).

Re: The Horror in the Standard Library

#164

Post by int_19h »

nly wrote: post #152
int_19h wrote: post #85

Earlier quoted context omitted.

It's not just about C. In C++, for example, you might be deleting a base-typed pointer to a derived instance, that can be one of several options that you don't know at compile time.

Doesn't matter, since you call the destructor for the derived class through the vtable. The function you end up calling knows the size of your object and its exact layout. The size at this point is compiled statically in to your program.

The destructor does not deallocate memory in C++, since it is called just the same for objects allocated in other ways.

It's also not a given that such an object even has a vtable. It's perfectly legal for it to not have one, and the memory is still supposed to be deallocated in full (only the base class dtor is invoked then, but if derived class additions are trivial, it's not necessarily a problem).

Now, yes, you could add a separate vtable slot for the deallocation function. Or just store the object size directly in the type info (that's usually attached to the vtable). But this is really just a way to optimize size storage for objects that already have a word utilized for shared type descriptor like a vtable.

Re: The Horror in the Standard Library

#165

Post by CamperBob2 »

badsectoracula wrote: post #92
CamperBob2 wrote: post #56

Earlier quoted context omitted.

Notes about deallocation. This allocator does not explicitly release memory. Because of this, memory debugging programs like valgrind or purify may notice leaks: sorry about this inconvenience. Operating systems will reclaim allocated memory at program termination anyway. Wow. This is worth a Linus Torvalds-level rant. Whoever accepted this code into the source tree needs to be put on GNU's version of a performance i…

This is actually a fine thing to do, i don't remember where i read it, but a good analogy for trying to free memory at program exit is like trying to clean the floors and walls of a building right before it is demolished. This is also why Valgrind separates reachable and unreachable memory and only considers unreachable memory as leaks.

How do you know the program is even supposed to terminate? Maybe it's a server, as in this case. Maybe the code will be reused someday as a subfunction or library within a larger application, instead of being launched and terminated directly by the OS. Maybe it's an embedded application in a 24/7 factory somewhere. Maybe it's on its way to the Kuiper Belt. Or maybe it's just supposed to stay up and running for longer than the average Windows 10 update period.

In any case, hiding this sort of behavior in a way that sucks down days of debugging time on the part of one expert programmer after another, after another, after another, is terrible engineering.

Re: The Horror in the Standard Library

#166

Post by garaetjjte »

frik wrote: post #86
bostik wrote: post #51

Earlier quoted context omitted.

> you can't trust libraries blindly, even one of the most used and broadly adopted ones There is a corollary to development and debugging. When things break in mysterious ways, we tend to go through a familiar song and dance. As experience, skills and even personal networks grow, we can find ourselves diving ever further in the following chain. 1. "It must be in my code." -- hours of debugging 2. "Okay, it must be so…

I've done the oscilloscope thing, but it was for IoT stuff - debugging broken a I2C communication with Arduino (8-bit 16MHz ATMega CPU). The Arduino software stack is not huge, there is no operating system involved. Our application is the only thing that runs on bare slow hardware with very limited memory. But this also makes debugging harder. The IDE is limited, you debug over serial output. You have to reflash the…

> The IDE is limited, you debug over serial output. You have to reflash the flash-memory after every re-compile

uh, you know that AVRs have debugWIRE (smaller parts) or JTAG (bigger parts)?

Re: The Horror in the Standard Library

#167

Post by maxlybbert »

gpderetta wrote: post #162
maxlybbert wrote: post #114

Earlier quoted context omitted.

Years ago I read in the Perl documentation ( http://perldoc.perl.org/perlfaq3.html#How-can-I-free-an-arra... ): > On most operating systems, memory allocated to a program can never be returned to the system. ... Some operating systems (notably, systems that use mmap(2) for allocating large chunks of memory) can reclaim memory that is no longer used ... Because you can't return unneeded memory to most operating system…

>I do think it's silly for operators new/delete to have a separate free list from malloc()/free(). They don't. This is a custom, simple, non-default pool allocator for standard containers (i.e nothing to do with new)

Thanks for the correction. But it's just as silly to create an allocator for the standard containers, when the allocator consists of little more than free list management, given that malloc/free and new/delete already do that.

It's especially silly for a library (programs may want custom memory management and libraries really shouldn't go out of their way to make that harder), and the fact that it's the standard library doesn't make it less silly.

Re: The Horror in the Standard Library

#168

Post by nickpsecurity »

api wrote: post #145
nickpsecurity wrote: post #27

Earlier quoted context omitted.

It was a great, gripping write-up. It also corroborated why I told api he was better off using a subset of C or safe language that generated it for software like this. I told him there were tons of ways to analyze or make safe C subsets but almost nothing available that will get similarly great results on C++ code. This was a good example of where its complexity and style of sneaking in abstractions bit him in the as…

(Original blog author here.) That's a nice idea, and we've considered "minus minus"ing the ZT core as part of an embedded port. But code like this that shleps a lot of structures around and works with JSON is eye gougingly painful to write in C and the chance of a worse and possibly exploitable memory bug is much higher. This is the first time we have encountered an actual problem with C++ compilers or runtimes.

You don't write those parts in C alone. You use something that shows the C is safe automatically, use tool that generates secure C from specs (eg Nail), and/ use safe language that compiles to C. This way, you get benefits of C ecosystem without risks of totally using C.

Re: The Horror in the Standard Library

#169

Post by cesarb »

s_kilk wrote: post #98
kyberias wrote: post #97

Earlier quoted context omitted.

Care to elaborate? Which C# compiler?

The official one, I think it was in .NET 3, but it was a few years ago at an old job, so I'm a bit hazy on the details. Basically we had a bug where a whole conditional branch was being skipped, and we traced it down to the branch being omitted entirely from the compiled IR. And no, it was nothing fancy, just something like: if (customer.country == "US") { doSomething(); } else { doDifferentThing(); } The whole `else…

It can get even more "fun" with Java. Your code can start running through an interpreter, then after a while suddenly be transformed by a JIT engine. The interpreter and the JIT engines (there's more than one JIT engine) have different bugs. The optimizations made by the JIT engine can depend on the data which went through your method before the JVM decided to optimize it.

I'm not finding it right now, but I recall seeing a few weeks ago a presentation with several of these sorts of bugs in a recent version of Java (all reported and fixed): after a number of iterations, it suddenly starts returning wrong results.

Re: The Horror in the Standard Library

#170

Post by [deleted] »

nly wrote: post #6

Actually it is C's malloc and free that is "broken". malloc() takes a size parameter, but free() doesn't. This imbalance means it can never be maximally efficient. Whatever GNU stdlibc++ is doing is probably, on balance, a net win for most programs. It's not exactly roses in C++ either of course. You can do better than the standard library facilities. Andrei Alexandrescu gave a great, entertaining, and technically el…

[deleted]
Post reply on HN