this conclusion might be wrong. the code in question while it might not be allocating/freeing memory it might be stumbling on memory blocks and corrupting mem management structures. Turning the flag on might be fixing the issue by mere luck because memory allocations, locations and structures would be different
The Horror in the Standard Library
21–30 of 222 posts
Re: The Horror in the Standard Library
#22Amazing write-up. Informative and gripping in its prose.
Re: The Horror in the Standard Library
#23Did you report the issue upstream with a patch? The solution to "the standard library is broken" is to fix the standard library, no? It's all free software after all.
It's a known issue according to what I've read. It's never been fixed. This may be due to the fact that it's hard to trigger and reproduce.
Re: The Horror in the Standard Library
#24Actually it is C's malloc and free that is "broken". malloc() takes a size parameter, but free() doesn't. This imbalance means it can never be maximally efficient. Whatever GNU stdlibc++ is doing is probably, on balance, a net win for most programs. It's not exactly roses in C++ either of course. You can do better than the standard library facilities. Andrei Alexandrescu gave a great, entertaining, and technically el…
I take it you didn't read the fine article.
Re: The Horror in the Standard Library
#25Did you report the issue upstream with a patch? The solution to "the standard library is broken" is to fix the standard library, no? It's all free software after all.
It's a known issue according to what I've read. It's never been fixed. This may be due to the fact that it's hard to trigger and reproduce.
Can you link to some of the things you've read? What exactly is the "known issue"?
Re: The Horror in the Standard Library
#26Re: The Horror in the Standard Library
#27Amazing write-up. Informative and gripping in its prose.
It was a great, gripping write-up. It also corroborated why I told api he was better off using a subset of C or safe language that generated it for software like this. I told him there were tons of ways to analyze or make safe C subsets but almost nothing available that will get similarly great results on C++ code. This was a good example of where its complexity and style of sneaking in abstractions bit him in the ass in a way that might be easier to spot in C, Ada/SPARK, a Wirth language, etc. C++ style is safer on average but highly-robust code is better in restricted, analyzed C if not a safe language.
Re: The Horror in the Standard Library
#28I'm a bit confused here.
>> Most operators in C++, including its memory allocation and deletion operators, can be overloaded. Indeed this one was.
Okay, well, firstly - the issue here seems to be a problem with the implementation of std::allocator, rather than anything to do with overloading global operator new or delete. Specifically, it sounds like the blog author is talking about one of the GNU libstdc++ extension allocators, like "mt_allocator", which uses thread-local power-of-2 memory pools.[1] These extension allocators are basically drop-in extension implementations of plain std::allocator, and should only really effect the allocation behavior for the STL containers that take Allocator template parameters.
Essentially, libstdc++ tries to provide some flexibility in terms of setting up an allocation strategy for use with STL containers.[2] Basically, in the actual implementation, std::allocator inherits from allocator_base, (a non-standard GNU base class), which can be configured during compilation of libstdc++ to alias one of the extension allocators (like the "mt_allocator" pool allocator, which does not explicitly release memory to the OS, but rather keeps it in a user-space pool until program exit).
However, according to the GNU docs, the default implementation of std::allocator used by libstdc++ is new_allocator [3] - a simple class that the GNU libstdc++ implementation uses to wrap raw calls to global operator new and delete (presumably with no memory pooling.) This allocator is of course often slower than a memory pool, but obviously more predictable in terms of releasing memory back to the OS.
Note also that "mt_allocator" will check if the environment variable GLIBCXX_FORCE_NEW (not GLIBCPP_FORCE_NEW as the author mentions) is set, and if it is, bypass the memory pool and directly use raw ::operator new.
So, it looks like the blog author somehow was getting mt_allocator (or some other multi-threaded pool allocator) as the implementation used by std::allocator, rather than plain old new_allocator. This could have happened if libstdc++ was compiled with the --enable-libstdcxx-allocator=mt flag.
However, apart from explicitly using the mt_allocator as the Allocator parameter with an STL container, or compiling libstdc++ to use it by default, I'm not sure how the blog author is getting a multi-threaded pool allocator implementation of std::allocator by default.
[1] https://gcc.gnu.org/onlinedocs/gcc-4.9.4/libstdc++/manual/ma...
[2] https://gcc.gnu.org/onlinedocs/gcc-4.9.4/libstdc++/manual/ma...
[3] https://gcc.gnu.org/onlinedocs/gcc-4.9.4/libstdc++/manual/ma...
Re: The Horror in the Standard Library
#29OMG, as I was reading this I thought, "man, this reminds me of a bug I ran into with std::string back in 2000", A few sentences later, and this is also about std::string and the STL.
Mine was different though, after tracking down a memory leak that was happening with the creation of just new empty string, I discovered in the stdlib that there was a shared pointer to the empty string with a reference count of how many locations were using it (ironic that this was intended to save allocations). It turned out this was on Intel and we had what was rare at the time, a multi-processor system. It turned out that the std::string empty string reference count was just doing a vanilla ++, no locking, nothing, variable not marked volatile, nothing.
A few emails with a guy in Australia, a little inline assembly to call a new atomic increment on the counter, and the bug was fixed. That took two weeks to track down, mostly because it didn't even cross my mind that it wasn't in my code.
From that point on, I realized you can't trust libraries blindly, even one of the most used and broadly adopted ones out there.
Re: The Horror in the Standard Library
#30Actually it is C's malloc and free that is "broken". malloc() takes a size parameter, but free() doesn't. This imbalance means it can never be maximally efficient. Whatever GNU stdlibc++ is doing is probably, on balance, a net win for most programs. It's not exactly roses in C++ either of course. You can do better than the standard library facilities. Andrei Alexandrescu gave a great, entertaining, and technically el…
If free() took a size, what would that buy you?
You are aware that malloc implementations tend to stick the size just before the part returned to the caller, right? eg. let's say you store size at p, return p+4 to caller, then have free() subtract 4 again to get at this "header"... So I'm guessing that's not your suggestion because free() wouldn't work at all without that kind of hack. Or more broadly, free() or realloc() and others already need to have some way to determine the size of the allocation based on the pointer, so they track that somehow in a way opaque to the caller.
So then what...? You want programs to be able to give back a prefix of the buffer? or...?