Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Horror in the Standard Library

zerotier.com

Post reply on HN
Print view View original

51–60 of 222 posts

Re: The Horror in the Standard Library

#51

Post by bostik »

bluejekyll wrote: post #29

OMG, as I was reading this I thought, "man, this reminds me of a bug I ran into with std::string back in 2000", A few sentences later, and this is also about std::string and the STL. Mine was different though, after tracking down a memory leak that was happening with the creation of just new empty string, I discovered in the stdlib that there was a shared pointer to the empty string with a reference count of how many…

> you can't trust libraries blindly, even one of the most used and broadly adopted ones

There is a corollary to development and debugging. When things break in mysterious ways, we tend to go through a familiar song and dance. As experience, skills and even personal networks grow, we can find ourselves diving ever further in the following chain.

1. "It must be in my code." -- hours of debugging

2. "Okay, it must be somewhere in our codebase." -- days of intense debugging and code spelunking

3. "It HAS TO be in the third party libraries" -- days of issue tracker excavations and never-before-enabled profiling runs

4. "It can't possibly be in stdlib..." -- more of the same, but now profiling the core runtime libraries

5. "Please let this not be a compiler bug" -- you become intensely familiar with mailing list archives

6. "I will not debug drivers. I will not debug drivers. I will not debug drivers."

7. "I don't even know anyone who could help me figure out the kernel innards."

8. "NOBODY understands filesystems!"

9. "What do you mean 'firmware edge case'?"

And the final stage, the one I have witnessed only one person ever achieve:

10. "Where is my chip lab grade oscilloscope?"

Apart from bullheadedness, this chain also highlights another trait of a good developer. Humility.

Re: The Horror in the Standard Library

#52

Post by yokaze »

charles-salvia wrote: post #28

I'm a bit confused here. >> Most operators in C++, including its memory allocation and deletion operators, can be overloaded. Indeed this one was. Okay, well, firstly - the issue here seems to be a problem with the implementation of std::allocator, rather than anything to do with overloading global operator new or delete. Specifically, it sounds like the blog author is talking about one of the GNU libstdc++ extension…

Call me confused, too. When I was once following up the allocator in the STL because of some performance issues, it was (more or less) going directly to malloc.

I've searched now the code for GLIBCXX_FORCE_NEW, and it seems it is used in the pool_allocator and the mt_allocator [1].

String uses std::allocator [2].

I agree, the blog entry seems to missing some information to reproduce the issue. It looks to me, that the author was jumping to a conclusion, which confirmed his initial "insight". Not surprising, if you are under pressure and working over the whole weekend and nights. Who hasn't been there.

What bug me, that the standard answer seems quite often, that the whole thing is broken, and we have to switch to a complete different implementation, and/or re-write it from scratch.

[1] https://github.com/gcc-mirror/gcc/search?p=1&q=GLIBCXX_FORCE...

[2] https://github.com/gcc-mirror/gcc/blob/master/libstdc%2B%2B-...

Re: The Horror in the Standard Library

#53

Post by kabdib »

bostik wrote: post #51
bluejekyll wrote: post #29

OMG, as I was reading this I thought, "man, this reminds me of a bug I ran into with std::string back in 2000", A few sentences later, and this is also about std::string and the STL. Mine was different though, after tracking down a memory leak that was happening with the creation of just new empty string, I discovered in the stdlib that there was a shared pointer to the empty string with a reference count of how many…

> you can't trust libraries blindly, even one of the most used and broadly adopted ones There is a corollary to development and debugging. When things break in mysterious ways, we tend to go through a familiar song and dance. As experience, skills and even personal networks grow, we can find ourselves diving ever further in the following chain. 1. "It must be in my code." -- hours of debugging 2. "Okay, it must be so…

I've done the oscilloscope thing, though it was only a vanilla couple-of-hundred Mhz scope on some pins that were bugging me, and not the full deal: "Gang way, we're cracking the lid of that thing and going in." That sounds exciting, I'd love to see it.

Once, I used a chunk of ice to cool down a chip, and that made it work. The hardware guys were unimpressed. But hey, they've got cans of Chill and they use them a lot, and this software guy took a while to realize the reason the board worked in the morning and was dead by lunch, and worked for a little while again after lunch, was temperature related.

There were some devs who tracked down a nasty bug in a processor's TLB. I only heard about that one, wish I had been there. I only had to deal with the fallout in the hypervisor. Note: If you have to spend 20ms hunting down and killing lies with all interrupts turned off and everything basically stopped in its tracks, you are no longer a real-time operating system.

Re: The Horror in the Standard Library

#54

Post by arunc »

I encountered exactly the same issue few years ago in UIDAI in one of our large scale biometric matchers and the resolution was exactly the same. After a week of debugging I found that the libstdc++ allocator was the culprit. I found [1] and confirmed the same, which helped in fixing this issue.

The thing that was more interesting (or sad) was to know that the GCC developers didn't expect the multithreaded applications to be long running.

"Operating systems will reclaim allocated memory at program termination anyway. "

[1] https://gcc.gnu.org/onlinedocs/libstdc++/manual/mt_allocator...

Re: The Horror in the Standard Library

#55

Post by cyphar »

hellofunk wrote: post #40
cyphar wrote: post #2

Did you report the issue upstream with a patch? The solution to "the standard library is broken" is to fix the standard library, no? It's all free software after all.

> The solution to "the standard library is broken" is to fix the standard library, no? It's all free software after all. Doesn't the author make that case at the end?

They make the case that it's broken with very strong language. They don't say "here is a patch" or even an outline of a patch (or a link to a bug report even), just histrionics.

> GNU libstdc++ is broken. This is pretty unforgiveable. [...] Adding wheels to the wheel is sometimes forgiveable when dealing with closed systems that you can't fix but libstdc++ and glibc are both open source GNU projects.

I think there's a disconnect in the author's mind with regards to how free software projects work. If something is broken, you as a user are empowered to fix it. I mean, the author even went through the trouble of reading the libstdc++ code and figuring out what happened inside new (which is likely enough information to write a preliminary patch). At the very least open a bug report about it (or link to an existing one).

Unlike proprietary software, there are many ways to improve free software and ranting online is rarely one of them. I get that this bug screwed them over big time and that they are angry about it. But converting that emotional energy into something useful would help many people other than themselves.

tl;dr: If you find yourself ranting about "why wasn't X fixed before" in a free software project, it might be helpful to realise that you have the opportunity to be the person who fixes it.

Re: The Horror in the Standard Library

#56

Post by CamperBob2 »

arunc wrote: post #54

I encountered exactly the same issue few years ago in UIDAI in one of our large scale biometric matchers and the resolution was exactly the same. After a week of debugging I found that the libstdc++ allocator was the culprit. I found [1] and confirmed the same, which helped in fixing this issue. The thing that was more interesting (or sad) was to know that the GCC developers didn't expect the multithreaded applicatio…

Notes about deallocation. This allocator does not explicitly release memory. Because of this, memory debugging programs like valgrind or purify may notice leaks: sorry about this inconvenience. Operating systems will reclaim allocated memory at program termination anyway.

Wow. This is worth a Linus Torvalds-level rant. Whoever accepted this code into the source tree needs to be put on GNU's version of a performance improvement plan.

Re: The Horror in the Standard Library

#57

Post by kabdib »

nly wrote: post #6

Actually it is C's malloc and free that is "broken". malloc() takes a size parameter, but free() doesn't. This imbalance means it can never be maximally efficient. Whatever GNU stdlibc++ is doing is probably, on balance, a net win for most programs. It's not exactly roses in C++ either of course. You can do better than the standard library facilities. Andrei Alexandrescu gave a great, entertaining, and technically el…

Most allocators will be able to pretty efficiently recover the size of the block you are freeing. And you can count on developers not getting the size right, for that to be a common error, and for everyone to cobble together their own, wildly different and probably slower ways of tracking sizes. So it doesn't really help.

malloc/free aren't a great API, but for other reasons (namely, that you want things like multiple arenas, good control over synchronization, decent debugging and introspection, leak-tracking, tagging for figuring out what a block really is when things get smashed, block enumeration, small block pools, placement for cache alignment, and . . . you get the idea).

Re: The Horror in the Standard Library

#58

Post by CiaranMcNulty »

bostik wrote: post #51
bluejekyll wrote: post #29

OMG, as I was reading this I thought, "man, this reminds me of a bug I ran into with std::string back in 2000", A few sentences later, and this is also about std::string and the STL. Mine was different though, after tracking down a memory leak that was happening with the creation of just new empty string, I discovered in the stdlib that there was a shared pointer to the empty string with a reference count of how many…

> you can't trust libraries blindly, even one of the most used and broadly adopted ones There is a corollary to development and debugging. When things break in mysterious ways, we tend to go through a familiar song and dance. As experience, skills and even personal networks grow, we can find ourselves diving ever further in the following chain. 1. "It must be in my code." -- hours of debugging 2. "Okay, it must be so…

11. "Hm maybe I should check my code again... ah there's the bug"

Re: The Horror in the Standard Library

#60

Post by louithethrid »

cyphar wrote: post #55
hellofunk wrote: post #40

Earlier quoted context omitted.

> The solution to "the standard library is broken" is to fix the standard library, no? It's all free software after all. Doesn't the author make that case at the end?

They make the case that it's broken with very strong language. They don't say "here is a patch" or even an outline of a patch (or a link to a bug report even), just histrionics. > GNU libstdc++ is broken. This is pretty unforgiveable. [...] Adding wheels to the wheel is sometimes forgiveable when dealing with closed systems that you can't fix but libstdc++ and glibc are both open source GNU projects. I think there's…

If its broken - it is commented out, or opt in via parameter. It is not shipped - AS IS and then when after weeks and weekends the bug is found, you dont just pose yourself at the wall of the crater - lean down and yell. "Well this is just great, you discovered a cavern. With the sweat of your brow, this could be a nice house one day. Allmost like the one we promised to deliver in the first place."
Post reply on HN