Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Horror in the Standard Library

zerotier.com

Post reply on HN
Print view View original

151–160 of 222 posts

Re: The Horror in the Standard Library

#151

Post by makomk »

kosma wrote: post #89
bostik wrote: post #51

Earlier quoted context omitted.

> you can't trust libraries blindly, even one of the most used and broadly adopted ones There is a corollary to development and debugging. When things break in mysterious ways, we tend to go through a familiar song and dance. As experience, skills and even personal networks grow, we can find ourselves diving ever further in the following chain. 1. "It must be in my code." -- hours of debugging 2. "Okay, it must be so…

> 10. "Where is my chip lab grade oscilloscope?" 11. "Shit, where do I borrow a spectrum analyzer and a set of near field probes? These things cost an arm and a leg!" Yes, STM32F1 MCUs generate inference that jams GPS receivers. No, it's not documented anywhere.

You'd be surprised how much you can find out just with a $10 TV tuner dongle and a piece of coax with a short section of the outer braid trimmed off at once end.

Re: The Horror in the Standard Library

#152

Post by nly »

int_19h wrote: post #85
nly wrote: post #75

Earlier quoted context omitted.

> you can count on developers not getting the size right, for that to be a common error, and for everyone to cobble together their own, wildly different and probably slower ways of tracking size You have to do this anyway. You either know the size of the thing you allocated the memory for or, if it's a block, you need to keep track of the size for bounds checking purposes. There are no circumstances in which you call…

It's not just about C. In C++, for example, you might be deleting a base-typed pointer to a derived instance, that can be one of several options that you don't know at compile time.

Doesn't matter, since you call the destructor for the derived class through the vtable. The function you end up calling knows the size of your object and its exact layout. The size at this point is compiled statically in to your program.

Re: The Horror in the Standard Library

#153

Post by Joeri »

bogomipz wrote: post #137

This was a nice write up, however I didn't follow how memory fragmentation was related to a memory leak. Can someone explain? I understand that alternate memory allocators would help with the fragmentation issue but how does the choice of allocators affect memory leakage?

When you have a rapid sequence of large and small allocations, your memory starts looking like swiss cheese. To fit a large allocation you need a contiguous block of free memory, but all the available blocks are too small, so it gets placed at the end where the free ram starts. Then another allocation is put right after it, the big chunk gets freed, and a small allocation is put in its place. Now instead of one big block of free memory you have a slightly smaller block, which is slightly too small to fit a large allocation, which has to be put at the end, and...

I ran into this issue once debugging a performance issue with a CAD file parser. It made a lot of copies of large and small chunks, and some CAD files would cause catastrophic fragmentation. Switching out the allocator for a smarter one fixed the problem. A few versions of delphi later they put that allocator in by default, so that now it is not prone to fragmentation anymore.

Re: The Horror in the Standard Library

#154

Post by nly »

kabdib wrote: post #128
nly wrote: post #75

Earlier quoted context omitted.

> you can count on developers not getting the size right, for that to be a common error, and for everyone to cobble together their own, wildly different and probably slower ways of tracking size You have to do this anyway. You either know the size of the thing you allocated the memory for or, if it's a block, you need to keep track of the size for bounds checking purposes. There are no circumstances in which you call…

Knowing a size and being forced to retain a size are different things. Here's a trivial example (imagine a system to process compressed audio data): (1) receive packet: compute actual size, allocate buffer, expand data into it, start DMA to a speaker or something (2) DMA-done: free the buffer There, I didn't need the size on the free. There are many, many many similar cases, in fact these cases probably dominate.

Both your speaker driver and your application code still needed to keep track of the buffer size. calling free(size) on your buffer would require no extra overhead

Re: The Horror in the Standard Library

#155

Post by nly »

theclaw wrote: post #119
nly wrote: post #76

Earlier quoted context omitted.

> The C++ delete[] operator doesn't take a size parameter either Yes it does. See overload 6 here http://en.cppreference.com/w/cpp/memory/new/operator_delete

Note that the size parameter is ignored by the standard library implementation. It is intended for use by user-defined implementations.

> Note that the size parameter is ignored by the standard library implementation

Only because they all call malloc() and free() under the covers. It was however an ABI break, and one the C++ community thought worth doing.

If you replace your malloc with jemalloc you can easily wire this up to sdallocx(). The C++ standard library obviously can't do this out of the box because it can't assume you are using jemalloc

Re: The Horror in the Standard Library

#156

Post by cortesoft »

logicallee wrote: post #82
theoh wrote: post #16

Earlier quoted context omitted.

I'm not sure "resmelting" is even a thing. I am sure that the GP is joking.

Guys, yes! I was joking. I was directly riffing on the common claim that C++ is "just a tool". This exact phrase, in quotes, has over a hundred thousand hits on Google[1], many of them immediately comparing it to a hammer or a screwdriver. The usual context is that C++ isn't really "unsafe" -- it's just a tool, how you use it is up to you. In the case of the standard library being broken, it is like a tool that is br…

I have no idea why I thought you were serious. Poe's law?

Re: The Horror in the Standard Library

#157

Post by s_kilk »

SideburnsOfDoom wrote: post #141
s_kilk wrote: post #98

Earlier quoted context omitted.

The official one, I think it was in .NET 3, but it was a few years ago at an old job, so I'm a bit hazy on the details. Basically we had a bug where a whole conditional branch was being skipped, and we traced it down to the branch being omitted entirely from the compiled IR. And no, it was nothing fancy, just something like: if (customer.country == "US") { doSomething(); } else { doDifferentThing(); } The whole `else…

> Anyway, the point is that the compiler fucked up it's handling of if/else statements, but only at that specific part of the code It would have to be specific. Put it this way: if this was a general bug and the "else" was always omitted, how long would it last before being found and fixed? Related, if you were to say to me "I found the issue, the compiler isn't correctly handling if/else statements" Then my first th…

> if you were to say to me "I found the issue, the compiler isn't correctly handling if/else statements" Then my first thought would be about your medication not about the compiler

And yet, it happened :)

And the senior engineers at the company looked at it and confirmed it was a compiler bug. Their best guess was that something about that part of the code was putting the compiler in a funny state, causing it to skip that particular `else` branch.

We reported it to Microsoft, but never heard anything back.

Re: The Horror in the Standard Library

#158

Post by api »

mjw1007 wrote: post #69
MaysonL wrote: post #64

Earlier quoted context omitted.

From the documentation (linked by arunc above)[0] "Notes about deallocation. This allocator does not explicitly release memory. Because of this, memory debugging programs like valgrind or purify may notice leaks: sorry about this inconvenience. Operating systems will reclaim allocated memory at program termination anyway. If sidestepping this kind of noise is desired, there are three options: use an allocator, like n…

That doesn't explain the «Some allocations were much larger than anything ZeroTier should need.» part.

Yes it does. The secondary C++ allocator is doing its own pooling on top of the primary allocator.

Re: The Horror in the Standard Library

#159

Post by Pica_soO »

s_kilk wrote: post #98
kyberias wrote: post #97

Earlier quoted context omitted.

Care to elaborate? Which C# compiler?

The official one, I think it was in .NET 3, but it was a few years ago at an old job, so I'm a bit hazy on the details. Basically we had a bug where a whole conditional branch was being skipped, and we traced it down to the branch being omitted entirely from the compiled IR. And no, it was nothing fancy, just something like: if (customer.country == "US") { doSomething(); } else { doDifferentThing(); } The whole `else…

Sounds like optimization going haywire, deducing that the statement under question would constantly evaluate to this term. Its valid to optimize a else statement out- if it will never be reached (Dead Codepath Optimizing out). Was there something akin to this in the statement?
Post reply on HN