Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mysterious Moving Pointers

blomqu.ist

Post reply on HN
Print view View original

41–50 of 68 posts

Re: Mysterious Moving Pointers

#41

Post by kentonv »

inetknght wrote: post #39
kentonv wrote: post #37

Earlier quoted context omitted.

In most (quite possibly all ) cases I don't know the upper bound even at runtime. So, this ends up requiring allocation.

You don't think it's useful to calculate some reasonable upper bound if only to avoid resource exhaustion?

Not really:

* In my experience arbitrary limits intended to prevent "resource exhaustion" tend to lead to incidents where the limit was hit but resources weren't really exhausted, so it just caused failures for no reason. Very common example is the ulimit on open file descriptors. The default limits feel like they were set last century and haven't been updated to account for the fact that we have way more memory these days. We should really just be enforcing an overall limit on memory usage (per-user or per-cgroup) and expect that to include the file descriptor table.

* If I did choose a limit, it would be quite high (to avoid aforementioned unnecessary incidents), but in practice most of the lists I'm thinking of would never get near that size, so pre-allocating arrays would waste a lot of memory.

Re: Mysterious Moving Pointers

#42

Post by im3w1l »

TillE wrote: post #3

Storing a pointer to memory that you did not explicitly allocate is always a red flag, I think. You really need to understand how everything works, and be very careful. I would default to just using std::unique_ptr in a situation like this, especially since using std::list suggests performance isn't critical here, so the additional indirection probably doesn't matter.

I mean the article explains the original author did take explicit steps to keep the Nodes fixed in memory (std::list), because they knew it could be a problem if the Nodes moved.

They just got tripped up by an obscure language feature that made the Nodes move anyway.

Re: Mysterious Moving Pointers

#43

Post by jinchengJL »

jnwatson wrote: post #5

This is a great reminder of the pox that was Microsoft of the early part of the millennium. Besides an allergy to investing in web standards, they were woefully behind in their language support. Their non-adoption of modern C++ standards held client security back for a decade, and arguable held language standards development back.

Having only worked with gcc and clang, OP’s code looked completely fine to me and I was baffled why many comments think the code is at fault. Judging by this page [1], I agree this is entirely MSVC’s doing.

[1] http://howardhinnant.github.io/container_summary.html

Re: Mysterious Moving Pointers

#44

Post by mabster »

olliej wrote: post #36
wffurr wrote: post #4

Earlier quoted context omitted.

What do you use instead of std::vector, map, unique_ptr, etc? I have a hard time thinking of C++ and the STL as separate. Even our internal utilities and such tend to be STL-like although often with safer defaults.

Essentially the same things but reimplemented safely - see WTF in webkit. There are still issues (the iterator API used by for(:) is very hard to make safe without terrible perf issues, though I was looking at this recently and the compilers are doing much better than they used to). Things like unique_ptr and shared_ptr do not meaningfully improve the security of c++ despite being presented as if they did (all seriou…

Yep, we've always had our own implementation of std::shared_ptr for this reason.

Either the reference is elsewhere (and now you have to dereference another area of memory occasionally which is the worst case for cache performance), or its alongside your object. If its alongside your object it's better to know it's there for padding, etc.

And it's easy to forget to allocate for the alongside case, so you can have hidden poor performance.

Re: Mysterious Moving Pointers

#45

Post by inetknght »

kentonv wrote: post #41
inetknght wrote: post #39

Earlier quoted context omitted.

You don't think it's useful to calculate some reasonable upper bound if only to avoid resource exhaustion?

Not really: * In my experience arbitrary limits intended to prevent "resource exhaustion" tend to lead to incidents where the limit was hit but resources weren't really exhausted, so it just caused failures for no reason. Very common example is the ulimit on open file descriptors. The default limits feel like they were set last century and haven't been updated to account for the fact that we have way more memory thes…

So you don't know how much to allocate and you also don't want to allocate too much.

It sounds like you want something built on top of std::vector but with your own rules about when to reserve() and how much.

Re: Mysterious Moving Pointers

#46

Post by mhh__ »

userbinator wrote: post #12

IMHO this is another case where C++'s hidden layers of complexity hides bugs that would've been obvious in plain C. In fact for this particular use-case I'd probably use indices instead of pointers.

On the flip side C projects often end up with things like linked lists and pointers to [it depends] because that lack of abstraction, while obvious, encourages the programmer to take shortcuts (otherwise no time for actual logic)

Re: Mysterious Moving Pointers

#47

Post by o11c »

kentonv wrote: post #28
o11c wrote: post #16

Earlier quoted context omitted.

The good news is that since C++ containers aren't special to the language, you can just implement your own wrapper classes that disable the copy ctor (and provide an explicit `.clone()` instead). Coupled with `#pragma GCC poison` it is pretty easy to blacklist legacy footguns in source files at least (though not in headers without some aggressive work). ... and yet, almost all vulnerabilities in C++ code are still wr…

Yeah I pretty much only use my own alternate container implementations (from KJ[0]), which avoid these footguns, but the result is everyone complains our project is written in Kenton-Language rather than C++ and there's no Stack Overflow for it and we can't hire engineers who know how to write it... oops. [0] https://github.com/capnproto/capnproto/blob/v2/kjdoc/tour.md

The problem with that is that it is not providing the standard library API, but rather its own API. A good alternative should only remove footguns (ideally, to the point that you can replace with typedefs and get correct behavior just with less protection in case of future changes).

Aside, it's embarrassing that all these alternative libraries fail to implement automatic correct signed/unsigned mixing. It's not like it's particularly hard to implement!

Re: Mysterious Moving Pointers

#48

Post by kentonv »

inetknght wrote: post #45
kentonv wrote: post #41

Earlier quoted context omitted.

Not really: * In my experience arbitrary limits intended to prevent "resource exhaustion" tend to lead to incidents where the limit was hit but resources weren't really exhausted, so it just caused failures for no reason. Very common example is the ulimit on open file descriptors. The default limits feel like they were set last century and haven't been updated to account for the fact that we have way more memory thes…

So you don't know how much to allocate and you also don't want to allocate too much. It sounds like you want something built on top of std::vector but with your own rules about when to reserve() and how much.

No, for the use cases I'm thinking of, I really do just want an intrusive linked list. Why would I want to invent a convoluted way to adapt std::vector when an intrusive linked list does exactly what I want?

(Example use case: Some number of objects want to register themselves as observers on some event, and unregister themselves later, in arbitrary order. The same object may register and unregister itself many times. std::unordered_set would be the best fit if performance doesn't matter but an intrusive linked list requires no allocation at all on behalf of the list.)

Re: Mysterious Moving Pointers

#49

Post by kentonv »

o11c wrote: post #47
kentonv wrote: post #28

Earlier quoted context omitted.

Yeah I pretty much only use my own alternate container implementations (from KJ[0]), which avoid these footguns, but the result is everyone complains our project is written in Kenton-Language rather than C++ and there's no Stack Overflow for it and we can't hire engineers who know how to write it... oops. [0] https://github.com/capnproto/capnproto/blob/v2/kjdoc/tour.md

The problem with that is that it is not providing the standard library API, but rather its own API. A good alternative should only remove footguns (ideally, to the point that you can replace with typedefs and get correct behavior just with less protection in case of future changes). Aside, it's embarrassing that all these alternative libraries fail to implement automatic correct signed/unsigned mixing. It's not like…

Many std library footguns are inherent to the API...

Re: Mysterious Moving Pointers

#50

Post by chaboud »

inetknght wrote: post #11
chaboud wrote: post #7

Earlier quoted context omitted.

That’s a bit like saying you know C++ but not streams or templates, or C but not floating point operations. It’s probably worth learning STL. Anyway, the reason to use move instead of copy is for performance. Move constructors are faster because they can leave the source object modified (e.g., take over control of a pointer to deep contents). This falls apart when the move constructor can throw, because the container…

> leaving the object before the exception modified and the code in an unrecoverable state It isn't likely to leave the code in an unrecoverable state even if recovery is calling std::terminate (or worse). It is likely to leave the data in an unrecoverable state. Imagine that a vector of 4 items was resized -- the first two objects move successfully, but the third one throws an exception. Then in your move function, y…

Sure, but assumptions about the state of data are made in real world code. It’s just a mess, which is why the code in question needs to break into jail and very explicitly indicate that it isn’t going to fire off exceptions (and hold to it). Honestly, C++ move semantics and default behaviors could be its own very lengthy conversation. It’s why the bulk of my C++ code over the years has been explicit about references and copies (since most of my C++ code has been about high performance real time rendering or data analysis).
Post reply on HN