presentation video: https://www.youtube.com/watch?v=PnPzeavMjmE
GC Tuning Confessions of a Performance Engineer
11–20 of 98 posts
Re: GC Tuning Confessions of a Performance Engineer
#12For some reason, GC is on of those areas that I have zero desire to learn sufficiently to tune. That's probably why OP has a successful career as a consultant. When making tradeoffs in development, a simpler GC with predictable behavior is worth losing a lot of raw performance to me. Thus I find myself drawn to platforms like erlang/elixir or Haskell where GC is isolated. I suppose the rational approach is build on H…
In 80% of the cases, simple GC ergonomics (i.e. let HotSpot figure out the tuning) are more than ok, and in 95% of the cases, trivial tuning is enough. The thing with Erlang is that the GC doesn't work on any shared memory data structure (like ETS), so you pretty much have to delegate any shared data to an out-of-process database, even in simple cases that are easily addressed with ConcurrentHashMap, ConcurrentSkipLi…
Good point, but ETS is still managed and owned by a process. So it still can be cleaned up taken care of. Moreover at the API level ETS even behaves as if there is a process that manages data (even though implementation-wise it is all done in C).
Also in Erlang large binaries are not copied but shared as well.
And whether large shared data-structures are needed is dependent on the application. I imagine since you've done work with spatial relationships, in that case for anything interesting to happen data has to be indexed, shared, queried in one place. But say building a large concurrent chat or messaging application with many but small state machine actor that might have lots of peer to peer interaction, then shared data is not as crucial.
Re: GC Tuning Confessions of a Performance Engineer
#13Earlier quoted context omitted.
I love GCs. They give you better memory throughput (in exchange for more footprint and higher latency, although latency can be made rather low), and they let you build and use very scalable concurrent data structures. On large machines with lots of cores and lots of RAM, they let you work with large, in-memory data sets very efficiently.
You will never beat a tuned system without GC with a GC system because of all the instructions necessary to traverse references and such. Having lots of cores and lots of RAM just means you can eat the cost more easily. It just means you can afford to let the VM do the work instead of the programmer. Also, no matter how much RAM you have, cache sizes are more or less the same, and cache line misses hurt.
> You will never beat a tuned system without GC with a GC system because of all the instructions necessary to traverse references and such.
That's not at all how it works. The generational hypothesis means that most objects die young. Allocating them is a simple, uncontended pointer bump in the thread-local allocation buffer (as fast as stack allocation), and freeing them is free, as they are never traversed. They are much, much (much) faster than malloc/free. They do however increase the frequency of young-gen so they have an indirect cost.
When it comes to throughput (i.e. total time the application spends doing memory management), modern GCs handily beat malloc/free. What you can do, however, with manual memory management is all sorts of arena allocations, but then you have to be careful when sharing pointers (Rust helps with that). Then there's the question of concurrent data structures, that are very, very hard to do well without a GC.
> cache line misses hurt.
What does that have to do with GCs? If anything, copying GCs bring related objects together, so the prefetcher can help. What affects cache misses (in Java) is the lack of arrays-of-structs, which is scheduled to be fixed, with the addition of value types, in Java 10.
Re: GC Tuning Confessions of a Performance Engineer
#14Earlier quoted context omitted.
I love GCs. They give you better memory throughput (in exchange for more footprint and higher latency, although latency can be made rather low), and they let you build and use very scalable concurrent data structures. On large machines with lots of cores and lots of RAM, they let you work with large, in-memory data sets very efficiently.
As with everything in engineering, this is a tradeoff. Not all access patterns work well with mark-and-sweep (or equivalent full traversal patterns). In particular, without needing to release or allocate memory, a GC pass wastes cycles. Furthermore, data structures must be compatible with the GC: traversal is proportional to the number of pointers (rooted or dangling) in the heap and stack at the time of running. Gen…
> traversal is proportional to the number of pointers (rooted or dangling) in the heap and stack at the time of running.
That's not quite how it works. Traversal is proportional to the number of pointers changed since the last collection (HotSpot's GCs do card marking).
I do agree there are tradeoffs, but they're much more nuanced than that. There is certainly a footprint tradeoff, and there is a latency tradeoff (that can be mitigated).
Re: GC Tuning Confessions of a Performance Engineer
#15For some reason, GC is on of those areas that I have zero desire to learn sufficiently to tune. That's probably why OP has a successful career as a consultant. When making tradeoffs in development, a simpler GC with predictable behavior is worth losing a lot of raw performance to me. Thus I find myself drawn to platforms like erlang/elixir or Haskell where GC is isolated. I suppose the rational approach is build on H…
Yeah that is why I like the BEAM VM as well. It is a really awesome piece of engineering build for massive concurrency and fault tolerance. It is nice to see the whole Elixir ecosystem take advantage of it.
That model of concurrency is important to reduce cognitive load on the programmer. Having concurrent units and mutable state isolated makes it much easier to understand what is going on, rather than having some large class or shared memory state modified from many code paths concurrently (maybe via callbacks, threads, signals, etc.)
Re: GC Tuning Confessions of a Performance Engineer
#16Having worked with all sorts of GCs in the past, I basically stopped using them altogether some years ago in favor of manual memory management. I have to say, it's been liberating and so much easier to write performant code since I know what memory I need instead of relying on the computer to guess (and giving the computer hints as to how to guess). I have no doubt that GC research and systems is making good forward…
I love GCs. They give you better memory throughput (in exchange for more footprint and higher latency, although latency can be made rather low), and they let you build and use very scalable concurrent data structures. On large machines with lots of cores and lots of RAM, they let you work with large, in-memory data sets very efficiently.
Not to mention that you get to hire performance engineers.
People may have overestimated their utility (that they are good for less things then commonly thought). Maybe not to the degree that the stereotypical C/++ would believe, but still an overestimation.
Maybe we just need to use more abstractions in the middle between manual and fully automatic memory management, like region-based memory management (just an example).
Re: GC Tuning Confessions of a Performance Engineer
#17Earlier quoted context omitted.
You will never beat a tuned system without GC with a GC system because of all the instructions necessary to traverse references and such. Having lots of cores and lots of RAM just means you can eat the cost more easily. It just means you can afford to let the VM do the work instead of the programmer. Also, no matter how much RAM you have, cache sizes are more or less the same, and cache line misses hurt.
> You will never beat a tuned system without GC with a GC system because of all the instructions necessary to traverse references and such. That's not at all how it works. The generational hypothesis means that most objects die young. Allocating them is a simple, uncontended pointer bump in the thread-local allocation buffer (as fast as stack allocation), and freeing them is free, as they are never traversed. They ar…
Comparing to malloc and free directly isn't very useful because that's pretty much the worst way to manage memory. Aside from memory arenas, you can just pool memory or preallocate too. I think concurrent data structures have very niche usage in practice (compared to say, a thread safe system).
>> cache line misses hurt.
> What does that have to do with GCs?
Everything. Sure you can use a copying GC and hope related objects go together. I tell the compiler to lump them together and exploit the prefetcher wherever possible.
Re: GC Tuning Confessions of a Performance Engineer
#18Earlier quoted context omitted.
I love GCs. They give you better memory throughput (in exchange for more footprint and higher latency, although latency can be made rather low), and they let you build and use very scalable concurrent data structures. On large machines with lots of cores and lots of RAM, they let you work with large, in-memory data sets very efficiently.
You will never beat a tuned system without GC with a GC system because of all the instructions necessary to traverse references and such. Having lots of cores and lots of RAM just means you can eat the cost more easily. It just means you can afford to let the VM do the work instead of the programmer. Also, no matter how much RAM you have, cache sizes are more or less the same, and cache line misses hurt.
The pragmatic solution is to write the critical parts of the code either completely allocation-free or using some kind of automatic memory management. To write large amounts of boilerplate and non performance sensitive code using manual memory management is likely time that could be better spent elsewhere. Similarly, tweaking a compute intensive part of a CLR/Jvm/Python/ruby program to fix GC performance is probably more work than just rewriting that particular piece of code with allocations in mind, in C if necessary.
Re: GC Tuning Confessions of a Performance Engineer
#19Earlier quoted context omitted.
You will never beat a tuned system without GC with a GC system because of all the instructions necessary to traverse references and such. Having lots of cores and lots of RAM just means you can eat the cost more easily. It just means you can afford to let the VM do the work instead of the programmer. Also, no matter how much RAM you have, cache sizes are more or less the same, and cache line misses hurt.
> You will never beat a tuned system without GC with a GC system because of all the instructions necessary to traverse references and such. That's not at all how it works. The generational hypothesis means that most objects die young. Allocating them is a simple, uncontended pointer bump in the thread-local allocation buffer (as fast as stack allocation), and freeing them is free, as they are never traversed. They ar…
I really dislike the comparisons of TLAB to stack allocation. Stack, by its nature, is going to be hot in cache. TLAB, once filled up, will be retired and possibly assigned to a different thread. But even if it's not assigned, it's constantly moving forwards, and not revisiting the same space. You'd need prefetch to be perfect, and then on top of that, you'd need to make sure that by the time you go to allocate again, the prefetched cachelines have not been evicted.
Languages/platforms with a GC should still use, support, and encourage stack allocation for temporary memory -- this is your TLAB!
There is a cost to traversing references; card marking and generational collectors just reduce the amount of references you need to visit, but it doesn't mean reference chasing isn't requiring extra instructions. Finally, don't forget that card marking requires write barriers, which is extra instructions (and possible cache misses) on each reference store (modulo trivial ones, such as new allocations, where JIT knows it's not required).
Re: GC Tuning Confessions of a Performance Engineer
#20Earlier quoted context omitted.
As with everything in engineering, this is a tradeoff. Not all access patterns work well with mark-and-sweep (or equivalent full traversal patterns). In particular, without needing to release or allocate memory, a GC pass wastes cycles. Furthermore, data structures must be compatible with the GC: traversal is proportional to the number of pointers (rooted or dangling) in the heap and stack at the time of running. Gen…
> traversal is proportional to the number of pointers (rooted or dangling) in the heap and stack at the time of running. That's not quite how it works. Traversal is proportional to the number of pointers changed since the last collection (HotSpot's GCs do card marking). I do agree there are tradeoffs, but they're much more nuanced than that. There is certainly a footprint tradeoff, and there is a latency tradeoff (th…
A card in the card table covers more than one object, so the entire dirty card needs to be scanned. Card marks only help with old->young references, and stack based roots still need to be visited each time, irrespective of what happened in the last collection.