Erlang/OTP: Garbage Collector
medium.com
Erlang/OTP: Garbage Collector
1–10 of 79 posts
Re: Erlang/OTP: Garbage Collector
#2The post glosses over the most important part of Erlang's GC: it collects process heaps separately. This transforms a hard problem (collecting a global heap with low latency despite concurrent mutators) to a _much_ simpler problem, at the price of more copying. Compare Java's G1 with Erlang's GC; the former hurts my head.
For those problems that are amenable to Erlang's model, this is a fine solution. The only real improvement here would be making collection incremental.
Re: Erlang/OTP: Garbage Collector
#3The post glosses over the most important part of Erlang's GC: it collects process heaps separately. This transforms a hard problem (collecting a global heap with low latency despite concurrent mutators) to a _much_ simpler problem, at the price of more copying. Compare Java's G1 with Erlang's GC; the former hurts my head. For those problems that are amenable to Erlang's model, this is a fine solution. The only real i…
I thought Erlang's Garbage collector was incremental by virtue of being per process.
A system may have tens of thousands of processes, using a gigabyte of memory overall, but if GC occurs in a process with a 20K heap, then the collector only touches that 20K and collection time is imperceptible. With lots of small processes, you can think of this as a truly incremental collector.
It's not incremental per process, but I'm not sure it would even matter that much in practice.
Re: Erlang/OTP: Garbage Collector
#4The post glosses over the most important part of Erlang's GC: it collects process heaps separately. This transforms a hard problem (collecting a global heap with low latency despite concurrent mutators) to a _much_ simpler problem, at the price of more copying. Compare Java's G1 with Erlang's GC; the former hurts my head. For those problems that are amenable to Erlang's model, this is a fine solution. The only real i…
there are lots of foot guns for the user with this model. because transferring data between processes involves copying this can become a problem. Erlang tries to optimise the handling of large binaries by using a separate reference counted heap. however, this introduces another set of issues where memory is 'leaked' because a smaller binary is holding a reference to a larger binary or because processes that have not been GC'd have not decremented the ref count of large binaries in the heap that they no longer user.
Re: Erlang/OTP: Garbage Collector
#5The post glosses over the most important part of Erlang's GC: it collects process heaps separately. This transforms a hard problem (collecting a global heap with low latency despite concurrent mutators) to a _much_ simpler problem, at the price of more copying. Compare Java's G1 with Erlang's GC; the former hurts my head. For those problems that are amenable to Erlang's model, this is a fine solution. The only real i…
I thought Erlang's Garbage collector was incremental by virtue of being per process. A system may have tens of thousands of processes, using a gigabyte of memory overall, but if GC occurs in a process with a 20K heap, then the collector only touches that 20K and collection time is imperceptible. With lots of small processes, you can think of this as a truly incremental collector. It's not incremental per process, but…
Yes, that is how it works, except (as you implicitly note) that large heaps in single processes can cause problems; allowing incremental collection per heap would flatten the latency profile further.
Re: Erlang/OTP: Garbage Collector
#6The post glosses over the most important part of Erlang's GC: it collects process heaps separately. This transforms a hard problem (collecting a global heap with low latency despite concurrent mutators) to a _much_ simpler problem, at the price of more copying. Compare Java's G1 with Erlang's GC; the former hurts my head. For those problems that are amenable to Erlang's model, this is a fine solution. The only real i…
I thought Erlang's Garbage collector was incremental by virtue of being per process. A system may have tens of thousands of processes, using a gigabyte of memory overall, but if GC occurs in a process with a 20K heap, then the collector only touches that 20K and collection time is imperceptible. With lots of small processes, you can think of this as a truly incremental collector. It's not incremental per process, but…
Large GC jobs get scheduled on dirty schedulers today (a background thread pool), since it's not OK to block a normal scheduler more than 1ms or so in Erlang. If they could be split into smaller chunks of work, perhaps it could be done on normal schedulers, making time allocation more fair.
Re: Erlang/OTP: Garbage Collector
#7The post glosses over the most important part of Erlang's GC: it collects process heaps separately. This transforms a hard problem (collecting a global heap with low latency despite concurrent mutators) to a _much_ simpler problem, at the price of more copying. Compare Java's G1 with Erlang's GC; the former hurts my head. For those problems that are amenable to Erlang's model, this is a fine solution. The only real i…
This is a good point, thanks! I will extend the topic or maybe will be better to provide new topic as continuation of the current topic - since putting everything in one article can be difficult to understand and will increase the article itself, making it more difficult to read.
Re: Erlang/OTP: Garbage Collector
#8The post glosses over the most important part of Erlang's GC: it collects process heaps separately. This transforms a hard problem (collecting a global heap with low latency despite concurrent mutators) to a _much_ simpler problem, at the price of more copying. Compare Java's G1 with Erlang's GC; the former hurts my head. For those problems that are amenable to Erlang's model, this is a fine solution. The only real i…
Erlang also has reference counters for things like strings that are immutable and can be shared between threads (processes in Erlang).
Overall this is a good model. Use GC for small per green thread heaps. Then use reference counters for shared immutable structures that cannot form cycles and copy everything else.
Re: Erlang/OTP: Garbage Collector
#9Scaling up an MQTTwebhook relay that I wrote in Elixir to 1000’s of long running connections, I found that I needed to manually trigger periodic GCs on my long lived processes.
As binary strings work their way through the pipelines via messages, it leaves binaries on the binary heap that don’t go away because the ref count stays above 1. There are a number of GC parameters one can tune on a per process level that might cause a long lived process to collect more aggressively. But my long lived processes have a natural “ratchet” point where it was just easy to throw a collect in. This solved all of my slow growth memory problems.
I’ve read elsewhere that Erlangs GC benefits often on the basis that must Erlanger processes are short lived.
Re: Erlang/OTP: Garbage Collector
#10The post glosses over the most important part of Erlang's GC: it collects process heaps separately. This transforms a hard problem (collecting a global heap with low latency despite concurrent mutators) to a _much_ simpler problem, at the price of more copying. Compare Java's G1 with Erlang's GC; the former hurts my head. For those problems that are amenable to Erlang's model, this is a fine solution. The only real i…
Wouldn't Erlang be much more efficient if it simply compiled to the JVM?