Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debugging a memory leak in a Clojure service

charanvasu.com

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Debugging a memory leak in a Clojure service

#11

Post by NightMKoder »

If your clojure pods are getting OOMKilled, you have a misconfigured JVM. The code (e.g. eval or not) mostly doesn't matter.

If you have an actual memory leak in a JVM app what you want is an exception called java.lang.OutOfMemoryError . This means the heap is full and has no space for new objects even after a GC run.

An OOMKilled means the JVM attempted to allocate memory from the OS but the OS doesn't have any memory available. The kernel then immediately kills the process. The problem is that the JVM at the time thinks that _it should be able to allocate memory_ - i.e. it's not trying to garbage collect old objects - it's just calling malloc for some unrelated reason. It never gets a chance to say "man I should clear up some space cause I'm running out". The JVM doesn't know the cgroup memory limit.

So how do you convince the JVM that it really shouldn't be using that much memory? It's...complicated. The big answer is -Xmx but there's a ton more flags that matter (-Xss, -XX:MaxMetaspaceSize, etc). Folks think that -XX:+UseContainerSupport fixes this whole thing, but it doesn't; there's no magic bullet. See https://ihor-mutel.medium.com/tracking-jvm-memory-issues-on-... for a good discussion.

Re: Debugging a memory leak in a Clojure service

#12

Post by pwagland »

NightMKoder wrote: post #11

If your clojure pods are getting OOMKilled, you have a misconfigured JVM. The code (e.g. eval or not) mostly doesn't matter. If you have an actual memory leak in a JVM app what you want is an exception called java.lang.OutOfMemoryError . This means the heap is full and has no space for new objects even after a GC run. An OOMKilled means the JVM attempted to allocate memory from the OS but the OS doesn't have any memo…

This is one of the areas where OpenJ9 does things a lot better than HotSpot. OpenJ9 uses one memory pool for _everything_, HotSpot has a dozen different memory pools for different purposes. This makes it much harder to tune HotSpot in containers.

Re: Debugging a memory leak in a Clojure service

#13

Post by pron »

> -XX:+TraceClassLoading -XX:+TraceClassUnloading

The best insight into the operation of the JVM is now obtained via a single mechanism, JFR (https://dev.java/learn/jvm/jfr/), the JDK's observability and monitoring engine. It records a whole lot of event types: https://sap.github.io/SapMachine/jfrevents/

See here for examples related to tracking memory: https://www.morling.dev/blog/tracking-java-native-memory-wit...

Re: Debugging a memory leak in a Clojure service

#14

Post by layer8 »

whiteros_e wrote: post #4
ayewo wrote: post #3

Interesting article. 1. I’m having a bit of trouble parsing this paragraph: > The reason eval loads a new classloader every time is justified as dynamically generated classes cannot be garbage collected as long as the classloader is referencing to them. In this case, single classloader evaluating all the forms and generating new classes can lead to the generated class not being garbage collected. To avoid this, a new…

Dynamic classes cannot be GC'd without the classloader being dereferenced. In this case, if eval used an existing classloader we would end up exhausting metaspace and leading to MaxPermGen exception. Initial Clojure implementation was checking for an already created classloader and tried to reuse. They had commented out the code that was doing it. Link to the code in the compiler: https://github.com/clojure/clojure/b…

Side note: You’re using the term “dereference” incorrectly (also in the article). It doesn’t mean “drop references”. It means “going from the reference to the thing being referenced”, or (in other words) “accessing the thing that is being referenced” [0]. It doesn’t mean the reference is going away.

[0] https://en.wiktionary.org/wiki/dereference#Verb

Re: Debugging a memory leak in a Clojure service

#15

Post by TacticalCoder »

ayewo wrote: post #3

Interesting article. 1. I’m having a bit of trouble parsing this paragraph: > The reason eval loads a new classloader every time is justified as dynamically generated classes cannot be garbage collected as long as the classloader is referencing to them. In this case, single classloader evaluating all the forms and generating new classes can lead to the generated class not being garbage collected. To avoid this, a new…

> It sounds like the solution they adopted was to instantiate a brand new classloader each time a dynamic class is evaluated ...

I was confused too (and I may still be) but that's now how I understood their solution.

Their solution, IIUC from reading TFA, is that they simply didn't use eval at all anymore. So the whole "eval loads a new classloader" thinggy (so that it can be GC'ed later on) is totally moot.

Re: Debugging a memory leak in a Clojure service

#16

Post by positr0n »

NightMKoder wrote: post #11

If your clojure pods are getting OOMKilled, you have a misconfigured JVM. The code (e.g. eval or not) mostly doesn't matter. If you have an actual memory leak in a JVM app what you want is an exception called java.lang.OutOfMemoryError . This means the heap is full and has no space for new objects even after a GC run. An OOMKilled means the JVM attempted to allocate memory from the OS but the OS doesn't have any memo…

> It never gets a chance to say "man I should clear up some space cause I'm running out".

To add to everything you said, depending on the type of framework you are using sometimes you don't even want it to do that. The JVM will try increasingly desperate measures, looped GC scans, ref processing, and sleeps with backoffs. With a huge heap, that can easily take hundreds to thousands of ms.

At scale, it's often better to just kill the JVM right away if the heap fills up. That way your open connections don't have all that extra latency added before the clients figure out something went wrong. Even if the JVM could recover this time, usually it will keep limping along and repeating this cycle. Obviously monitor, collect data, and determine the root cause immediately when that happens.

Re: Debugging a memory leak in a Clojure service

#17

Post by adityaathalye »

MBlume wrote: post #10

The article makes it sound like the system was using eval (probably on a per-request basis, not just on start-up), and also like ceasing to use eval was pretty trivial once they realized eval was the problem. I'd be curious why they were using eval and what they were able to do instead.

My thoughts exactly... off-label Eval usage.

That said, their little eval misadventure has alerted me to the details of how Clojure's eval works. I learned something today, thanks OP.

Re: Debugging a memory leak in a Clojure service

#18

Post by ayewo »

whiteros_e wrote: post #4
ayewo wrote: post #3

Interesting article. 1. I’m having a bit of trouble parsing this paragraph: > The reason eval loads a new classloader every time is justified as dynamically generated classes cannot be garbage collected as long as the classloader is referencing to them. In this case, single classloader evaluating all the forms and generating new classes can lead to the generated class not being garbage collected. To avoid this, a new…

Dynamic classes cannot be GC'd without the classloader being dereferenced. In this case, if eval used an existing classloader we would end up exhausting metaspace and leading to MaxPermGen exception. Initial Clojure implementation was checking for an already created classloader and tried to reuse. They had commented out the code that was doing it. Link to the code in the compiler: https://github.com/clojure/clojure/b…

Thanks for linking directly to the specific line number in the Compiler.java [1] code referenced in the original article.

Not sure why they have that if statement that always evaluates to true:

   if(true)//!LOADER.isBound())
I usually prefer using the GitHub permalink [1] as it is easy for the line number to go out of sync.

[1] https://github.com/clojure/clojure/blob/f376cf62bb0c30f72b0d...

Re: Debugging a memory leak in a Clojure service

#19

Post by pjmlp »

NightMKoder wrote: post #11

If your clojure pods are getting OOMKilled, you have a misconfigured JVM. The code (e.g. eval or not) mostly doesn't matter. If you have an actual memory leak in a JVM app what you want is an exception called java.lang.OutOfMemoryError . This means the heap is full and has no space for new objects even after a GC run. An OOMKilled means the JVM attempted to allocate memory from the OS but the OS doesn't have any memo…

Depends on what JVM version is being used as well, as key guideline use the latest version, or at least the latest LTS.

Folks insisting in using Java 11 or worse, Java 8, for containers are in for a surprise.

This on OpenJDK, as sibling comment points out, there are other JVMs as well.

Re: Debugging a memory leak in a Clojure service

#20

Post by wokwokwok »

roenxi wrote: post #8

This article showcases 2 harder-to-articulate features of Clojure: 1) Digging in to Clojure library source code is unsettlingly easy. Clojure's core implementation has 2 layers - a pure Clojure layer (which is remarkably terse, readable and interesting) and a Java layer (which is more verbose). RT (Runtime) happens to be one of the main parts of the Java layer. The experience of looking into a clojure.core function a…

While both of those things are true, I’d be hesitant to call out this:

> Once we moved to other tasks, we started seeing the pods go OOMKilled. We took turns looking into the issue, but we couldn’t determine the exact cause.

As a particular “yay clojure” kind of moment.

This was an obscure bug/“feature” in the clojure standard library. That’s not normal, and having to dig into the clojure standard library, even if it only a line or two, is certainly not something I’d be particularly calling out as standard practice or “easy” maintenance.

The standard library is for the most part enormously reliable.

You should almost never have to do this.

Post reply on HN