Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Found a Missing Scala Class

heapanalytics.com

Post reply on HN
Print view View original

31–40 of 45 posts

Re: How We Found a Missing Scala Class

#31

Post by mcherm »

Lazare wrote: post #19
userbinator wrote: post #16

NoClassDefFoundError? But it’s right there! Although in this case the cause was very different, it reminds me of an old "trap for young players" with loading shared libraries dynamically --- the library itself can exist and be readable and executable, and yet attempting to load it fails with a "file not found" error. This happens when one of its dependencies , directly or indirectly, is missing.

I ran across another similar-yet-very-different example of this once in a completely different language. In my case, we added a new class, it worked fine on dev, then failed on staging (and would have failed in production if we'd let it go that far). This was confusing, because we were using Vagrant to ensure our dev and staging environments were identical. What could be going on? Well, our linux VMs were being hoste…

> a good reminder that 1) having dev and staging the same is really important

Actually, I think it proves that having staging and PROD the same is particularly important.

Re: How We Found a Missing Scala Class

#32

Post by jpitz »

userbinator wrote: post #16

NoClassDefFoundError? But it’s right there! Although in this case the cause was very different, it reminds me of an old "trap for young players" with loading shared libraries dynamically --- the library itself can exist and be readable and executable, and yet attempting to load it fails with a "file not found" error. This happens when one of its dependencies , directly or indirectly, is missing.

I recall, back in the mid-to-late 2000s, investigating OutOfMemoryError on a Windows server, to find that the cause was an inability to create a thread, because the kernel buffer space for managing threads was exhausted ( many thousands of threads created )

https://dzone.com/articles/outofmemoryerror-unable-create

Re: How We Found a Missing Scala Class

#33

Post by pharrington »

kazinator wrote: post #8
koube wrote: post #2

FYI this domain is blocked by default for uBlock users.

Wild-assed kazinator guess: probably be a substring/regex match on "analytics". Must be a tracking domain!

While Heap Analytics indeed is tracking software, clicking the "temporarily unblock" button allowed me to read a pretty good detective story.

Re: How We Found a Missing Scala Class

#34

Post by tjoener »

thecatspaw wrote: post #23
djsumdog wrote: post #21

The moment the article mentioned "Fat jar" I knew that'd be the problem. I don't recommend using any type of fat jar plugin (like OneJar) or even Google Guice for that matter. Custom class loaders are a nightmare. Thanks to Docker containers, you should never really need a far jar again. Just find a decent Docker packager for your build system (sbt, gradle, etc.) and it can plop all your dependencies in there in a ni…

What is problematic about Fat jars? The problem seemed to be Flink's implementation to unload the FatJar's classloader when erroring. This would have happened with slim jars as well, wouldnt it? I also dont see how docker relates exactly, you can have hundreds of library jars in a classpath with standard classloaders, no docker required

Also Scala's lambdas create new anonymous classes, but Java's lambdas are kind of bootstrapped static methods in the existing class file.

It does get converted to a class at runtime, but then it's long past the classloader anyway.

Re: How We Found a Missing Scala Class

#35

Post by shanipribadi »

drob wrote: post #3

Heap CTO here – would love to answer any questions you have. This was my first exposure to btrace, which a super useful swiss army knife for JVM debugging. That made this a worthwhile adventure for sure.

Hello,

I'm curious how do you guys operate the flink cluster, do you have a single huge shared flink cluster where people can submit any kind of jobs for various applications/streams. Or do you have multiple smaller flink clusters for specific use cases? Are they on Mesos/YARN/k8s, or just plain vms/baremetals?

Which flink version are you guys on? I'm very excited about Flink 1.6, especially related with FLIP-6 [https://cwiki.apache.org/confluence/pages/viewpage.action?pa...]

Just wondering if you guys have any thoughts on that.

Thank you.

Re: How We Found a Missing Scala Class

#36

Post by andonisus »

drob wrote: post #3

Heap CTO here – would love to answer any questions you have. This was my first exposure to btrace, which a super useful swiss army knife for JVM debugging. That made this a worthwhile adventure for sure.

Great article. I had been hunting down a similar issue in Java 8 with maven dependencies. I was getting the same error, but confirmed that the dependent jars were correctly included on the classpath. I eventually gave up and decided to take a different programming approach that did not include these missing classes, but I think I will revisit it to see if a class loader is getting closed somehow.

Re: How We Found a Missing Scala Class

#37

Post by drob »

shanipribadi wrote: post #35
drob wrote: post #3

Heap CTO here – would love to answer any questions you have. This was my first exposure to btrace, which a super useful swiss army knife for JVM debugging. That made this a worthwhile adventure for sure.

Hello, I'm curious how do you guys operate the flink cluster, do you have a single huge shared flink cluster where people can submit any kind of jobs for various applications/streams. Or do you have multiple smaller flink clusters for specific use cases? Are they on Mesos/YARN/k8s, or just plain vms/baremetals? Which flink version are you guys on? I'm very excited about Flink 1.6, especially related with FLIP-6 [ htt…

We're running a single flink cluster. Engineers can run whatever jobs they need, but we aren't writing new flink jobs that often so the load is pretty predictable. We have a single digit number of jobs at the moment.

We are on 1.3.2 at the moment, running on EC2 vms.

Re: How We Found a Missing Scala Class

#38

Post by drob »

andonisus wrote: post #36
drob wrote: post #3

Heap CTO here – would love to answer any questions you have. This was my first exposure to btrace, which a super useful swiss army knife for JVM debugging. That made this a worthwhile adventure for sure.

Great article. I had been hunting down a similar issue in Java 8 with maven dependencies. I was getting the same error, but confirmed that the dependent jars were correctly included on the classpath. I eventually gave up and decided to take a different programming approach that did not include these missing classes, but I think I will revisit it to see if a class loader is getting closed somehow.

Ooh, check it out and let me know what you find! It would make me really happy if this post helped someone debug something when they had previously hit a dead end.

Re: How We Found a Missing Scala Class

#39

Post by SuspiciousSwan »

rozap wrote: post #13
jjjensen90 wrote: post #9

Earlier quoted context omitted.

Yeah... Reading this, it smacked of a possible combination of poor tool choice and over-engineering (which I've been guilty of plenty). I built a video processing/workflow application in Scala with Akka a few years ago and debugging that was hard enough, eventually it was refactored to a simpler Kotlin/Spring application... Actor systems are great for certain use cases but you can really hurt the transparency of your…

Counterpoint: debugging erlang systems in production is a cakewalk. The tracing and introspection tools that come bundled in OTP make tracking problems down really easy. It's really hard to go back to systems that don't have erlang level visibility, so much so that it's kind of a crutch sometimes. This is an ecosystem problem and not something inherent in a program using an actor abstraction.

Systems in general tend to have a way that they like to be used. Erlang has all of the tools to support that model, and from my limited experience with it, works great when you respect it. I wouldn't reach for it due to my inexperience, but if I had someone like you on my team, I'd love to learn.

Java also has really great tooling.. when it is used like it wants to be. I don't have enough erlang experience to know if it is true there or not, but with Java, I've found that smart but green devs (like it sounds like Heap has) tend to reach for systems that solve a lot of their initial problems and cover up their initial ignorance, but then trade it for operational problems down the road. More experienced developers tend to build systems that are boring, explicit, take longer to get to Beta, but don't require a pager.

drob mentioned that they aren't writing flink jobs very often, which makes me think that they are probably using it for some sort of rollups / stream processing of their analytics data. If the business logic of those is complex, they'll probably have a bad time with outliers that fuck up their cardinality assumptions. If it isn't super complex, they probably didn't take the time to model their data correctly with boring java classes, and keep pushing complexity into the interactions between their actors.

Either way, they would have a lot faster and easier to maintain system if they forced themselves to pretend to be stuck with java 1.6 except where 1.8 stuff improved performance and readability (basically avoid abusing streams and reflection)

Post reply on HN