How Akka Cluster Works: Actors Living in a Cluster • Hacker News Board

Re: How Akka Cluster Works: Actors Living in a Cluster

#61

Post by hugofirth » Sun, Jan 17, 2021, 5:05 PM UTC

kdps wrote: post #50

hugofirth wrote: post #45

To offer a slightly dissenting opinion, we’ve had many issues with Akka over the years: - if you roll your cluster membership a lot the dotted version vectors which are created by Akka distributed data grow unbounded. Eventually they will start making gossip messages exceed the default maximum size (a few kB IIRC) and fail to send. - in the presence of heavy GC Akka cluster has a really bad time. Members will flip fl…

Regarding garbage collection: do you think ZGC or Shenandoah could reduce the problems you mentioned?

I believe it would have alleviated some of the issues yes. In general I’m excited for the benefits “pauseless” GC can bring to soft real-time systems on the JVM. Unfortunately, for now we have to continue supporting G1 and friends.

Re: How Akka Cluster Works: Actors Living in a Cluster

#62

Post by randomopining » Mon, Jan 18, 2021, 3:17 PM UTC

vinay_ys wrote: post #56

valenterry wrote: post #54

Earlier quoted context omitted.

I mean, you are right that Akka Cluster is JVM based and hence can bring the problems you mentioned. But then again, most high frequency trading also runs on the JVM, so it can often be worked around. > If this happens frequently enough (to affect say 5% of your users – enough to kill your game experience), is the benefits of latency gain from in-memory object reads wiped out? For this specific use-case, I don't thin…

> But then again, most high frequency trading also runs on the JVM, so it can often be worked around. JVM is an awesome piece of technology. And you can do robotic control systems to high-frequency trading systems with it with careful programming. But I've seen a lot of Java code running in production suffering from latency jitters and needing continuous profiling and optimization by a small group of performance engi…

Doesn't it have the new GC's that are supposed to be minimal pause and gamechangers?

Re: How Akka Cluster Works: Actors Living in a Cluster

#63

Post by valenterry » Mon, Jan 18, 2021, 11:48 PM UTC

randomopining wrote: post #62

vinay_ys wrote: post #56

Earlier quoted context omitted.

> But then again, most high frequency trading also runs on the JVM, so it can often be worked around. JVM is an awesome piece of technology. And you can do robotic control systems to high-frequency trading systems with it with careful programming. But I've seen a lot of Java code running in production suffering from latency jitters and needing continuous profiling and optimization by a small group of performance engi…

Doesn't it have the new GC's that are supposed to be minimal pause and gamechangers?

Yes, but for GC still must happen and sometimes even milliseconds can be a problem, depending on the use-case. It really depends on what your business does, so the OPs concerns with the JVM are very valid.

At the same time, I think (while not being an expert) that in the majority of the cases, the GC will not be a problem and that the time you can save from using Akka Cluster allows you to optimize your system more than enough to make up for any GC latency problems, in almost every system.

The only technology that might be better/comparable here is the erlang VM, but I have never used it myself.

Re: How Akka Cluster Works: Actors Living in a Cluster

#64

Post by acjohnson55 » Tue, Jan 19, 2021, 2:28 PM UTC

29athrowaway wrote: post #44

acjohnson55 wrote: post #33

Earlier quoted context omitted.

I used Cluster Singleton with much success, starting at the very beginning of 2016: https://github.com/artsy/atomic-store/

Are you still using Akka these days?

Sorry for the delay!

Nope. The two companies I have since worked have mature products that aren't built on the JVM or Scala. But I'd absolutely consider it if I were to build a system with similar constraints again.

There was a big learning curve, but the end result was a system that was built largely correctly, on a tight timeline, solving some tricky technical and business challenges.

Re: How Akka Cluster Works: Actors Living in a Cluster

#65

Post by reubenbond » Wed, Jan 20, 2021, 2:02 PM UTC

alexis2b wrote: post #52

t-writescode wrote: post #48

Earlier quoted context omitted.

You're not just talking online games, you're talking enormous-world MMOs, and that's such a far cry from the areas that I have worked with and thought about how to work with in any detail that I can't usefully add anything. If you're trying to manage the concurrent state of 10s of thousands of players in a game, all server-side, and you want specific actors to handle that single player and you don't want a globally p…

Interesting! Not trying to divert from Akka which is a wonderful piece of engineering, but your comment reminded me of Microsoft’s take at the Actor model with Project Orléans - which was used for the backend side of the Halo / XBox MMO [0]. I think the GA version of Project Orléans is now called Service Fabric, although I never had the pleasure to try it. [0] https://youtu.be/I91ZU8tEJkU

Orleans and Service Fabric are different. Orleans has been running in production for some time now and is actively developed on GitHub: https://github.com/dotnet/orleans. Teams inside Microsoft run it on top of Service Fabric (and Kubernetes, etc.) More details in this talk: https://youtu.be/KhgYlvGLv9c

Service Fabric has something called Reliable Actors which are heavily inspired by Orleans.

Source: I'm the project lead for Orleans