Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Akka Cluster Works: Actors Living in a Cluster

lightbend.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: How Akka Cluster Works: Actors Living in a Cluster

#41

Post by [deleted] »

macintux wrote: post #26
jpcooper wrote: post #6

Carl Hewitt, the inventor of the actor model, wrote this paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3418003 , which he posted a link to on a recent post here on Erlang. In it, he claims that Godel's Incompleteness Theorem is not true, and that the actor model is more general than the Turing machine. I am open to entertaining the idea. I've seen that his ideas have been discredited elsewhere on HN. I w…

Dr. Hewitt often pops up to discuss Erlang and the actor model here, so you might have the opportunity to ask him for more details.

[deleted]

Re: How Akka Cluster Works: Actors Living in a Cluster

#44

Post by 29athrowaway »

acjohnson55 wrote: post #33
29athrowaway wrote: post #21

Earlier quoted context omitted.

That's cool. I wish I had this in 2014.

I used Cluster Singleton with much success, starting at the very beginning of 2016: https://github.com/artsy/atomic-store/

Are you still using Akka these days?

Re: How Akka Cluster Works: Actors Living in a Cluster

#45

Post by hugofirth »

To offer a slightly dissenting opinion, we’ve had many issues with Akka over the years:

- if you roll your cluster membership a lot the dotted version vectors which are created by Akka distributed data grow unbounded. Eventually they will start making gossip messages exceed the default maximum size (a few kB IIRC) and fail to send.

- in the presence of heavy GC Akka cluster has a really bad time. Members will flip flop in marking each other unavailable. Eventually this will render the leader unable to perform its duties and you will struggle to (for example) allow a previously downed member to rejoin the cluster.

- orderly actor system shutdown will also fail under high GC, which is problematic as sometimes you need to restart your actor system.

- split-brain resolution is really really hard to get right. The Akka team have recently made theirs open source I believe which is good, but back when we were building with Akka cluster it required a Lightbend subscription.

- If you aren’t all in on Actors, the integration point between Akka and the rest of your codebase can be a little odd. You often feel like you should reach for `Patterns.ask` (a way of sending a message to an actor and then getting a Future back which will complete on a particular response) but then people tell you that’s an Anti pattern.

————

Having said all the above, if you’re able to go all in on the Actor pattern and you’re unlikely to hit high GC then you should give Akka cluster a try. The problems it tackles are genuinely hard and you should build on their hard work if you can. In particular they offer (in distributed-data) the most robust/complete set of CRDTs I’ve yet come across. Many other CRDT libraries expect you to bring your own gossip protocol and transport layer.

Re: How Akka Cluster Works: Actors Living in a Cluster

#46

Post by vinay_ys »

playing_colours wrote: post #39
halfmatthalfcat wrote: post #14

I use Akka Cluster extensively with Persistence. It's an amazing piece of technology. Before I went this route, I tried to make Akka Cluster work with RabbitMQ however I realized (like another poster here) that you're essentially duplicating concerns since Akka itself is a message queue. There's also a ton of logistics with Rabbit around binding queues, architecting your route patterns, etc that add extra cognitive o…

I remember using Akka in the domain of IoT. A persistent actor represented a sensor: state and history of readings. There was a great feature in Akka Persistence: if an actor went idle for a while - no new sensor data, it would be offloaded from memory to the storage (Cassandra). As soon as the sensor started to send new signals again or the sensor state was queried by a user the actor got loaded back to memory.

How much sensor data is kept in memory in the actor object? What's the cost tradeoff between memory vs an SSD? I wonder if SSD based solution would still be cheaper and more scalable than live memory objects based solution.

Re: How Akka Cluster Works: Actors Living in a Cluster

#47

Post by vinay_ys »

valenterry wrote: post #38
t-writescode wrote: post #36

Earlier quoted context omitted.

At that point, you're still operating on a single machine, though, and you don't need Akka Cluster for that.

I don't understand what you mean. In my scenario, multiple machines are used and necessary for the same service (otherwise there is no point in using Akka Cluster). Example: online games. You have room/game being created on the fly and it is destroyed an hour later. There are many of these rooms and they are distributed over multiple machines. You can't really use rabbitMQ here, it's not performant enough. And even i…

If you want to host a game state object in memory (vs serialized and saved to SSD) because you have lots of frequent write/reads happening to that object in a very short window of time such that the IO cost and CPU cost (ser/deserialisation) is higher and the incremental latency is a blocker, then this design of hosting a full-blown object in memory within your runtime makes sense (number of reads per game object per second must be high and must be sustained for a good number of seconds for this tradeoff math to be in this design's favor, given today's SSD costs vs memory costs).

But I wonder if you will suffer from random GC pauses, inability to carefully isolate different behaviors into different resource clusters, resulting in uncontrolled blast radius etc.

If you are anyway doing persistence (because you care to not lose game progress), and whenever a cluster node dies you need to resurrect game state from persistence, I wonder if you will get the game state restored within a bounded latency.

If this happens frequently enough (to affect say 5% of your users – enough to kill your game experience), is the benefits of latency gain from in-memory object reads wiped out?

Re: How Akka Cluster Works: Actors Living in a Cluster

#48

Post by t-writescode »

valenterry wrote: post #38
t-writescode wrote: post #36

Earlier quoted context omitted.

At that point, you're still operating on a single machine, though, and you don't need Akka Cluster for that.

I don't understand what you mean. In my scenario, multiple machines are used and necessary for the same service (otherwise there is no point in using Akka Cluster). Example: online games. You have room/game being created on the fly and it is destroyed an hour later. There are many of these rooms and they are distributed over multiple machines. You can't really use rabbitMQ here, it's not performant enough. And even i…

You're not just talking online games, you're talking enormous-world MMOs, and that's such a far cry from the areas that I have worked with and thought about how to work with in any detail that I can't usefully add anything.

If you're trying to manage the concurrent state of 10s of thousands of players in a game, all server-side, and you want specific actors to handle that single player and you don't want a globally persistent state, then I suppose this makes sense.

I've never worked with anything of that scale though.

Re: How Akka Cluster Works: Actors Living in a Cluster

#49

Post by hawk_ »

dragosmocrii wrote: post #37
halfmatthalfcat wrote: post #18

Earlier quoted context omitted.

You could but I don't think it would be the best tool for it. When I think of Akka, I'm using it because I don't want to worry about which node in my cluster any given actor is on, I just want to be able to scale horizontally and messages are routed appropriately. Akka embraces the "let it fail" mentality where as nodes go down (just as pods go down in Kuberentes), you don't have to worry about where your processes a…

Sounds like Akka and Erlang/Elixir have a lot in common

Yes Akka is essentially erlang's actor model for the JVM

Re: How Akka Cluster Works: Actors Living in a Cluster

#50

Post by kdps »

hugofirth wrote: post #45

To offer a slightly dissenting opinion, we’ve had many issues with Akka over the years: - if you roll your cluster membership a lot the dotted version vectors which are created by Akka distributed data grow unbounded. Eventually they will start making gossip messages exceed the default maximum size (a few kB IIRC) and fail to send. - in the presence of heavy GC Akka cluster has a really bad time. Members will flip fl…

Regarding garbage collection: do you think ZGC or Shenandoah could reduce the problems you mentioned?
Post reply on HN