Actor system for the JVM developed by Electronic Arts

So pulling from the alluded to real life example, we wrote a job scheduling system. Now, each job could be defined by a lifecycle. It wasn't just a single fire and forget command, but multiple in time (a prep start, start, stop, clear, commands), and also tracking status of an external resource (looking for error states or ambiguities, etc), and could also be modified (change of start and stop times, including "now" for both).

Now, using actors, this was trivial to do. Each actor was basically a glorified state machine, with each command, status update, etc, a message coming in that updated the internal state, and caused it to change state, and, where appropriate, update anything it needed to (such as the internal timer for when it would stop the job). We loaded up everything that had to happen in the next 12 hours, via an actor that would wake up and load more periodically (with an actor registry to find and confirm what existed already, as well as used to route incoming commands from users to change jobs). There is no complexity in managing a global state of priority between the actors; that's a 'solved problem' by the time slicing the underlying actor engine is doing for us; it's well tested, well proven, and invisible to us. Our implementation allows us to treat each job as existing in isolation, and write our code as though it and it alone is the job that needs to be executed. There is no global synchronization state to manage (well, with the caveat of taking the infinite list of future jobs, and ensuring we have actual processes equating to those scheduled to start in the next 12 hours, but that's a comparatively simple problem, and a proper use of sequential ordering. We want to execute on these jobs first), nor should their be based on the problem we're trying to solve.

Using a queue and traditional threading model though? Well, we'd need a synchronized priority queue, as items could have their priority changed at any time, from multiple sources. We'd also still need a registry. We'd need a pool of workers, that are mostly just blocking...hope we don't have > (number of workers) things needing to execute concurrently. Our complexity to manage all of this is high; the level of testing needed to validate that there aren't emergent bugs, also high. Our semi-realtime requirements are probably out the window (high contention on the queue, or high enough concurrency to exhaust the thread pool could lead to very measurable delays in updates). And ultimately, it all comes to the fact that the queue exists to pretend that the state of the world is sequential, and that we then had to then contort ourselves around to bring back to a state that allows for suitable concurrency. Updates to a job can end up killing a worker and inserting something back onto the queue, can remove something from the queue and reinsert it at a different position, etc, all while other workers are trying to access the queue simultaneously.

Actors are the simpler solution.

The comment about an actor's mailbox being a queue is missing the point (though, amusingly, demonstrating it). Queues are sequential. Actors are capable of doing only one thing at a time, and so when they are requested to do multiple things, they will do them sequentially. A mailbox filling with things that don't demand sequential processing is a code smell in an actor system. This is useful if you have limited resource access (queries on a DB connection, say), or need things to be done sequentially, and are very much akin to threads. But if you actually want multiple things concurrently...you should have multiple actors, not send them into the mailbox of one actor (which is very much the queue situation mentioned above; you imposed a sequential ordering on things you want done concurrently. Why?). That's the key difference; in most other languages, the queue is taking multiple things you actually want concurrently, and imposing a sequence to them. In the actor model, your queue is in fact things you DO want done sequentially; the things you want done concurrently should get federated out to multiple actors.

Not always. Goblins[0] is an example of an actor system that supports both synchronous and asynchronous communication between actors, with synchronous communication only possible if both actors are in the same “vat” (a kind of event loop).

[0] https://docs.racket-lang.org/goblins/

The OSS project I work on, Dapr (Distributed Application Runtime - an incubated CNCF project) implements the virtual actor pattern if anyone is interested.

https://docs.dapr.io/developing-applications/building-blocks...

https://github.com/dapr/dapr

So I think the Erlang distinction is a little unrelated (I mean, it's technically accurate, but not because of the error recovery; that's an implementation detail. The original Actor formulation Carl Hewitt proposed was heavily influenced by Smalltalk semantics; whereas Smalltalk was truly OO, in that everything is an Object that passed messages, one issue it had was that objects did not equate to threads of execution. The Actor model, then, said each object executed independently. But in the same way that Smalltalk did not have primitives; even integers are always Objects, the Actor model stipulates there are no primitives; even integers are Actors). But I'll take a stab at answering what I think you're asking.

If your services don't deal with internal mutable state, nor high degrees of concurrency then there isn't much gain to be had with an actor system. That said, that begs the question of what the queue is for; just create more instances, since there's no internal state to share.

As soon as you start having internal mutable state and high levels of concurrency, that's where the actor model applies. Queues don't exist for concurrency (you don't need them; just create more executors), they exist for imposing sequence where it is needed (an obvious case; you have a DB connection, you want to only have one query at a time. So every desired query goes into a queue, and the process at the end that owns the DB connection pulls from it). Internal mutable state gets stored inside of an actor; updates and reads get serialized on that actor.

At the highest level, I would describe the actor model as taking a 'successful' model for distributed computing, and making it the only model you use, even locally.

In, let's say Java, for instance, using standard concurrency approaches, it matters where a process lives. My way of operating/communicating to another thread of execution (unit of concurrency) is very, very different than my way of operating/communicating to another machine. Locally I have threads and locks and need to be very mindful. When communicating to another machine, I send a message and that's it (maybe I expect a response, and timeout if I don't get one, but that's really just the same thing, the other machine sending a message).

I don't actually need a queue involved for theoretical correctness unless I need to process messages in sequence (after all, I could have multiple copies of the other process, and send a message to each of them). Now, in the real world I do, simply because if my concurrency gets too large it can't be handled by what the units of concurrency already available (instances, threads, whatever), and scale up takes time, but that's really just a special case of why I need to impose a sequence on messages (handle these first, then handle these, rather than handle all of them concurrently).

The actor model makes this the local model of communication (and so makes the impedance mismatch negligible between local and distributed; so much so that some languages it's actually irrelevant whether you're sending a message to a local actor, or a remote one). Scaling concurrency up internally just means spin up a new actor. When you need to serialize, you send messages to the same actor, where it ends up in a queue.

So it's not solving a different problem, exactly, if that problem is "how do we write systems that can do multiple things at once", but the specifics, complexity, etc, tend to be pretty different. The problems it's solving are a bit more subjective than simply "can we handle this problem", and more "how well do our tools and mental model lend themselves to the problem we're trying to solve".

Been digging my self deeper and deeper into Dapr, Orleans and similar for several years now.

Super cool tech! so thank you for your contributions and future work on this.