Combining event sourcing and stateful systems
1–10 of 41 posts
Re: Combining event sourcing and stateful systems
#2How long did it take you to come up with this approach? How many meetings etc? As a lone developer I always wonder this stuff.
Re: Combining event sourcing and stateful systems
#3How long did it take you to come up with this approach? How many meetings etc? As a lone developer I always wonder this stuff.
It took several hours of individual research, watching talks, reading blog posts; and took several pair-programming sessions of several hours over the span of four weeks to come up with a solution we liked.
We informed our client that this was a new area for us and that we didn't have hands-on experience with, but that we believed it would be beneficial to spend time to explore it, as it would be an elegant solution to several of their business problems. They agreed and we kept them in the loop with weekly meetings.
We're now in the phase of actually implementing real-life processes, the project will probably be in active development for another year or two.
Re: Combining event sourcing and stateful systems
#4You should take a look at Microsoft's Durable Functions which pairs event sourcing + (optional) actor model + serverless. It's some pretty neat tech.
I tried doing something similar to this several years ago, and here's a few issues I ran into:
1. Pub/sub in Event Sourcing is a bad idea. It's really hard to get right. (what to do if sub happens after pub due to scaling issues/infrastructure, etc?) Instead it's better to push commands deliberately to a process manager that handles the inter-domain communication and orchestration.
2. Concurrency. Ensuring aggregates are essentially single-threaded entities is a must. Having the same aggregate id running in multiple places can cause some really fun bugs. This usually requires a distributed lock of some sort.
3. Error handling. I ended up never sending a command to a domain directly, instead I sent it to a process manager that could handle all the potential failure cases.
Re: Combining event sourcing and stateful systems
#5How were the diagrams made?
Re: Combining event sourcing and stateful systems
#6How were the diagrams made?
With https://excalidraw.com/
Believe me: a whole new world will open once you've discovered it.
Re: Combining event sourcing and stateful systems
#7How long did it take you to come up with this approach? How many meetings etc? As a lone developer I always wonder this stuff.
It took several hours of individual research, watching talks, reading blog posts; and took several pair-programming sessions of several hours over the span of four weeks to come up with a solution we liked. We informed our client that this was a new area for us and that we didn't have hands-on experience with, but that we believed it would be beneficial to spend time to explore it, as it would be an elegant solution…
How do you approach estimating effort for this sort of thing? I find it awkward enough in Scrum to guess up front how many days of effort research will take and commit to delivering a plan or design by the end. If you have clients and aren't strictly bound by someone else's framework, they still want some rough idea how long research will take. Especially if the client is footing the bill. If the research is un-billed time, then the estimate is critical to you. What do you do?
Re: Combining event sourcing and stateful systems
#8I did a formal model of an event source system used in production and it was quite illuminating. It turns out that concurrency is something one should take into account when designing these systems. Versioning often refers to two things:
1. The event data itself; when business cases change or understanding grows we wish to add, remove, or change the type of different fields in an event record.
2. The current state of a projected model
The latter is what requires some form of co-ordination otherwise you can end up with events being written in an incorrect order and produce the wrong state.
It is a good idea though to avoid event sourcing all of your models. Microsoft wrote about their experiences implementing an event-sourced application and how they reached that conclusion [0]. In my experience it's because of temporal properties: event sourced systems are inherently eventually consistent systems. When you have domain models that depend on one another you will need to be quite certain that A eventually leads to B which eventually leads to C and that if a failure happens along the way that nothing is lost or irrecoverable.
[0] https://docs.microsoft.com/en-us/previous-versions/msp-n-p/j...
Re: Combining event sourcing and stateful systems
#9We encounter a similar problem at my current job (mixing systems that we want to keep stateful with systems that we want to make “real-time”/stream-based).
I think you’ve covered most of the problems you’ll encounter. One thing that sticks out to me is downtime: how will your order subscriber handle a product publisher that’s down or otherwise delayed? Then, the events will be potentially out of order, is that a problem for you?
On another note, we follow the same bounded context principles, but we implemented it with Kafka+confluent, since that infrastructure and those libraries were already available. Teams make their data accessible via a mix of “raw” Kafka topics and very refined gRPC services. Your subscriber is implemented as a cron job that reads from N stream and “reduces” them to 1 stream.
FWIW, we also store a transaction log in each of our databases, so we can generate a stream of object states relatively easily later on. This has helped a lot with converting old tables into streams, and vice versa.
The only thing that’s a persistent issue is schema changes. My only recommendation there is to never make them... In all seriousness, keep your data models small, and whenever you want to experiment with a schema change, add the new data as a FK’d table with its own transaction log, rather than a schema mutation to your core table. It’s never worth the headache if you take data integrity seriously.
Re: Combining event sourcing and stateful systems
#10You should take a look at Microsoft's Durable Functions which pairs event sourcing + (optional) actor model + serverless. It's some pretty neat tech. I tried doing something similar to this several years ago, and here's a few issues I ran into: 1. Pub/sub in Event Sourcing is a bad idea. It's really hard to get right. (what to do if sub happens after pub due to scaling issues/infrastructure, etc?) Instead it's better…
For anyone interested in event sourcing with the actor model I've built an open source Elixir library called Commanded (https://github.com/commanded/commanded) which takes advantage of Erlang's BEAM VM to host aggregate processes. There's also an event store implemented in Elixir which uses Postgres for storage (https://github.com/commanded/eventstore).
The actor model provides the guarantee that requests to a single instance are processed serially, while requests to different instances can be processed concurrently. Distributed Erlang allows these instances to be scaled out amongst a cluster of nodes with transparent routing of commands to the instance, regardless of which connected node it is running on.
In Elixir and Erlang, the OTP platform provides the building blocks to host an aggregate instance as a process (as a `GenServer`). Following the "functional core, imperative shell" style I model the domain code as pure functions with the host process taking care of any IO, such as reading and appending the aggregate's events.