Event Sourcing Is Hard (2019)

It is frustrating to argue whether "Event Sourcing" is good or bad if we have different definitions of what it is, but I don't know of a better name for the thing that I think is good, so it is helpful to point at a body like this to say this is what I mean.

For a long time, I thought about mailing lists: Back in the 1990s we had mailing list applications that you would send email to and they would do various things based on the subject line or the recipient (To) address or something like that, and so you would "join the stream" (subscribe to the list) with one email, and "leave the stream" (unsubscribe) with another; You could publish a new message (it just had to fit in an RFC821 datagram) and processing would both distribute those messages and build "views" of that data in archives accessible out-of-band (such as on the web or in gopherspace). Sequencing messages could be done by simply refusing (tempfail) messages received out of order, and persistent errors would be surfaced (eventually) in the form of an NDR which could itself have a process running on those messages.

I think it is a small thing to imagine a lower-latency email queue, changing the datagram or to manipulate the log (queue) independent of normal processing (such as removing PII -- or in our case, spam messages!) and to create other "processors" for messages for administrative tasks (like setting up customer accounts, or shutting them off for non-payment!) with a little bit of shell scripting, that if you had this kind of experience, most of what constitutes "Event Sourcing" probably doesn't seem very hard, and if you haven't had this kind of experience, that these things may dominate the design (e.g. Confluent+anything) and lead unfairly to a bad impression about "Event Sourcing"

That's not to say I don't think there are hard parts to these kinds of architectures, just that I think those hard parts are usually worth it.

If you don’t have strong ordering guarantees, it also means your state machine has to accept transitions from/to any state (eg comutative). Otherwise if ordering is important, your event log must have some ordering built-in (exactly once semantics) or you implement re-ordering and replay as past events arrive (a branching model).

I believe the point OP is trying to make, is that integration with a messaging service/queue is a separate concern from what event sourcing solves but that the internal event pipeline often looks very similar to a networked one and so the two solutions are conflated.

Firstly, to make sure we're talking about the same thing, where are you setting the bar for whether or not we can call something "Event Sourcing"? For instance, just to clarify, in our system events were indeed the source of truth. At the heart of everything sat a ledger, that ledger was stored in a database, and events were added to it as you describe in points 1-4 (hand waving away minor differences). You got a consistent, up-to-date state of an aggregate by replaying the ledger in order and applying all the events. So, to my understanding, I'd call that textbook. However, the problem may lay in our actual definition of the word.

>You don't need to cache the write model state anywhere, as the state is in events

I'd have to understand a bit more about where you're coming from with this one. While, yep, the state is in the events and you get the latest state by playing them back, that materialization is not free. If you're doing it for a single aggregate, then it's generally not a huge deal at small scales. However, once you need to make decisions against more than one thing, or compare things, or query things, that materialization cost becomes prohibitive, and you've got to set up materialized projections. With our tech stack, that had to be an eventually consistent process, and what got us into event streaming, and which ultimately caused most of my ugly crying.