Event Sourcing Is Hard (2019)

We had a guy who spent two weeks writing a report script that collated data uploaded into S3 and wrote out another data lump into S3 after some post processing then sent this data lump to a guy via email. This entire thing had a bunch of lambda layers to pull in the pipeline, a build pipeline in jenkins, terraform to deploy it. The python script itself was about 200 lines of the equivalent of spreadsheet cell manipulation basically.

Only after it was implemented we found out it was only run every 3 months and took the original dude 5 mins to paste it into his existing excel workbook and the data popped out in another sheet instantly. This was of course a “massive business risk” and justified a project to replace it. What it should have been was a one page thing in confluence with the business process in it and the sheet attached. But no it now needs a GitHub repo full of shite and 2 separate people to understand every facet of it who require 3x the salary of the original dude each. And every 3 months, something is broken or it pukes an error which requires someone 3 hours of debugging and reverse engineering the stack to work out why.

Fundamentally, event sourcing, as the above, is the outcome of seeing something shiny and covered in marketing and wanting to play with it on any available business case rather than doing a rational evaluation of suitability. There are use cases for this stuff, but sometimes people don’t know when to apply the knowledge they have.

As for ES itself, I have resisted this one big time, settling on CQRS class functionality which is quite frankly no difference to query/mutate database APIs using stored procedures in PL/sql I dealt with in the dark ages. People also don’t have the architectural skills or conceptual ability generally to rationalise the change on data consistency on top of that or the recovery concerns. I would rather find another way to skin the scalability cat than use that architecture.

> In effect, the raw event stream subscription setup kills the ability to locally reason about the boundaries of a service. Under "normal" development flows, you operate within the safe, cozy little walls which make up your service.

I wonder if this isn't more a case of the "cozy little walls" turning out to be a lie.

If your business requirements are that this user action has to trigger those half-dozen barely related background tasks, then your design will always be messy and complex, no matter if you use service calls, event queues, polling or whatever else.

In such a situation, I think a better question would be which design lets you most easily reason about the entire system - e.g., what is the dependency structure, how is the causal chain organized, etc.

Of course there are also "secondary" concerns, such as: Which design lets you make changes to the system easily, can you still easily develop and test things in isolation that can be isolated, etc. I think services and events actually have some upsides here (but I still may be in the "seduced" phase...)

I think this calls more for a better API/dependency management (and treating events as full-fledged APIs!) and a good tracing solution which lets you follow an action through different services (and event queues).

> You're free to make choices about implementation and storage and then, when you're ready, deal with how those things get exposed to the outside world. It's one of the core benefits of "services". However, when people are reaching into your data store and reading your events directly, that 'black box' property goes out the window

Again, I would absolutely treat events just as much as a public API as REST. Only because the causal realtionship is inversed doesn't mean you can throw away schemas, versioning, dependency tracking etc.

For an interactive application it's an absolute nightmare when you don't have read-your-own-writes support (including synchronization for "indexes"). Async background updates aren't always appropriate or easy to do.