[deleted]
Event Sourcing Is Hard (2019)
11–20 of 126 posts
Re: Event Sourcing Is Hard (2019)
#12Re: Event Sourcing Is Hard (2019)
#13Lots of hype about event sourcing and CQRS, usually in combination. 99% won't need it. Has the potential to ruin your team and even company unless you have a well-above-average engineering team.
Re: Event Sourcing Is Hard (2019)
#14I'm busy rewriting a project to go back to regular SQL. Whoever thought Event Sourcing was a good idea should retire from software development.
Re: Event Sourcing Is Hard (2019)
#15> The idea of a keeping a central log against which multiple services can subscribe and publish is insane. Is event sourcing about storing 1 authoritative, immutable, serialized narrative of events, or is it about tying together a pile of microservices? I think the part where it goes wrong is where we try to subscribe to these events (i.e. push). This adds delivery semantics to the mix and forces a lot more constrain…
I've built these systems. One major problem is that both scenarios are called event sourcing by different folks. The Confluent (Kafka) people will tell you that ES is a streaming approach while the EventStore folks will tell you that it's about replays, aggregates and immutability. Different blogs refer to wildly different use cases with the same name and almost no one states their assumptions up front.
Don't get me started on explaining snapshots, why you don't need snapshots until replays take 1000ms, why replays won't take 1000ms until the topics are thousands of events deep, why the team won't believe that this isn't a problem, and why you will definitely need snapshots if you keep scaling.
Re: Event Sourcing Is Hard (2019)
#16ES is really great, but just for a narrow set of service types and when it's applied locally in a closed bussines domain. But as a global system architecture, nope, I tried that once and it only brought more costs.
Re: Event Sourcing Is Hard (2019)
#17Silly question: isn't the way you're supposed to handle the observer pattern thing to be that you can follow processing chains using correlation IDs? Ie. the initial event of a dataflow gets tagged with a unique ID and then there's some database magic that lets you track that ID through the system? Like, I can see where that would go wrong once you merge multiple events together, but that seems a different failure point than described here.
And while we're at it, the whole point of eventsourcing is that they're not supposed to model "internal" process state? They're supposed to be the ground source of truth; whatever the process has to do to mangle that into a model it can work with should be no skin off the events' back.
The rest seems very, very correct to my experience. Especially not knowing the pain points until you're past the toy level.
Re: Event Sourcing Is Hard (2019)
#18Silly question: isn't the way you're supposed to handle the observer pattern thing to be that you can follow processing chains using correlation IDs? Ie. the initial event of a dataflow gets tagged with a unique ID and then there's some database magic that lets you track that ID through the system? Like, I can see where that would go wrong once you merge multiple events together, but that seems a different failure po…
"Database magic" is worrisome
Re: Event Sourcing Is Hard (2019)
#19I've worked with several event sourcing systems and was even seduced into implementing one out of sheer hubris once. These problems are ever present in every ES project I've had the misfortune of coming into contact with. It doesn't even mention the worst part that comes afterwards, when you realize after all of that pain that it is only used by a single person in the company to generate a noncritical report comparing a meaningless KPI that could have been manually done in four hours by a different intern every quarter. By the time the tooling is up and running enough to make a stable system, no one will trust it enough to use ES's landmark features except for a few developers still coming off the koolaid.
99% of the time when ES sounds like a good idea, the answer is to just use Postgres. Use wal2json and subscribe to the WAL stream - if you really need time travel or audit logs or whatever, they'll be much cheaper to implement using WAL. If you need something more enterprisey to sell to the VP-suite, use Debezium.
Event sourcing sounds so awesome in theory and it is used to great effect in many demanding applications (like Postgres! WAL = event sourcing with fewer steps) but it's just too complex for non-infrastructure software without resources measured in the 10s or 100s of man years.
Re: Event Sourcing Is Hard (2019)
#20I think one major problem is that "Event Sourcing" can mean subtly different things to different people.
> The idea of a keeping a central log against which multiple services can subscribe and publish is insane.
This really doesn't mean "Event Sourcing" to me, it sounds like enterprises that have decided Event Sourcing == Kafka (or some cloud-hosted IOT-branded variant) and treat the central broker/coordinators/confluent-cloud crap as "the central log"
To me, the fundamental idea is that changes are recorded in a meaningful format, not the result of changes; What I mean by "meaningful" is important: I don't think SQL statement replication (as implemented by popular SQL databases) constitutes "event sourcing", because the only part of the application that typically consumes this is the SQL database, but if you're storing JSON in a log (or better: some useful binary format) then you're probably doing event sourcing.
With this (perhaps broad) view of Event Sourcing, I would say I have been building all of my database applications for the last 30 years as "event sourcing", and I've never had any trouble building applications (even large ones!) using this model: It lends it self to an excellent user-experience simply by letting the UI "in on it", and the article seems to recognise this:
> Event sourcing needs the UI side to play along
If your user makes a change, and for whatever reason that change takes time to do, it is better to tell the user you're going to do it and then do it rather than wait for the HTTP request (or TCP message or whatever) to "complete" the transaction online because network always fails and bad error messages (the default!) cause users to do stupid things.
But when you use Google Cloud's UI, you can see how nice this can be: You make a change, you can look in the event log to see your changes, you can see if they've been processed or not, and you can examine and share the results simply by sharing the results pages (instead of having to make screenshots or other ghastly things).
I think for many applications this is worth it, but there aren't good tools for event sourcing (in my opinion) so it may be for a lot of applications (especially the ones I don't work on) the juice just isn't worth the squeeze -- but to me, this suggests value in developing those tools, not in ignoring this amazing thing.