Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event Sourcing is Hard

chriskiehl.com

Post reply on HN
Print view View original

81–90 of 166 posts

Re: Event Sourcing is Hard

#81

Post by victorNicollet »

statictype wrote: post #72
spricket wrote: post #50

I agree with the author completely. I worked on a fairly large system using event sourcing, it was a never-ending nightmare. Maybe with better tooling someday it will be usable, but not now. Events are pretty much a database commit log. This is extremely space innefficient to keep around. And not nearly use useful as you might think. Re-runs need to happen pretty often as you change how events are handled. Even in ou…

>Re-runs need to happen pretty often as you change how events are handled. Aren't you supposed to take Snapshots every now and then to solve that problem?

Migrating snapshots just moves the pain of changes : instead of letting the new projection logic replay all past events, you write code to migrate snapshots and hope that it will give the same result as the replay.

In those cases, I would rather do the replay, but do it offline before the release.

Re: Event Sourcing is Hard

#82

Post by hueving »

AndrewKemendo wrote: post #4

The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

The basic idea of event sourcing is to store the actions rather than the end state of the action. This allows actions to interleave and systems calculate the endstate from all of the actions.

Think of ATMs. They don't update the balance of your bank account directly. They just record a debit against it and then the sum of all of your credits and debits is your balance. This avoids it having to have some kind of lock on your account during the transition and even allows significant delays from various transaction sources.

Re: Event Sourcing is Hard

#83

Post by bitcharmer »

spricket wrote: post #50

I agree with the author completely. I worked on a fairly large system using event sourcing, it was a never-ending nightmare. Maybe with better tooling someday it will be usable, but not now. Events are pretty much a database commit log. This is extremely space innefficient to keep around. And not nearly use useful as you might think. Re-runs need to happen pretty often as you change how events are handled. Even in ou…

I agree that it's hard, however doable and pays benefits if you know what you're doing. I worked on 3 successful implementations for finance sector and we could replay a few million messages per second. Have a look at how we achieved that in LMAX: https://martinfowler.com/articles/lmax.html

Sorry to say it, but clearly you must have been doing something wrong or employing event sourcing where it does not belong.

Re: Event Sourcing is Hard

#84

Post by garyclarke27 »

Interesting- Sounds to me like you were reinventing a database - PostgreSQL has taken over 30 years, has 400 contributers and 1.1 Million LOC, no surprise to me, it was a bit tricky! PG has a rock solid ACID enabling transaction log - the Event Source - you can access this easily via logical decoding functions. You can easily replicate this data into tables, if you need to keep it, you can also add system and applicable timestamps to get a bi-temporal db, to enable queries to go back in time. You mentioned impedence mismatch with GUI, relational databases are famous for ths same problem, fantatic tools like PostGrest and recently GraphQL integration have already been built to address this.

Re: Event Sourcing is Hard

#85

Post by vidarh »

ZenPsycho wrote: post #11
AndrewKemendo wrote: post #4

The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

It seems to be one of those things that you probably don't need. And when you do need it, it becomes obvious that you need it. Specifically, in my research, you really shouldn't use it until it becomes painfully necessary to horizontally scale writes .

Parts of it can be useful, but you don't need to split out an event bus to get auditing for example. As you say, you can avoid that until/unless you need to scale writes.

In the meantime, you can look for inbound data that naturally correspond to immutable events and apply some of the ideas to that. E.g. that form a user submits? It's reasonably an immutable event. Many of them won't matter to you, because you'll never care to audit it. But some might.

E.g. we have projections of financials being submitted by third parties. Being able to go back and audit how original form submissions relate to changes in other system state is useful, or just being able to re-run old reports after fixing bugs and confirming that the reports show what they should before/after certain events. So instead of just storing the end state, we're increasingly looking to store the original external signals that triggered those changes, and build transformations as views over that event log, and then where we need it only drive transformations to tables we don't event the same way, often with a suitable reference to the source event(s).

It avoids the problems in the article for the most part (some, such as changes in the structure of the events will always be an issue), but gets enough of the benefits to be worth it, because it's only applied to data we have that it genuinely fits (where we have clear, natural event sources, often but not always external submissions of data) where we have a need (whether for complexity reasons or because of external auditing requirements) to be able to get past views of data.

Re: Event Sourcing is Hard

#86

Post by Turbots »

I agree with some of his points, but I think he mainly saw all the bad examples doing event sourcing.

On the matter of tooling in the Java world (where the author said he was working in), there's Spring Cloud Stream (or Spring Integration if you want more finegrained control) which does most of the plumbing for you in terms of connection handling, messaging, publish/subscribe mechanisms, error handling, consumer groups, etc... First class support for RabbitMQ, Kafka and a couple of other third-party message brokers like anything JMS, Amazon SQS, and more.

If you don't like Spring, Akka is a really mature actor-based, message-driven toolkit that could really help him on the tooling side.

Re: Event Sourcing is Hard

#87

Post by Turbots »

jsmeaton wrote: post #51
manigandham wrote: post #33

Event Sourcing = everything that happens is an event. Save all the events and you can always get to the latest state, as well as what things look like at any time in the past. What an "event" is depends on your business domain and the granularity of processing. It's very common in enterprise apps with complex workflows (like payment processing or manufacturing). Good fit for functional programming techniques and make…

What about when event structures change? Now you’re having to push versions into your events and keeping every version of your serialisation format. Redux often does not have to keep track of versions, because the event stream is consistent for that session.

When doing versioning, I would suggest you only maintain the latest two versions at any given time. Supporting more than two versions should be avoided and consuming applications should upgrade to the latest version whenever they can.

Consumer-driven contract testing is a good way of testing consumers against new versions of producers. Usually, when the producer only adds a field to an event, the consumer side wont break anyway.

Re: Event Sourcing is Hard

#88

Post by hvidgaard »

l8again wrote: post #73

If people are indeed directly accessing a stream, they are violating a fundamental service oriented principle - Teams must communicate with each other through their service interfaces. It is worth understanding this concept with clarity. Assuming we go with Kafka, even if we don't need any additional functionality other than what Kafka provides out of the box, it would still need to be wrapped in a service and treate…

This is one of the core issues that make many ES systems complex. A service owns an event in the same sense that it owns internal state. It's internal to that service and only expose what it finds appropriate, in the way it finds appropriate. That is almost never directly as an event on a public eventbus.

Re: Event Sourcing is Hard

#90

Post by vidarh »

fagnerbrack wrote: post #22

First we need to agree on what "Event Sourcing" means. In my view you don't need to implement every single Event Sourcing pattern to have an "Event Sourced" system. Say you have a TODO List app (yes, pretty cliche but that's ok). In that TODO list you have a in which you post the state of the TODO list to the server. That state is stored in the database in the form of an event "TODO_LIST_SAVED". When you want to "rep…

I agree with this a lot. I posted a comment elsewhere in this thread about our use of events, and it boils down to selectively picking the entities we need to be able to reason about past states of, and storing the new states of those, and then deriving views from that state. For many uses we don't even ever need to then explicitly apply state transformations on that to derive a materalized form of the present state - a suitable view is often sufficient. For some we do need to apply transformations into new tables, but we can do that selectively. We still always have the database as a single source of truth, as we're lucky not to need to scale writes beyond that, which simplifies things a lot.

What it gives us is ability to rerun and regression test all reporting at point in time for those data sources we model as events, and ability to re-test all the code that does transformations on that inbound data, because we don't throw it away.

"Our" form of event sourcing is very different from the "cool" form: We don't re-model most internal data changes as events. We only selectively apply it to certain critical changes. A user changing profile data is not critical to us. A partner giving us data we can't recreate without going back to them and telling them a bug messed up our data is. For data that is critical like that, being able to go back and re-create any transformations from the original canonical event is fantastic.

And as long as there is an immutable key for the entity, rather than just for the entity at time t(n), we can reference from non-evented parts of the system to either entity at time t(n) or entity at time t(now()) trivially, depending on need.

Post reply on HN