Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event Sourcing is Hard

chriskiehl.com

Post reply on HN
Print view View original

61–70 of 166 posts

Re: Event Sourcing is Hard

#61

Post by latencyloser »

ta93754829 wrote: post #45

One of the problems I think I see with event sourcing is its inability to scale. You have to guarantee the order of events, right? How do you do that in a large scale distributed system with eventual consistency, without incurring an insane synchronization time penalty? I'd genuinely love to hear if you have a good solution for this, because if you do I have a use case I need it for, so this ain't a troll comment!

You generally only have to guarantee the ordering of _some_ events--not all of them. Ideally you want a way to partition your data such that ordering is only important per partition. For example, when I worked in derivatives trading, we only needed to guarantee ordering by product class, but events could be generally unordered given multiple product classes, as long as each individual one was ordered. Similarly, the telemetry systems I work on now guarantee ordering per sensor (duh) and per sensor grouping, but not across _all_ sensors on the vehicle. Partitioning makes for easy scaling (just make a new partition yay).

Re: Event Sourcing is Hard

#62

Post by eropple »

jsmeaton wrote: post #51
manigandham wrote: post #33

Event Sourcing = everything that happens is an event. Save all the events and you can always get to the latest state, as well as what things look like at any time in the past. What an "event" is depends on your business domain and the granularity of processing. It's very common in enterprise apps with complex workflows (like payment processing or manufacturing). Good fit for functional programming techniques and make…

What about when event structures change? Now you’re having to push versions into your events and keeping every version of your serialisation format. Redux often does not have to keep track of versions, because the event stream is consistent for that session.

Typically you write an in-place migration for every version, yes. Or you snapshot your working-copy database and archive the event stream up to that point, and you play it back using the appropriate version of the processing code.

It kinda sucks and there isn't a great answer. But there are a lot of use cases where there are real benefits to having that log go back to the start.

Re: Event Sourcing is Hard

#63

Post by linkmotif »

barbecue_sauce wrote: post #53
linkmotif wrote: post #25

Earlier quoted context omitted.

As zenpsycho said, all you get is eventual consistency across aggregates, if you’re talking about projection aggregates. As you say, for domain aggregates you can wire up transactions by writing your own 2PC on top of Kafka exactly once semantics. I would recommend only to people who are really committed to the idea or know what they are doing and modeling. It took me a long time because I went from zero knowledge of…

DDD in this context = Domain Driven Design?

si

Re: Event Sourcing is Hard

#64

Post by dragonwriter »

jsmeaton wrote: post #51
manigandham wrote: post #33

Event Sourcing = everything that happens is an event. Save all the events and you can always get to the latest state, as well as what things look like at any time in the past. What an "event" is depends on your business domain and the granularity of processing. It's very common in enterprise apps with complex workflows (like payment processing or manufacturing). Good fit for functional programming techniques and make…

What about when event structures change? Now you’re having to push versions into your events and keeping every version of your serialisation format. Redux often does not have to keep track of versions, because the event stream is consistent for that session.

> What about when event structures change? Now you’re having to push versions into your events and keeping every version of your serialisation format.

Sure. Events are like filled out forms, and there is a reason forms that are significant, modified over time, and where determining meaning from historical forms is a expected business need tend be versioned (often by form publication date). If you ever need to reconstruct/audit history (whatever your data model), you need a log of what happened when that faithfully matches what was recorded at the time, and you need definitions of the semantics as to how the aggregation of all that should map to a final state. Event sourcing is a pretty direct reification of that need, and, yes, versioning is part of what is needed to make that work.

Re: Event Sourcing is Hard

#65

Post by meetbryce »

rwilson4 wrote: post #15
AndrewKemendo wrote: post #4

The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

Events are the cornerstone of product analytics. If you want to understand what your users are doing on your platform, and to look for opportunities to improve the user experience, events are a big part of that.

Events in that context are totally different. Sure there's overlap, but that's mostly a coincidence.

Re: Event Sourcing is Hard

#67

Post by yenwel »

If you want to do event sourcing. Look at persistent actors from akka (jvm and clr). It solve a lot of feaures as discussed here with replay, snapshotting, event upgrades etc.

Re: Event Sourcing is Hard

#68

Post by ZenPsycho »

ta93754829 wrote: post #45

One of the problems I think I see with event sourcing is its inability to scale. You have to guarantee the order of events, right? How do you do that in a large scale distributed system with eventual consistency, without incurring an insane synchronization time penalty? I'd genuinely love to hear if you have a good solution for this, because if you do I have a use case I need it for, so this ain't a troll comment!

I guess, whatever a Kafka cluster does, that's how you do it.

Re: Event Sourcing is Hard

#69

Post by manigandham »

jsmeaton wrote: post #51
manigandham wrote: post #33

Event Sourcing = everything that happens is an event. Save all the events and you can always get to the latest state, as well as what things look like at any time in the past. What an "event" is depends on your business domain and the granularity of processing. It's very common in enterprise apps with complex workflows (like payment processing or manufacturing). Good fit for functional programming techniques and make…

What about when event structures change? Now you’re having to push versions into your events and keeping every version of your serialisation format. Redux often does not have to keep track of versions, because the event stream is consistent for that session.

The most common approach is to add versions to the events. The good thing is that with event sourcing, the exact cutover and lifetimes of these schema versions can be known (and even recorded as events themselves).

Downstream apps and consumers that don't need to be compatible with the entire timeline can then migrate code over time and only deal with the latest version. You have to deal with schemas anytime you have distributed communications anyway, but event sourcing provides a framework for explicit communication and this is one area where it can make things easier.

Post reply on HN