Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event Sourcing is Hard

chriskiehl.com

Post reply on HN
Print view View original

91–100 of 166 posts

Re: Event Sourcing is Hard

#91

Post by vidarh »

hueving wrote: post #82
AndrewKemendo wrote: post #4

The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

The basic idea of event sourcing is to store the actions rather than the end state of the action. This allows actions to interleave and systems calculate the endstate from all of the actions. Think of ATMs. They don't update the balance of your bank account directly. They just record a debit against it and then the sum of all of your credits and debits is your balance. This avoids it having to have some kind of lock…

In my view most of the problems with event sourcing that people run into boils down to overdoing it and modelling large parts of their systems as sets of events instead of maintaining an event log just for those things that falls out naturally as discrete events from the rest of your design.

Having certain critical events, and especially complex ones, logged as events that can be replayed and reconstructed and model transformations as operations over the state of those logs can be invaluable. Building the system around modelling all changes as events is amazingly painful.

Re: Event Sourcing is Hard

#92

Post by rhizome31 »

AndrewKemendo wrote: post #4

The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

For a fun introduction to this concept and the motivations behind it, you may watch Greg Young's talk https://youtu.be/JHGkaShoyNs

Re: Event Sourcing is Hard

#93

Post by kccqzy »

Some contrarian opinion: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...

Basic idea is to make the event log dumb. Instead of capturing business logic, make them as dumb as SQL commit logs: this row was inserted at this time, that row was deleted at that time. Sort of like a glorified WAL file. Benefits of this scheme? Querying old database states and auditing.

Re: Event Sourcing is Hard

#94

Post by BlackFly »

manigandham wrote: post #33

Event Sourcing = everything that happens is an event. Save all the events and you can always get to the latest state, as well as what things look like at any time in the past. What an "event" is depends on your business domain and the granularity of processing. It's very common in enterprise apps with complex workflows (like payment processing or manufacturing). Good fit for functional programming techniques and make…

> Save all the events and you can always get to the latest state

This isn't actually true and good event sourcing guides will point out why this isn't true. Event based systems are naturally racey, on a rerun of a stream of events the order in which the events is processed may change and you might therefore get a different result than the first time.

For example, if you have 1 item remaining in inventory but two people attempt to purchase it at the same time, that is a natural race and only 1 of them can succeed. If you are operating at the sort of scale where you require this kind of horizontal scaling, you may be in the territory where these sorts of conflicts become common.

Re: Event Sourcing is Hard

#95

Post by willvarfar »

bitcharmer wrote: post #83
spricket wrote: post #50

I agree with the author completely. I worked on a fairly large system using event sourcing, it was a never-ending nightmare. Maybe with better tooling someday it will be usable, but not now. Events are pretty much a database commit log. This is extremely space innefficient to keep around. And not nearly use useful as you might think. Re-runs need to happen pretty often as you change how events are handled. Even in ou…

I agree that it's hard, however doable and pays benefits if you know what you're doing. I worked on 3 successful implementations for finance sector and we could replay a few million messages per second. Have a look at how we achieved that in LMAX: https://martinfowler.com/articles/lmax.html Sorry to say it, but clearly you must have been doing something wrong or employing event sourcing where it does not belong.

Hmm. You are both right?

If LMAX fits your problem, and you're fine with the distribution aspects, and you can adopt a homogeneous architecture etc, then it works well.

But the way Event Sourcing is normally sold and implemented is you emit events in some components, written in some mix of languages and half of them probably already legacy, into some 'event bus' thing that you adopt and is probably written internally in some other paradigm, and then you consume these events in several different programs all implemented in other languages by other teams at other times, and you probably do all this across the atlantic!

I live in a gazillion-events-per-second world, perhaps not as hectic as finance, but sadly bigger events and sadly globally distributed, and sadly running on amazon (which will be orders of magnitude slower overall than if you have dedicated hardware once you're at this scale because of, aha, aws's complete lack of 'mechanical sympathy' ;) ). It sucks. Oh I wish I had an LMAX-shaped problem.

Re: Event Sourcing is Hard

#96

Post by sterlind »

bitcharmer wrote: post #83
spricket wrote: post #50

I agree with the author completely. I worked on a fairly large system using event sourcing, it was a never-ending nightmare. Maybe with better tooling someday it will be usable, but not now. Events are pretty much a database commit log. This is extremely space innefficient to keep around. And not nearly use useful as you might think. Re-runs need to happen pretty often as you change how events are handled. Even in ou…

I agree that it's hard, however doable and pays benefits if you know what you're doing. I worked on 3 successful implementations for finance sector and we could replay a few million messages per second. Have a look at how we achieved that in LMAX: https://martinfowler.com/articles/lmax.html Sorry to say it, but clearly you must have been doing something wrong or employing event sourcing where it does not belong.

That article is masterwork, thank you. It seems like having a "hub and spokes" architecture is key to getting event sourcing right. LMAX's Business Logic Processor is the only thing sourcing outputs, so you don't hit dependency hell. Also, your business process is literally a process, rather than living on the ledger.

It reminds me of well-designed Paxos control systems like Borg, where the journal serves as the ledger and all business logic lives in one scheduler thread.

Re: Event Sourcing is Hard

#97

Post by antofar »

I've worked in a couple of ES based trading system (matching engines and algo trading both) and it was definitely the way to go, especially on the equity side were you can restart or snapshot the system every day. In my startup, I've designed the core transactional system and it's serving us well for 2 years now. It comes with its own challenges and you need more senior devs than your average crud system hence you use it only for parts which are better specified and less subject to change. As everything it's a trade off.

Re: Event Sourcing is Hard

#98

Post by dtech »

BlackFly wrote: post #94
manigandham wrote: post #33

Event Sourcing = everything that happens is an event. Save all the events and you can always get to the latest state, as well as what things look like at any time in the past. What an "event" is depends on your business domain and the granularity of processing. It's very common in enterprise apps with complex workflows (like payment processing or manufacturing). Good fit for functional programming techniques and make…

> Save all the events and you can always get to the latest state This isn't actually true and good event sourcing guides will point out why this isn't true. Event based systems are naturally racey, on a rerun of a stream of events the order in which the events is processed may change and you might therefore get a different result than the first time. For example, if you have 1 item remaining in inventory but two peop…

There should be 1 point in the system that said "user x has ordered item a". That then becomes an event. The command from user x and y to order item a might be a race condition to see which command gets processed first, but once it has been turned into an event it's immutable and without race conditions. You should never replay commands.

Re: Event Sourcing is Hard

#99

Post by koliber »

BlackFly wrote: post #94
manigandham wrote: post #33

Event Sourcing = everything that happens is an event. Save all the events and you can always get to the latest state, as well as what things look like at any time in the past. What an "event" is depends on your business domain and the granularity of processing. It's very common in enterprise apps with complex workflows (like payment processing or manufacturing). Good fit for functional programming techniques and make…

> Save all the events and you can always get to the latest state This isn't actually true and good event sourcing guides will point out why this isn't true. Event based systems are naturally racey, on a rerun of a stream of events the order in which the events is processed may change and you might therefore get a different result than the first time. For example, if you have 1 item remaining in inventory but two peop…

Preventing race conditions in message-queue based architectures is a neat challenge. Depending on your problem domain, there are some approaches that work really well. It is likely that these don't work for all cases though.

The approach I've seen work well is to divide your messages into categories, such that events that belong to the same category MUST be handles in order, while events in different categories can be handled in any order.

Say you have ordered updates to USERs. Say you have 10 event processors. Each processors handles events for 1/10th of the users in your system. You come up with a way of partitioning your users: user_id % 10 for example. For any given user instance, only ONE processor would process ALL the events for that user. They would be processed in order based on some ordering key (i.e. time stamp, event ID).

What is tricky is that this partitioning requires first-order attention. If it gets messed up, everything crumbles and you are in for long nights and working weekends. When it works, you have a hugely efficient event processing machine that can handle anything the pipe (message queue) can throw at it.

Re: Event Sourcing is Hard

#100

Post by koliber »

AndrewKemendo wrote: post #4

The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

It is used in finance systems a lot.

Event-based systems that run on message queues are the backbone of money. These often co-exist with batch systems that process huge files (which also can be thought of as a journal of events to be processed in a batch).

Post reply on HN