Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event Sourcing Is Hard (2019)

chriskiehl.com

Post reply on HN
Print view View original

71–80 of 126 posts

Re: Event Sourcing Is Hard (2019)

#71

Post by adamauckland »

picardo wrote: post #66
adamauckland wrote: post #54

Event Sourcing is absolutely brilliant if you want to build an offline first/distributed system which will become eventually consistent. A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events. I would abso…

> A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events. This isn't a great use case for Event Souring either. An inspector must be in front of the tree to inspect it, right? And they can see each other i…

Where did I say multiple inspectors are inspecting a tree? There might indeed be multiple people dealing with a tree. The tree owner, an inspector, a tree surgeon, the local authorities (if the trees are protected).

Synchronising data at row level when offline becomes online is extremely hard to get right and reliable.

With the Event Sourcing model, every new piece of data gets thrown into the bucket , with each new piece of data the view becomes bigger.

Re: Event Sourcing Is Hard (2019)

#72

Post by invisible »

Adding something like change-data-capture to a system and using that for some of the purposes that event sourcing touts seems like it goes a lot further and doesn't require making a complex, hard-to-understand system. Having a CDC system to publish events and record those can give the biggest benefits and sit alongside a much simpler system. A downside is that the real-time feature goes away.

Re: Event Sourcing Is Hard (2019)

#73

Post by bruth »

claytonjy wrote: post #59
bruth wrote: post #44

Earlier quoted context omitted.

I have not watched that clip, but as I said above, Confluent isn't a good resource for defining this term since Kafka cannot be used to do ES. I would suggest an article like: https://domaincentric.net/blog/eventstoredb-vs-kafka Also refer to the list of great resources @oskar_dudycz posted in another comment: https://github.com/oskardudycz/EventSourcing.NetCore#1319-th...

that first article finds only one issue with Kafka-for-ES: limited number of partitions. Use pulsar, problem solved? Doesn't have the support or ecosystem of Kafka, but surely more than eventstoreDB.

They aren't solving the same problem. An event sourced entity (should) have a very small scope. It acts as a consistency boundary around some bit of state. Each event in "the stream" for that entity represents a state change. This is analogous to a database row which represents the _current_ state of the entity being modeled for that table.

Like a table, there can be an arbitrary number of rows (>millions), thus you could have millions of event streams. You can't have millions of Kafka partitions. The scope/scale of a Kafka partition is not designed for that small granularity.

Again, I have no issue with Kafka and I think Pulsar is superior in the event streaming/platform space. I also the Kafka ecosystem is impressive and I love listening Confluent Cloud podcast! But it is not designed for this use case.

Re: Event Sourcing Is Hard (2019)

#74

Post by oftenwrong »

This matches closely with my experience working at a company that was mostly built on event-sourcing. I would add a few things, some of which are touched upon in the article:

- The general problem with event-sourcing (at small scale) is that it forces you to have to think about and handle many things that can mostly be ignored with a more typical persistence approach. This overhead of complexity was one of the factors that (IMO) killed a company I worked for. Event-sourcing feels relatively "raw" in this way. However, this also provided a valuable learning experience as a developer.

- I'm curious about better ways to do event-sourcing. For example, using Datomic is apparently like a well-thought-out, pre-made event-sourcing system: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...

- If you intend to embark on an event-sourcing journey, read Versioning in an Event- Sourced System https://leanpub.com/esversioning/read , even if only to better understand the complexity of this one aspect. I agree with the author of this submission that it is hard to fully grasp the complexities of event-sourcing without trying it, and hitting many road bumps along the way, since it's not an especially well-trodden path.

- Being able to redefine the interpretation of an event as the system evolves feels like a superpower at times. Similarly, being able to bring a new part of the system online, and have it based on a rich history of everything ever done in the system, is very cool. One of the major benefits of event-sourcing is how adaptable it is in systems that change over time.

- Beware CQRS and eventual consistency. This adds a lot of complexity and is not a prerequisite for event-sourcing. It's also telling that ordinarily-simple problems to solve, like preventing duplicate username registrations, are hand-waved away by CQRS proponents: https://web.archive.org/web/20191101010824/http://codebetter... . You will face this class of problems, and you may not be able to hand-wave it away so easily.

Re: Event Sourcing Is Hard (2019)

#75

Post by xwolfi »

mpfundstein wrote: post #58
minimeme wrote: post #48

Earlier quoted context omitted.

Event sourcing can be a very powerful pattern if used correctly. You don't need to combine ES with Eventual Consistency. ES can be implemented in a totally synchronous manner and it works very elegantly capturing nicely all the busines events you need. You get a very detailed audit for free and you don't loose importan business data. You can travel back in time, construct different views of data (projections) etc. Mo…

you sound like a consultant. few questions: - what was your biggest ES system that you worked on? - how many people worked on it? - how did the ES system particulary solve your problem? - what was the tech stack? Thanks

Not the OP, not a consultant, the biggest ES system I work on is an order management system in one of the biggest investment bank in APAC, which processes orders from clients to 13 stock exchanges.

40 people approx work on it in Asia, maybe around 100 globally at the raw dev level.

I feel it's a sort of false good idea for our particular problems. Clients trade ether at the 10ms latency for high value orders or sub-ms for latency-sensitive ones. They conceptualise what they want to buy and a bunch of changes they want to apply on that bulk: for instance, change in quantity, price limits, time expiry and try to make money by matching a target price either by very closely following a prediction curve or spending as little time on doing so (giving us only a constraint and asking us to fit it by doing our own curve fitting algos).

The tech stack is pure java with a kernel of C++ for networking, with as little external dependency as possible. And no GC beyond the first few minutes after startup (to prealloc all the caches). Everything is self built.

How does it solve the problem: it simply is the only way we can think of to over optimize processing. If you want to fit a millisecond or sub millisecond target (we use fpga for that), you must cut the fat as much as possible. Our events are 1kb max, sent in raw tcp, the queue is the network switch send queue, the ordering has to he centrally managed per exchange since we have only one final output stream but we can sort of scale out the intermediary processing (basically for one input event how to slice in multiple output events in the right order).

I'd say it doesnt work: we run into terrifying issues the author of the main link pointed out so well (the UI, God, it's hard, the impossible replays nobody can do, the useless noise, solving event stream problems more than business problems etc), but I d also say I cant imagine any heavier system fitting better the constraint. We need to count the cycles of processing - we cant have a vendor library we cant just change arbitrarily. We cant use a database, we cant use heavier than tcp or multicast. I'll def try another bank one day to see how others do because I m so curious.

Re: Event Sourcing Is Hard (2019)

#76

Post by larve »

picardo wrote: post #66
adamauckland wrote: post #54

Event Sourcing is absolutely brilliant if you want to build an offline first/distributed system which will become eventually consistent. A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events. I would abso…

> A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events. This isn't a great use case for Event Souring either. An inspector must be in front of the tree to inspect it, right? And they can see each other i…

Presumably the application shows both the inspection of a single tree (by a single inspector), and handles missing events gracefully, as well as an overview of the state of a whole forest as different inspection events are coming in. I am in fact building such a tree inspection application, and all the client has to care about is the delivery of the events the UI has created, which is handled by a library. I don't need to concern myself with retrying or persisting my inspection events, and can collate the events in the same manner on the local device than what the backend will ultimately do. In this case the event sourcing model really matches the domain: inspection events that will ultimately be delivered.

Re: Event Sourcing Is Hard (2019)

#77

Post by adamauckland »

larve wrote: post #76
picardo wrote: post #66

Earlier quoted context omitted.

> A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events. This isn't a great use case for Event Souring either. An inspector must be in front of the tree to inspect it, right? And they can see each other i…

Presumably the application shows both the inspection of a single tree (by a single inspector), and handles missing events gracefully, as well as an overview of the state of a whole forest as different inspection events are coming in. I am in fact building such a tree inspection application, and all the client has to care about is the delivery of the events the UI has created, which is handled by a library. I don't ne…

You nailed it! Those missing events will slot into place as offline devices come online.

It works for any model where the use case doesn't really matter about up-to-the-minute accuracy, but for case generation and collation over time, it really excels.

We found the ES events wayyy to chatty to push back out to mobile (in 2012) so we push the aggregates back out to the mobiles.

Plus, if you're capturing GPS coordinates, running it through Kibana for real time map reporting is really exciting to watch!

Re: Event Sourcing Is Hard (2019)

#78

Post by nhumrich »

dm3 wrote: post #60
nhumrich wrote: post #56

Earlier quoted context omitted.

I re-read the article after this comment, and I would have to disagree. The article does never attempt to explain "what" recent sourcing means to them, so it's hard to know for sure. However, they do mention populating state from an event log which contains meaningless events, so I have to assume they _are_ talking about event sourcing. What leads you to believe they are taking about just streaming, and not sourcing?

They were probably also sourcing their state from the events. However most of their problems come from sharing the events between the modules/services which is not a part of Event Sourcing.

I am confused when you say sharing events isn't part of event sourcing. How does a service populate it's state from other services event source if it can't access it's events?

Re: Event Sourcing Is Hard (2019)

#79

Post by nhumrich »

oskar_dudycz wrote: post #65
dm3 wrote: post #60

Earlier quoted context omitted.

They were probably also sourcing their state from the events. However most of their problems come from sharing the events between the modules/services which is not a part of Event Sourcing.

I think that they were just building the stale read models, and used them as the write model, which created the whole confusion. Regarding the sharing events between module, it's one of the most common and the most dangerous mistakes. It's a leaking abstraction that will eventually create a distributed monolith. It has only downsides of monolith and microservices, without the upsides. I wrote longer on the topic of i…

What you are talking about is CQRS, which is a very valid pattern, and pairs well with event sourcing, but is not necessary part of event sourcing. You don't have to split your read and write models for events sourcing.

Re: Event Sourcing Is Hard (2019)

#80

Post by picardo »

adamauckland wrote: post #71
picardo wrote: post #66

Earlier quoted context omitted.

> A good example would be a tree inspection application where jobs can be pushed out to mobile inspectors who might not have phone signal but can still collect the data. Once synchronised, views are simply additive. More data can be added to cases simply by adding events. This isn't a great use case for Event Souring either. An inspector must be in front of the tree to inspect it, right? And they can see each other i…

Where did I say multiple inspectors are inspecting a tree? There might indeed be multiple people dealing with a tree. The tree owner, an inspector, a tree surgeon, the local authorities (if the trees are protected). Synchronising data at row level when offline becomes online is extremely hard to get right and reliable. With the Event Sourcing model, every new piece of data gets thrown into the bucket , with each new…

> Synchronising data at row level when offline becomes online is extremely hard to get right and reliable.

Why? Are the users in different time zones? If the time on their devices are synchronized, why is it hard to get right? I'm not trying to pick an argument with you, but trying to figure out if you've considered all the possible alternatives.

Post reply on HN