Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event Sourcing Is Hard (2019)

chriskiehl.com

Post reply on HN
Print view View original

81–90 of 126 posts

Re: Event Sourcing Is Hard (2019)

#81

Post by nhumrich »

oskar_dudycz wrote: post #61
nhumrich wrote: post #56

Earlier quoted context omitted.

I re-read the article after this comment, and I would have to disagree. The article does never attempt to explain "what" recent sourcing means to them, so it's hard to know for sure. However, they do mention populating state from an event log which contains meaningless events, so I have to assume they _are_ talking about event sourcing. What leads you to believe they are taking about just streaming, and not sourcing?

Could you provide an exact quote? I haven't been able to find any usage of "populate". I found: `the raw event stream subscription setup kills the ability to locally reason about the boundaries of a service.` `you have to talk to the people who will be consuming the events you produce to ensure that the events include enough data for the consuming system to make a decision` `wire a fleet of services together via an e…

You seem to be saying that event sourcing and event streaming are mutually exclusive.

In almost all cases of event sourcing I have seen, event sourcing is event streaming, but more. When you say event sourcing, where do other services source from, if there is no stream? Yes, event sourcing typically uses a database instead of a real "queue", but it still functions like a queue. You still need consumers to subscribe to events, in order to populate state, as timely as possible.

Re: Event Sourcing Is Hard (2019)

#82

Post by adamauckland »

picardo wrote: post #80
adamauckland wrote: post #71

Earlier quoted context omitted.

Where did I say multiple inspectors are inspecting a tree? There might indeed be multiple people dealing with a tree. The tree owner, an inspector, a tree surgeon, the local authorities (if the trees are protected). Synchronising data at row level when offline becomes online is extremely hard to get right and reliable. With the Event Sourcing model, every new piece of data gets thrown into the bucket , with each new…

> Synchronising data at row level when offline becomes online is extremely hard to get right and reliable. Why? Are the users in different time zones? If the time on their devices are synchronized, why is it hard to get right? I'm not trying to pick an argument with you, but trying to figure out if you've considered all the possible alternatives.

For a start, you're going to have to define your schema. Does one tree have one row or is it one case has one row?

By updating at row level, you've instantly lost the ability to tell the user what time this data changed, unless you're throwing everything in an audit table and at this point you're already halfway to Event Sourcing.

ES gives us a convenient way to throw a pile of information over time and not have to worry about defining the final schema up front, we can do that when constructing the views of aggregates.

Re: Event Sourcing Is Hard (2019)

#83

Post by minimeme »

mpfundstein wrote: post #58
minimeme wrote: post #48

Earlier quoted context omitted.

Event sourcing can be a very powerful pattern if used correctly. You don't need to combine ES with Eventual Consistency. ES can be implemented in a totally synchronous manner and it works very elegantly capturing nicely all the busines events you need. You get a very detailed audit for free and you don't loose importan business data. You can travel back in time, construct different views of data (projections) etc. Mo…

you sound like a consultant. few questions: - what was your biggest ES system that you worked on? - how many people worked on it? - how did the ES system particulary solve your problem? - what was the tech stack? Thanks

No, I'm not a consultant. Maybe lead developer is the most accurate title for me:) By what criteria the biggest system do you mean? They certainly weren't toy projects, these are real systems in active use solving real world problems for thousands of users. The ES part of these systems is mainly implemented in .NET using excellent Marten DB as ES store on top of PostgreSQL. I would say that ES changes drastically how you model things and see business. It forces you to identify and define meaningful events. These are often actually something that also non programmers understand so it also improves creatly communication with your clients(or in DDD terms domain experts:)) as a by-product. Scaling also has not been a real issue, these systems can handle thousands of operations per second without exotic hardware and setups, all this mostly synchronously.

Re: Event Sourcing Is Hard (2019)

#84

Post by datavirtue »

nocture wrote: post #13

Lots of hype about event sourcing and CQRS, usually in combination. 99% won't need it. Has the potential to ruin your team and even company unless you have a well-above-average engineering team.

This. Why do people try unproven tech in a business? A well-above-average engineering team isn't going to go for this. I would leave before this shit show even got started. The problem should be begging for this pattern if it fits. Like those mentioning the use of ES for high speed stock trading, for example.

One of my unwritten requirements for any system is being able to drop in inexperienced devs for maintenance...after I leave. Most people are likely to shoot themselves in the foot by not understanding (or even being aware of) database isolation levels much less grokking an event pipeline.

Re: Event Sourcing Is Hard (2019)

#85

Post by picardo »

adamauckland wrote: post #82
picardo wrote: post #80

Earlier quoted context omitted.

> Synchronising data at row level when offline becomes online is extremely hard to get right and reliable. Why? Are the users in different time zones? If the time on their devices are synchronized, why is it hard to get right? I'm not trying to pick an argument with you, but trying to figure out if you've considered all the possible alternatives.

For a start, you're going to have to define your schema. Does one tree have one row or is it one case has one row? By updating at row level, you've instantly lost the ability to tell the user what time this data changed, unless you're throwing everything in an audit table and at this point you're already halfway to Event Sourcing. ES gives us a convenient way to throw a pile of information over time and not have to w…

> For a start, you're going to have to define your schema. Does one tree have one row or is it one case has one row?

It seems to me you're defining a schema regardless, but with event sourcing your schema is embedded in your events rather than your database. And you're putting off the reconciliation to projection phase. I get it. But you still need to worry about writing transformations for your data to bring it up to date with your latest schema... That is often the most frustrating part.

Re: Event Sourcing Is Hard (2019)

#86

Post by oskar_dudycz »

nhumrich wrote: post #79
oskar_dudycz wrote: post #65

Earlier quoted context omitted.

I think that they were just building the stale read models, and used them as the write model, which created the whole confusion. Regarding the sharing events between module, it's one of the most common and the most dangerous mistakes. It's a leaking abstraction that will eventually create a distributed monolith. It has only downsides of monolith and microservices, without the upsides. I wrote longer on the topic of i…

What you are talking about is CQRS, which is a very valid pattern, and pairs well with event sourcing, but is not necessary part of event sourcing. You don't have to split your read and write models for events sourcing.

Could you elaborate, I think that I'm missing your point.

Re: Event Sourcing Is Hard (2019)

#87

Post by oskar_dudycz »

nhumrich wrote: post #81
oskar_dudycz wrote: post #61

Earlier quoted context omitted.

Could you provide an exact quote? I haven't been able to find any usage of "populate". I found: `the raw event stream subscription setup kills the ability to locally reason about the boundaries of a service.` `you have to talk to the people who will be consuming the events you produce to ensure that the events include enough data for the consuming system to make a decision` `wire a fleet of services together via an e…

You seem to be saying that event sourcing and event streaming are mutually exclusive. In almost all cases of event sourcing I have seen, event sourcing is event streaming, but more. When you say event sourcing, where do other services source from, if there is no stream? Yes, event sourcing typically uses a database instead of a real "queue", but it still functions like a queue. You still need consumers to subscribe t…

They're not mutually exclusive, but they're orthogonal. They're different patterns that happen to integrate with each other. Event Sourcing is about durable state stored and read as event, Event Streaming is about moving events from one place to another.

Event stores can integrate with the streaming platforms, to publish events and move them forward. Most of them have even built-in implementation of the Outbox Pattern that allows to publish events. They're usually called subscriptions. They can be used to build read models or triggers other flows, or route events to streaming platforms.

The essential point is that for read models, you don't expect to have the same guarantees as for write models. It's fine to be facing idempotency, even out of order, as read models shouldn't be used (in general) for business logic validation. In Event Streaming, or models are stale. In Event Sourcing, write model is not stale, you can be sure that you're making a decision on the current state. Thus, most of the points mentioned in the article, are just from struggles to use event streaming solution as tools for storing events. Which is not the issue of Event Sourcing or event stores per se, but using the wrong tool for the job.

Re: Event Sourcing Is Hard (2019)

#88

Post by valw »

alecco wrote: post #4

Event Sourcing sucks. I blame OOP abstractions taken to an extreme. https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...

I wrote the blog post you cited (thanks!) but I disagree with both statements: that is not what is meant in the article.

1. I don't think Event Sourcing sucks - I think we are lacking accessible technology for supporting it. 2. For most difficulties encountered in Event Sourcing, I would rather blame distribution taken to the extreme.

Re: Event Sourcing Is Hard (2019)

#89

Post by bob1029 »

DrBazza wrote: post #68
bob1029 wrote: post #67

Earlier quoted context omitted.

I am currently looking at a new iteration of our product that uses an event log as the principal data store. I agree that if we start from "zero" and don't even provide the option to build things the wrong way it would go smoothly. "everything is an event" is very powerful if its not just some add-on policy. For us, the entire working set would be able to live in memory. Larger objects would be left as pointers to th…

> Snapshots of working set would be taken on a daily basis in hopes of keeping recovery under ~5 minutes during service restarts or updates. Exactly this. We took hourly snapshots (for a trading system), and could restart the system in a second or so. Another nice side effect of an event driven system is that you start to think about what other data you can put in there. And then you realise you don't necessarily nee…

> And then you realise you don't necessarily need a database any more

This is precisely where we want to be.

We are already exceptionally minimal when it comes to database usage - SQLite is all we use today. If we are in a place where 100% of the business facts go to an immutable/append-only log, we can handle the entire persistence mechanism with a simple filestream write command. Not having to worry about mutating prior data solves a large portion of database engine problems automagically.

Re: Event Sourcing Is Hard (2019)

#90

Post by oskar_dudycz »

nhumrich wrote: post #78
dm3 wrote: post #60

Earlier quoted context omitted.

They were probably also sourcing their state from the events. However most of their problems come from sharing the events between the modules/services which is not a part of Event Sourcing.

I am confused when you say sharing events isn't part of event sourcing. How does a service populate it's state from other services event source if it can't access it's events?

Did you read my article that I linked, or my initial explanation of the Event Sourcing flow?
Post reply on HN