Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What they don’t tell you about event sourcing

medium.com

Post reply on HN
Print view View original

71–80 of 81 posts

Re: What they don’t tell you about event sourcing

#71

Post by ht85 »

hesdeadjim wrote: post #57
Yokohiii wrote: post #47

Earlier quoted context omitted.

Idk why people are saying that ES data is always immutable. They can be by default sure, but if a facility is useful to change the history, why not?

If you need to change history you can just create new events that accomplish your mutation -- and even mark them as a type 'change history' or some such obvious identifier so when you inspect the event stream you know exactly what you are looking at.

To avoid having to implement the whole logic for any event you'd want to revert, you can have a "cancel" event that points to another event.

You'd read the cancel events first and simply skip the events they point to.

Re: What they don’t tell you about event sourcing

#72

Post by jacques_chester »

hesdeadjim wrote: post #57
Yokohiii wrote: post #47

Earlier quoted context omitted.

Idk why people are saying that ES data is always immutable. They can be by default sure, but if a facility is useful to change the history, why not?

If you need to change history you can just create new events that accomplish your mutation -- and even mark them as a type 'change history' or some such obvious identifier so when you inspect the event stream you know exactly what you are looking at.

> If you need to change history you can just create new events that accomplish your mutation -- and even mark them as a type 'change history' or some such obvious identifier so when you inspect the event stream you know exactly what you are looking at.

"Right to be forgotten" isn't the same as "right to be redacted".

Re: What they don’t tell you about event sourcing

#73

Post by mianos »

hesdeadjim wrote: post #57
Yokohiii wrote: post #47

Earlier quoted context omitted.

Idk why people are saying that ES data is always immutable. They can be by default sure, but if a facility is useful to change the history, why not?

If you need to change history you can just create new events that accomplish your mutation -- and even mark them as a type 'change history' or some such obvious identifier so when you inspect the event stream you know exactly what you are looking at.

I agree, this is 100% normal in accounting, as the earlier thread pointed out. If there is an error you add journal entries to the end to make the adjustment. It is funny that this sort of thing was invented in accounting in 1494.

Re: What they don’t tell you about event sourcing

#74

Post by paragraft »

likeclockwork wrote: post #70
paragraft wrote: post #66

Earlier quoted context omitted.

Conflict resolution is a separate problem that isn't addressed by Eventual Consistency. If I make a write in an EC system, that write may eventually be accepted, it may be rejected, or it could be resolved through something smarter like a CRDT. EC just says that I won't know that immediately; that different parts of the system can have different views of the truth at the same time. And for most business systems that'…

Thanks for answering. Is Eventual Consistency a necessary property of a CQRS/Event Sourcing architecture?

I don't think it is, but—as usual with EC—if you don't want it there's a performance cost. The way you'd avoid it in a model where writes go into one queue/connection and reads come out somewhere else is when you'd do a write you'd have to block and wait on that write being acknowledged on the reader side (or you build in some sort of side channel to whatever your write handler is so it can tell you when a write has been accepted, if you're doing conflict resolution on the write side (some systems don't and push it off to read-time)).

But at that point you're just simulating something you probably already had before going to CQRS/ES, so you would only want to do that in very select cases. Otherwise there's really no point to the architecture...

Re: What they don’t tell you about event sourcing

#75

Post by solipsism »

jandrewrogers wrote: post #67
einrealist wrote: post #61

Earlier quoted context omitted.

You can encrypt events and throw away the keys, if data should be made inaccessible. Of course, it adds complexity. But its already being done.

This does not work for many data models, for both technical and economic (e.g. increasing costs by multiple orders of magnitude) reasons. Many real systems would require hundreds of millions of active encryption keys, encrypting data that is smaller than the encryption block size. Every database architecture that exists today is designed with the deep assumption that scalable fine-grained deletion will never be requi…

Every database architecture that exists today is designed with the deep assumption that scalable fine-grained deletion will never be required, largely because we don't have good computer science for how to do it

Can you explain this? Are you talking about something different from deleting individual rows?

Re: What they don’t tell you about event sourcing

#76

Post by hvidgaard »

sbov wrote: post #64
joefreeman wrote: post #10

Good article. I've spent the last year migrating to an event sourced system, so thought I'd share some thoughts. On the eventual consistency point, I've found you can get quite far with having the read model managing the race condition. This probably doesn't work everywhere, but in our system, multiple users can accept an invitation, so we have something like `InvitationAccepted{invitation_id, user_id}`. It's possibl…

> It's up to the read model to ask, 'has this invitation already been accepted?' I feel like you've skipped over the interesting part of your strategy here. If it's an eventually consistent system, what keeps the read model from having the wrong answer to this question?

Not OP, but I am working with a CQRS system.

In CQRS eventual consistency does not mean that we have multiple servers such that we have 2 servers with 2 different answers. It means from a command is issued to an event is propagated to all read models, there is a delay.

You need to handle the race condition at some point or another. From a CQRS point of view, a user accepting an invitation is just an event like any other event. What happens based on that event must account for the possibility that multiple users have accepted, and it's a rather straight forward thing to solve with an ES. The "accept" event with the lowest sequence number is the first.

Having the read side handle it would probably mean that when you have a read model for accepted invitations, you ignore all but the first accepted of an "invitationId".

Re: What they don’t tell you about event sourcing

#77

Post by hvidgaard »

jmull wrote: post #44
vorpalhex wrote: post #32

Earlier quoted context omitted.

That's not "Command Query Responsibility Segregation" (CQRS). That's modeling your data as a time series - which is a totally valid and perfectly useful model in many cases, but has nothing to do with the architectural pattern known as CQRS. Martin Fowler gives the following simple definition of CQRS: > At its heart is the notion that you can use a different model to update information than the model you use to read…

But what is the distinction between what Fowler describes and what I describe? There's not really a contradiction with what you describe either. I have a distinct write store and read store, with very different models. As you say, the write store is the source of truth. Since the read store is updated synchronously with the write store there's no need for a queue between them. Indeed there are also multiple projectio…

You're right. Nothing about CQRS demands any kind of asynchronization or distributed system. It is quite simply a system where you have different models for updating and querying the system. You don't even need to have an event store for it to be a CQRS system, but it makes so much sense that I have a hard time imaging using CQRS without and ES of some kind.

Re: What they don’t tell you about event sourcing

#78

Post by hvidgaard »

Yokohiii wrote: post #49
theptip wrote: post #46

Earlier quoted context omitted.

I don't think you need a separate queue; if you have an "Events" table then you can just write everything there. It solves the consistency problem because you can create your event inside a transaction, which will rollback if another event touching the same source is created simultaneously. E.g. if you have these incompatible events in a ledger: CreditAccount(account_id=123, amount=100) DebitAccount(account_id=123, a…

If your idea in the example is that the second "debit" is created by another transaction while your transaction is in progress, then this will not work out. Firstly it requires a dirty read, which is nothing I would rely on in a transaction. Secondly, if the dirty read works, assuming the outcome of several rows is just a read operation, which forces you to rollback on the client and still leaves a window for inconsi…

No matter how you model things and no matter what technology you use, a race condition like this needs to be handled one way or another. You either handle it such that the data is always consistent, or you handle inconsistent data.

You can use an SQL database with transaction and locking to ensure that you will never debit money that isn't there. Or you can save commands in a queue that only a single process at any given time (that incidentally includes the SQL scenario). Or you can use a distributed consensus algorithm with multiple stage commits. There is no way around it.

Re: What they don’t tell you about event sourcing

#79

Post by marxama »

theptip wrote: post #40

I don't see much discussion of event-sourcing simply using a SQL database (i.e. skipping the CQRS part). This would allow you to keep your CP (strongly-consistent) semantics. While this clearly wouldn't work in high-volume cases (i.e. where you _actually_ need CQRS), it seems like this would be the simplest option for many systems. I see a lot of articles advocating for immediately jumping into CQRS, which seems like…

I went with this approach on a recent project, for two reasons: * Tracing/history/auditing * Structuring the service around these events, it became trivial to add new "event types", rather than expanding some big hairy PATCH endpoint or similar. In other words, when I needed a user entity to be able to belong to a group, I just added a new event type, "user/join-group". Having a single endpoint for all events had two other nice benefits: batching became trivial, as well as doing several events transactionally (all events in one requests are processed in one transaction).

It's been running in production for a couple of months now, and it's been working great! The main drawback I can think of was that it obviously didn't work well with Swagger out-of-the-box, had to do some custom handling for showing all the available event types.

Re: What they don’t tell you about event sourcing

#80

Post by jandrewrogers »

solipsism wrote: post #75
jandrewrogers wrote: post #67

Earlier quoted context omitted.

This does not work for many data models, for both technical and economic (e.g. increasing costs by multiple orders of magnitude) reasons. Many real systems would require hundreds of millions of active encryption keys, encrypting data that is smaller than the encryption block size. Every database architecture that exists today is designed with the deep assumption that scalable fine-grained deletion will never be requi…

Every database architecture that exists today is designed with the deep assumption that scalable fine-grained deletion will never be required, largely because we don't have good computer science for how to do it Can you explain this? Are you talking about something different from deleting individual rows?

A "row" is a logical abstraction. It doesn't exist as a physical thing in many database systems, especially modern ones.

Furthermore, "delete" is commonly defined as "will not be returned in a future selection operation" -- there is no implication that any data is physically deleted and permanently inaccessible. Avoiding physical deletion is done for very good technical reasons to support features and performance that everyone is accustomed to in a database.

Post reply on HN