61–70 of 81 posts
Earlier quoted context omitted.
Basically implement double entry accounting. I think ideally you want something like Rich Hickey speaks about when he speaks of Datomic. An append only database. You can see what the previous values for that row were, along with schema changes.
I worry about GDPR with respect to Kafka and other append-only databases.
So, where does that happen? When getting written into the read model? Then what, it emits an event saying clearly that the previous action failed? All while there's a client waiting for results?
I'd love to see a worked example based on discrete resources. Such as two people, a box, and a ball; where the ball can be held by either person or in the box, and a person can take the ball from the box but not from another person.
I find the concept intriguing but I don't really get it and I haven't been able to identify in any of the writing where "the buck stops".
Good article. I've spent the last year migrating to an event sourced system, so thought I'd share some thoughts. On the eventual consistency point, I've found you can get quite far with having the read model managing the race condition. This probably doesn't work everywhere, but in our system, multiple users can accept an invitation, so we have something like `InvitationAccepted{invitation_id, user_id}`. It's possibl…
Good article. I've spent the last year migrating to an event sourced system, so thought I'd share some thoughts. On the eventual consistency point, I've found you can get quite far with having the read model managing the race condition. This probably doesn't work everywhere, but in our system, multiple users can accept an invitation, so we have something like `InvitationAccepted{invitation_id, user_id}`. It's possibl…
I feel like you've skipped over the interesting part of your strategy here. If it's an eventually consistent system, what keeps the read model from having the wrong answer to this question?
If you do even the most basic research into Event Sourcing you will find people warning of these trade offs (foremost amongst them Greg Young, the person who originally set out the idea)
I've never quite been able to get my head all the way around "Eventual Consistency". I don't understand how actions that could conflict or resources that could be contended are supposed to work? At some point, something has to say, given two actions A and B that are in conflict, the later action B must fail. So, where does that happen? When getting written into the read model? Then what, it emits an event saying clea…
Earlier quoted context omitted.
I worry about GDPR with respect to Kafka and other append-only databases.
You can encrypt events and throw away the keys, if data should be made inaccessible. Of course, it adds complexity. But its already being done.
Every database architecture that exists today is designed with the deep assumption that scalable fine-grained deletion will never be required, largely because we don't have good computer science for how to do it. As experienced database operations people know, if you are required to do this kind of delete then the sane way is to completely rebuild your storage with the data to be deleted filtered out, if you have ample excess capacity -- it is often much faster than editing the existing storage. For some large-scale systems, there is no plausible solution.
This is an interesting computer science challenge -- a database kernel designed for efficient deletion -- that I've thought a lot about over the last year dealing with GDPR compliance. It definitely isn't a thing that exists today and encryption doesn't help.
Wait, event sourcing / immutable data doesn't scale.
I was doing event sourcing in 2010, and loved it. It was incredible. But by the time I started hearing people call it "event sourcing" I had already moved on to:
State-based, graph CRDTs.
They combine the best of event sourcing with the best of distributed state replication, and are super scalable!
Now even the Internet Archive[1] is running it (in 2014 I implemented it into a library that they are now using - https://github.com/amark/gun )
The last section on Operational Flexibility and the inability to change the event history raises a very good point. Like most of the issues, the solution requires experience to know when you are at the Goldilocks point (Just Right). This specific issue has a lot in common with managing database migrations in django or any other migration system. The ideal situation is to create migrations that can always be rolled ba…
Idk why people are saying that ES data is always immutable. They can be by default sure, but if a facility is useful to change the history, why not?
I've never quite been able to get my head all the way around "Eventual Consistency". I don't understand how actions that could conflict or resources that could be contended are supposed to work? At some point, something has to say, given two actions A and B that are in conflict, the later action B must fail. So, where does that happen? When getting written into the read model? Then what, it emits an event saying clea…
Conflict resolution is a separate problem that isn't addressed by Eventual Consistency. If I make a write in an EC system, that write may eventually be accepted, it may be rejected, or it could be resolved through something smarter like a CRDT. EC just says that I won't know that immediately; that different parts of the system can have different views of the truth at the same time. And for most business systems that'…