Live data from Hacker News

Combining event sourcing and stateful systems

stitcher.io

31–40 of 41 posts

Re: Combining event sourcing and stateful systems

#31
post #28

Earlier quoted context omitted.

Is (2) motivated by using aggregates which do not commute or by trying to do distributed modifications on a single value? One common technique if you have commutative aggregates is to have each writer just write to their own spot and then do a range query on read to re-join. If your aggregates don’t commute this, of course, doesn’t work and you’re stuck in “single threaded” land. I do remember reading a paper that av…

What do you mean by commutative aggregates? My understanding is that an aggregate is state, whereas commutativity is a property between two operations on said state. Do you mean something like CRDTs?

I think what's meant is that the event stream can be reordered and give the same result.

This isn't necessarily an abuse of terminology - I believe if we look at an operator that takes sequences of events and appends them, what we're talking about is in fact commutativity of that operator (in terms of the impact on the system).

Re: Combining event sourcing and stateful systems

#32
post #28

Earlier quoted context omitted.

Is (2) motivated by using aggregates which do not commute or by trying to do distributed modifications on a single value? One common technique if you have commutative aggregates is to have each writer just write to their own spot and then do a range query on read to re-join. If your aggregates don’t commute this, of course, doesn’t work and you’re stuck in “single threaded” land. I do remember reading a paper that av…

What do you mean by commutative aggregates? My understanding is that an aggregate is state, whereas commutativity is a property between two operations on said state. Do you mean something like CRDTs?

I'm not familiar with CRDTs, though I'll look into them -- thanks for the pointer. To clarify my original idea/question, the following example:

Yeah sure, imagine you're trying to compute count of how many times you've seen something. This is a SUM aggregate.

I assume the reason the author said "Ensuring aggregates are essentially single-threaded entities is a must" was because they are mutating a single key, if two different processes try to change the same value, you get inconsistent state.

An example of this would be:

Current K/V Pair: "A" -> 2

Modifier 1: Reads "A", gets 2

Modifier 2: Reads "A", gets 2

Modifier 1: Writes "A" -> 3

Modifier 2: Writes "A" -> 3

This means our sum is now inconsistent, as we would expect the value to be 4 in a single-threaded system.

However, because we are doing a sum, we can use commutativity to remove this conflict. Instead of each writer trying to write to a single key, you might make a compound key (Id, )

Current Value Pairs: [("A", "Modifier 1") -> 1, ("A", "Modifier 2") -> 1]

Modifier 1: Reads ("A", "Modifier 1"), gets 1

Modifier 2: Reads ("A", "Modifier 2"), gets 1

Modifier 1: Writes ("A", "Modifier 1") -> 2

Modifier 2: Writes ("A", "Modifier 2") -> 2

Then, a reader could just ask for all the keys with the prefix "A" (This is the range query). So a reader gets back (2, 2), which then can now merge into 4 as SUM is a commutative operation. Because the reader is taking care of the final aggregation step, there's no concurrency conflict so you can get away with having any number of writers as long as the read and read-side computation is cheap enough.

Some aggregates are commutative only in certain forms, like AVERAGE. To explain further, if one writer says the AVERAGE is 5 and another says it is 7, I can't combine those to say that the global average is 6.

However, if each writer stores both SUM and COUNT, I can use (COUNT1 + COUNT2)/(SUM1 + SUM2). This is because division doesn't commute, so I have to delay the non-commutative operation until the reader if I want to be able to merge two data sources.

Edited: Formatting

Re: Combining event sourcing and stateful systems

#33

I'm working with an event sourced system and we did some mistakes in the process of the design, so some areas that didn't need event sourcing do have it. The biggest downside has been the UI: events are not real time and these objects are just CRUD stuff, so the user wants to see that you saved what he has just written. You might not have this information yet, so you need to mitigate it, for example updating the UI t…

This is super important and I cannot stress it enough! If your events contain words like, "create, update, delete, associate, disassociate," then you're building a weak domain model that won't benefit from the added complexity of deriving state from the source of events. Your events should use the same words your customer would actually use to describe their business process. For example, a system to manage intake of…

While this is correct for a LOT of applications, if you ever end up in a situation where multiple automated systems and manual users may be concurrently updating and associating/disassociating records, and last-write-wins won't cut it (due to e.g. some fields being co-dependent) you're going to need to merge their change streams in a custom way. "UpdatedRecordData(Type, ID, Value, Actor, Context, Date)" may not be a domain-specific vocabulary but it's kind of necessary if you're building a system with a data model that can be extended by clients. Event-Sourced Salesforce will be a thing!

Re: Combining event sourcing and stateful systems

#34

Earlier quoted context omitted.

Reactors can keep their own state including their current position in the event stream. When a replay is initiated it ignores events older than it's current "head."

What happens when the server that holds the thing that holds that state is restarted?

That's an important question to ask!

Can you assume the reactor has durable, local storage with atomic transactions?

The answer is to model your design and use a sufficient level of rigour in validating that your system meets your requirements.

Maybe you could use a database server that has the right properties to ensure your reactor could survive a restart.

What if you want to add multiple reactors so that you can process an event stream with a high volume of events, faster?

Re: Combining event sourcing and stateful systems

#35
post #28

Earlier quoted context omitted.

What do you mean by commutative aggregates? My understanding is that an aggregate is state, whereas commutativity is a property between two operations on said state. Do you mean something like CRDTs?

I'm not familiar with CRDTs, though I'll look into them -- thanks for the pointer. To clarify my original idea/question, the following example: Yeah sure, imagine you're trying to compute count of how many times you've seen something. This is a SUM aggregate. I assume the reason the author said "Ensuring aggregates are essentially single-threaded entities is a must" was because they are mutating a single key, if two…

I see. I think you may be using a different definition of "aggregate" than the article and OP are referring to. Your "aggregate" is that from database systems like SQL and Excel, while the OP's "aggregate" is a different concept taken from the lexicon of Domain-Driven Design.

In DDD, an "aggregate" is a collection of state that can only be changed atomically, as a whole, by the outside world. It helps to think concretely of an Actor, i.e. some entity that owns a collection of state, such that the only way to mutate that state is to send a message to the actor. No matter how many messages come in from the outside world, only one is being handled at a time by the entity, and there are no concerns of concurrent access. Necessarily though, all messages are handled in a linear order.

From Martin Fowler's article on aggregates [0]:

> Aggregates are the basic element of transfer of data storage - you request to load or save whole aggregates. Transactions should not cross aggregate boundaries.

I think you'll be interested in CRDTs. They're like DDD aggregates where mutation operations are essentially always commutative, so there's a bit more alignment between the senses of "aggregate" being confused here.

[0] https://www.martinfowler.com/bliki/DDD_Aggregate.html

Re: Combining event sourcing and stateful systems

#36
post #28

Earlier quoted context omitted.

What do you mean by commutative aggregates? My understanding is that an aggregate is state, whereas commutativity is a property between two operations on said state. Do you mean something like CRDTs?

I think what's meant is that the event stream can be reordered and give the same result. This isn't necessarily an abuse of terminology - I believe if we look at an operator that takes sequences of events and appends them, what we're talking about is in fact commutativity of that operator (in terms of the impact on the system).

You're right, you can line the two concepts up to some extent. (I mentioned CRDTs because they're one way to do that in a particularly nice way.) But given the context of the question, it seemed like teasing out the difference would be helpful.

Re: Combining event sourcing and stateful systems

#37

Earlier quoted context omitted.

It took several hours of individual research, watching talks, reading blog posts; and took several pair-programming sessions of several hours over the span of four weeks to come up with a solution we liked. We informed our client that this was a new area for us and that we didn't have hands-on experience with, but that we believed it would be beneficial to spend time to explore it, as it would be an elegant solution…

How do you approach estimating effort for this sort of thing? I find it awkward enough in Scrum to guess up front how many days of effort research will take and commit to delivering a plan or design by the end. If you have clients and aren't strictly bound by someone else's framework, they still want some rough idea how long research will take. Especially if the client is footing the bill. If the research is un-bille…

Everything starts with trust, of course. We've proven ourselves in several large projects and within our open source community before. The client knows that.

There's never any estimate of "so much hours will be spent in total on this research", we just honestly communicate with the client along the way, and they give us their trust.

Whether we can keep that trust is up to us.

Re: Combining event sourcing and stateful systems

#38

You should take a look at Microsoft's Durable Functions which pairs event sourcing + (optional) actor model + serverless. It's some pretty neat tech. I tried doing something similar to this several years ago, and here's a few issues I ran into: 1. Pub/sub in Event Sourcing is a bad idea. It's really hard to get right. (what to do if sub happens after pub due to scaling issues/infrastructure, etc?) Instead it's better…

> Pub/sub in Event Sourcing is a bad idea I find this point surprising. I would say the exact opposite. I would say that pub/sub and event sourcing are two sides of the same coin: events. > what to do if sub happens after pub That should only ever be a problem with a non-durable transport that doesn't have serialized writes per topic. Which, admittedly, can be pretty common. But it's not so much an event sourcing or…

> I find this point surprising. I would say the exact opposite. I would say that pub/sub and event sourcing are two sides of the same coin: events.

I meant in the context of getting it right. I didn't experiment with all the pub/sub systems at the time, but most I experimented with would lose data in a catastrophic event and cause inconsistencies. This was several years ago though.

Re: Combining event sourcing and stateful systems

#39
post #24
post #22

Earlier quoted context omitted.

I'm really interested to understand your comment better. Can you give an example for "presumptive view of a division of responsibilities" and generally the whole comment? Something like "bad way" vs "good way"? Thanks!

It's abstract, but I'll try to get something down. First, look at what happens to the system from the outside, say a web request that leads to a web response. In between, information is gathered from other areas (databases, program logic) and combined with the request data. There are also possibly other effects generated (writes to database state, messages to other users, etc.). Now take all of those “effects”--the w…

It is the same as arguing whether lambda calculus is better than pi-calculus or a Turing machine.

These are all isomorphic structures. Neither of them can do more than the other.

For example - you’re speaking of dependencies, etc - but any language based on statements can be reduced to a dependency graph defined by it’s single-assignment form.

Event sourcing is not a panacea.

Re: Combining event sourcing and stateful systems

#40

Earlier quoted context omitted.

> Pub/sub in Event Sourcing is a bad idea I find this point surprising. I would say the exact opposite. I would say that pub/sub and event sourcing are two sides of the same coin: events. > what to do if sub happens after pub That should only ever be a problem with a non-durable transport that doesn't have serialized writes per topic. Which, admittedly, can be pretty common. But it's not so much an event sourcing or…

> I find this point surprising. I would say the exact opposite. I would say that pub/sub and event sourcing are two sides of the same coin: events. I meant in the context of getting it right. I didn't experiment with all the pub/sub systems at the time, but most I experimented with would lose data in a catastrophic event and cause inconsistencies. This was several years ago though.

> most I experimented with would lose data in a catastrophic event and cause inconsistencies

Fair enough. Those are probably message buses or message queues that are ephemeral transports. Since event sourcing is predicated upon permanent storage of events, there's no way to lose events that have already been committed (unless someone actually physically deletes the events).

Post reply on HN