Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dual Nature of Events in Event-Driven Architecture

reactivesystems.eu

Post reply on HN
Print view View original

41–50 of 66 posts

Re: The Dual Nature of Events in Event-Driven Architecture

#41

Post by chipdart »

candiddevmike wrote: post #9
williamdclt wrote: post #7

Earlier quoted context omitted.

> each consumer can perform their own targeted lookups as needed that puts you into tricky race condition territory, the data targeted by an event might have changed (or be deleted) between the time it was emitted and the time you're processing it. It's not always a problem, but you have to analyse if it could be every time. It also means that you're losing information on what this event actually represents: looking…

There should be some law that says strictly serialized process should never be broken into discreet services. Distributed locks and transactions are hell.

> Distributed locks and transactions are hell.

Which distributed transaction scenario have you ever dealt with that wasn't correctly handled by a two-phase commit or at worst a three-phase commit?

Re: The Dual Nature of Events in Event-Driven Architecture

#42

Post by chipdart »

gunnarmorling wrote: post #20
marcosdumay wrote: post #10

Earlier quoted context omitted.

Your event data must not be mutable. That's kind of the first rule of any event-based system. It doesn't really matter the architecture, if you decide to name the things "event", everybody's head will break if you make them mutable. If you decide to add mutation there in some way, you will need to rewrite the event stream, replacing entire events.

It's not about mutability of events, but about mutating the underlying data itself. If the event only says "customer 123 has been updated", and a consumer of that event goes back to the source of the event to query the full state of that customer 123, it may have been updated again (or even deleted) since the event was emitted. Depending on the use case, this may or may not be a problem. If the consumer is only inter…

> (...) and a consumer of that event goes back to the source of the event to query the full state of that customer 123, it may have been updated again (or even deleted) since the event was emitted.

So the entity was updated. What's the problem?

Re: The Dual Nature of Events in Event-Driven Architecture

#43

Post by hcarvalhoalves »

chipdart wrote: post #40
hcarvalhoalves wrote: post #39

The "produce a trigger event then have the consumer reach back to the producer and fetch data" can be an anti-pattern. You expose yourself to all kinds of race conditions and cache incoherency, then you start trying to fix with cache invalidation or pinning readers, and the result is you can't scale readers well. If you're a using a message queue, the message should convey necessary information such that, if all mess…

> If you're a using a message queue (...) Events and messages are entirely different things. They might look similar, but their responsibilities are completely different. The scenario you're describing matches the usecases for messages, not events.

The situation of messages vs events is analogous to append-only database vs update-in-place database. You get exposed to the same issues at scale if you rely on the later.

Being notified only _when_ something happened isn't always useful the world is changing underneath you (it _can be_ useful in particular situations, when you know state is final, but not as a general architecture principle).

Re: The Dual Nature of Events in Event-Driven Architecture

#44

Post by hcarvalhoalves »

withinboredom wrote: post #11
candiddevmike wrote: post #9

Earlier quoted context omitted.

There should be some law that says strictly serialized process should never be broken into discreet services. Distributed locks and transactions are hell.

The best way to avoid distributed locks and transactions is to manually do the work. For example, instead of doing a distributed lock on two accounts when transferring funds, you might do this (which is the same as a distributed transaction, without the lock): 1. take money from account A 2. if failed, put money back into account A 3. put money into account B 4. if failed, put money back into account A In other words…

Here's how you can do this. You have 3 accounts: A, B and in-flight.

1. Debit A, Credit in-flight.

2. Credit B, Debit in-flight.

If 1. fails, nothing happened and everything is consistent.

If 2. fails, you know (because you have money left on in-flight), and you can retry later, or refund A.

This way at no point your total balance decreases, so everything is always consistent.

Re: The Dual Nature of Events in Event-Driven Architecture

#45

Post by chipdart »

hcarvalhoalves wrote: post #43
chipdart wrote: post #40

Earlier quoted context omitted.

> If you're a using a message queue (...) Events and messages are entirely different things. They might look similar, but their responsibilities are completely different. The scenario you're describing matches the usecases for messages, not events.

The situation of messages vs events is analogous to append-only database vs update-in-place database. You get exposed to the same issues at scale if you rely on the later. Being notified only _when_ something happened isn't always useful the world is changing underneath you (it _can be_ useful in particular situations, when you know state is final, but not as a general architecture principle).

> The situation of messages vs events is analogous to append-only database vs update-in-place database. You get exposed to the same issues at scale if you rely on the later.

Not really. Messages vs events is a foundational design trait whose discussion involves topics such as adopting distinct messaging patterns such as message queues or pub-sub. They have completely different responsibilities and solve completely different problems.

Re: The Dual Nature of Events in Event-Driven Architecture

#46

Post by hcarvalhoalves »

chipdart wrote: post #45
hcarvalhoalves wrote: post #43

Earlier quoted context omitted.

The situation of messages vs events is analogous to append-only database vs update-in-place database. You get exposed to the same issues at scale if you rely on the later. Being notified only _when_ something happened isn't always useful the world is changing underneath you (it _can be_ useful in particular situations, when you know state is final, but not as a general architecture principle).

> The situation of messages vs events is analogous to append-only database vs update-in-place database. You get exposed to the same issues at scale if you rely on the later. Not really. Messages vs events is a foundational design trait whose discussion involves topics such as adopting distinct messaging patterns such as message queues or pub-sub. They have completely different responsibilities and solve completely di…

I don't see how using pub-sub or not changes how you model the data. It should be orthogonal. Do you have a good example?

Re: The Dual Nature of Events in Event-Driven Architecture

#47

Post by hcarvalhoalves »

kaba0 wrote: post #18

I am going on a bit of a tangent here, but I always wondered, are those of you who use absolutely huge event-driven architectures, have you ever got yourself into a loop? I can't help but worry about such, as event systems are fundamentally Turing-complete, and with a complex enough system it doesn't seem too hard to accidentally send an event because A, which will eventually, multiple other events later again cases…

Yes, it happens. A way to deal with it is carrying some counter on the message metadata and incrementing it every time a consumer passes it along, so you can detect recursions. Another is having messages carry a unique id, and consumers record already seen messages.

Re: The Dual Nature of Events in Event-Driven Architecture

#48

Post by switch007 »

dkarl wrote: post #19

Events are published observations of facts. If you want to be able to use them as triggers, or to build state, then you have to choose the events and design the system to make that possible, but you always have to ensure that systems only publish facts that they have observed. Most issues I've seen with events are caused by giving events imprecise names, names that mean more or less than what the events attest to. Fo…

Such an important point about naming things accurately. It can be hard, so we often take the easy path

But bad names manifest as a multitude of problems much later on.

I wonder if this is an area LLMs can help us with because really a lot of us do struggle with it. I'm going to investigate!

Re: The Dual Nature of Events in Event-Driven Architecture

#49

Post by switch007 »

hcarvalhoalves wrote: post #47
kaba0 wrote: post #18

I am going on a bit of a tangent here, but I always wondered, are those of you who use absolutely huge event-driven architectures, have you ever got yourself into a loop? I can't help but worry about such, as event systems are fundamentally Turing-complete, and with a complex enough system it doesn't seem too hard to accidentally send an event because A, which will eventually, multiple other events later again cases…

Yes, it happens. A way to deal with it is carrying some counter on the message metadata and incrementing it every time a consumer passes it along, so you can detect recursions. Another is having messages carry a unique id, and consumers record already seen messages.

Do you consider it a requirement for every message?

Like, the problem sounds bad enough to warrant it. If not, now do you choose when to apply it?

Our architects have a habit of ignoring these kind of issues and when you suggest making things like this a requirement they accuse you of excessive concern!

Re: The Dual Nature of Events in Event-Driven Architecture

#50

Post by stevenalowe »

Thank you for this well-written, well-reasoned, and thoroughly enjoyable article!

Yet I am troubled by it, and must disagree with some of the premises and conclusions. Only you know your specific constraints, so my 'armchair architecting' may be way off target. If so, I apologize, but I was particularly disturbed by this statement:

"events that travel between services have a dual role: They trigger actions and carry data."

Yes, sort of, but mostly no. Events do not "trigger" anything. The recipient of an event may perform an action in response to the event, but events cannot know how they will be used or by whom. Every message carries data, but a domain event is specifically constrained to be an immutable record of the fact that something of interest in the domain has happened.

The notion of modeling 'wide' vs 'short' events seems to ignore the domain while conflating very different kinds of messages - data/documents/blobs, implementation-level/internal events, domain events, and commands.

Modeling decisions should be based on the domain, not wide vs short. Domain events should have names that are meaningful in the problem domain, and they should not contain extraneous data nor references to implementation details/concepts. This leads to a few suggestions:

* Avoid Create/Read/Update/Delete (CRUD) event names as these are generic implementation-level events, not domain events. Emit such events "under the hood" for replication/notification if you must, but keep them out of the domain model.

* Name the domain event specifically; CRUD events are generally undesirable because they (a) are an implementation detail and (b) are not specific enough to understand without more information. Beware letting an implementation decision or limitation corrupt the domain model. In this example, the BookingUpdated event adds no value/information, makes filtering for the specific event types more complex, and pollutes the domain language with an unnecessary and potentially fragile implementation detail (the name of the db table Booking, which could just as easily have been Reservation or Order etc). SeatSelected is a great domain event name for a booking/reservations system. BookingSeatSelected if there is further scope beyond Booking that might have similar event names. BookingUpdated is an implementation-level, internal event, not part of the problem domain.

* What data is necessary to accurately record this domain event? A certain minimal set of relevant data items will be required to capture the event in context. Trying to anticipate and shortcut the needs of other/future services by adding extraneous data is risky. Including a full snapshot of the object even more risky, as this makes all consumers dependent on the entire object schema.

* The notion of "table-stream duality" as presented is likewise troublesome, as that is an implementation design choice, not part of the domain model. I don't think that it is a goal worthy of breaking your domain model, and suggest that it should not be considered at all in the domain model's design. Doing so is a form of premature optimization :)

* That said, separating entity and event streams would keep table-stream duality but require more small tables, i.e. one domain event type per stream and another Booking stream to hold entity state as necessary. A Booking service can subscribe to SeatSelected et al events (presumably from the UI's back-end service) and maintain a separate table for booking-object versions/state. A SeatReserved event can be emitted by the Booking service, and no one has to know about the BookingUpdated event but replication hosts.

Thanks again for writing and posting this, it really made me think. Good luck with your project!

Post reply on HN