Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dual Nature of Events in Event-Driven Architecture

reactivesystems.eu

Post reply on HN
Print view View original

11–20 of 66 posts

Re: The Dual Nature of Events in Event-Driven Architecture

#11

Post by withinboredom »

candiddevmike wrote: post #9
williamdclt wrote: post #7

Earlier quoted context omitted.

> each consumer can perform their own targeted lookups as needed that puts you into tricky race condition territory, the data targeted by an event might have changed (or be deleted) between the time it was emitted and the time you're processing it. It's not always a problem, but you have to analyse if it could be every time. It also means that you're losing information on what this event actually represents: looking…

There should be some law that says strictly serialized process should never be broken into discreet services. Distributed locks and transactions are hell.

The best way to avoid distributed locks and transactions is to manually do the work. For example, instead of doing a distributed lock on two accounts when transferring funds, you might do this (which is the same as a distributed transaction, without the lock):

1. take money from account A

2. if failed, put money back into account A

3. put money into account B

4. if failed, put money back into account A

In other words, perform compensating actions instead of doing transactions.

This also requires that you have some kind of mechanism to handle an application crash between 2 and 3, but that is something else entirely. I've been working on this for a couple of years now and getting close to something really interesting ... but not quite there yet.

Re: The Dual Nature of Events in Event-Driven Architecture

#12

Post by [deleted] »

withinboredom wrote: post #11
candiddevmike wrote: post #9

Earlier quoted context omitted.

There should be some law that says strictly serialized process should never be broken into discreet services. Distributed locks and transactions are hell.

The best way to avoid distributed locks and transactions is to manually do the work. For example, instead of doing a distributed lock on two accounts when transferring funds, you might do this (which is the same as a distributed transaction, without the lock): 1. take money from account A 2. if failed, put money back into account A 3. put money into account B 4. if failed, put money back into account A In other words…

[deleted]

Re: The Dual Nature of Events in Event-Driven Architecture

#13

Post by candiddevmike »

withinboredom wrote: post #11
candiddevmike wrote: post #9

Earlier quoted context omitted.

There should be some law that says strictly serialized process should never be broken into discreet services. Distributed locks and transactions are hell.

The best way to avoid distributed locks and transactions is to manually do the work. For example, instead of doing a distributed lock on two accounts when transferring funds, you might do this (which is the same as a distributed transaction, without the lock): 1. take money from account A 2. if failed, put money back into account A 3. put money into account B 4. if failed, put money back into account A In other words…

> This also requires that you have some kind of mechanism to handle an application crash between 2 and 3, but that is something else entirely

Like a distributed transaction or lock. This is the entire problem space, your example above is very naive.

Re: The Dual Nature of Events in Event-Driven Architecture

#14

Post by jwarden »

bob1029 wrote: post #3

The triggering of the action is a direct consequence of the information an event contains. Whether or not an action is triggered should not be the responsibility of the event. If you are writing events with the intention of having them invoke some specific actions, then you should prefer to invoke those things directly. You should be describing a space of things that have occurred, not commands to be carried out. By…

Another architecture might be that the service responsible for Seat Selection emits a `SeatSelected` event, and another service responsible for updating bookings emits a `BookingUpdated(Reason: SeatSelected)` "fat" event. Same for `PaymentReceived` and `TicketIssued`.

Both events would "describe a space of things that occurred" as @bob1029 suggests.

The seat selection process for an actual airline probably needs to be more involved. @withinboredom recommends:

  - SeatTimeLimitedReserved {41A, 15m}
  - SeatAssignedTo {UserA}
  - SeatBooked {41A}
In which case, only SeatBooked would trigger a BookingUpdated event.

Re: The Dual Nature of Events in Event-Driven Architecture

#16

Post by bob1029 »

williamdclt wrote: post #7
bob1029 wrote: post #3

The triggering of the action is a direct consequence of the information an event contains. Whether or not an action is triggered should not be the responsibility of the event. If you are writing events with the intention of having them invoke some specific actions, then you should prefer to invoke those things directly. You should be describing a space of things that have occurred, not commands to be carried out. By…

> each consumer can perform their own targeted lookups as needed that puts you into tricky race condition territory, the data targeted by an event might have changed (or be deleted) between the time it was emitted and the time you're processing it. It's not always a problem, but you have to analyse if it could be every time. It also means that you're losing information on what this event actually represents: looking…

> an event might have changed (or be deleted) between the time it was emitted

Then I would argue it isn't a meaningful event. If some attributes of the event could become "out of date" such that the logical event risks invalidation in the future, you have probably put too much data into the event.

For example, including a user's preferences (e.g., display name) in a logon event - while convenient - means that if those preferences ever change, the event is invalid to reference for those facts. If you only include the user's id, your event should be valid forever (for most rational systems).

> your consumer has to query the service that dispatched the event

An unfortunate but necessary consequence of integrating multiple systems together. You can't take out a global database lock for every event emitted.

Re: The Dual Nature of Events in Event-Driven Architecture

#17

Post by exabrial »

Well put!

We do a lot of event driven architecture with ActiveMQ. We try to stick messaging-as-signalling rather than messaing-as-data-transfer. These are the terms we came up with, I'm sure Martin Fowler or someone else has described it better!

So we have SystemA that is completing some processing of something. It's going to toss a message onto a queue that SystemB is listening on. We use an XA to make sure the database and broker commit together1. SystemB then receives the event from the queue and can begin it's little tiny bit of business processing.

If one divides their "things" up into logical business units of "things that must all happen, or none happen" you end up with a pretty minimalistic architecture thats easy to understand but also offers retry capabilities if a particular system errors out on a single message.

It also allows you to take SystemB offline and let it's work pile up, then resume it later. Or you can kick of arbitrary events to test parts of the system.

1: although if this didn't happen, say during a database failure at just the right time, the right usage of row locking, transactions, and indexes on the database prevent duplicates. This is so rare in practice but we protect against it anyway.

Re: The Dual Nature of Events in Event-Driven Architecture

#18

Post by kaba0 »

I am going on a bit of a tangent here, but I always wondered, are those of you who use absolutely huge event-driven architectures, have you ever got yourself into a loop? I can't help but worry about such, as event systems are fundamentally Turing-complete, and with a complex enough system it doesn't seem too hard to accidentally send an event because A, which will eventually, multiple other events later again cases A.

Is it a common occurence' and if it happens is it hard to debug/fix? Does Kafka and other popular event systems have something to defend against it?

Re: The Dual Nature of Events in Event-Driven Architecture

#19

Post by dkarl »

Events are published observations of facts. If you want to be able to use them as triggers, or to build state, then you have to choose the events and design the system to make that possible, but you always have to ensure that systems only publish facts that they have observed.

Most issues I've seen with events are caused by giving events imprecise names, names that mean more or less than what the events attest to.

For example, a UI should not emit a SetCreditLimitToAGazillion event because of a user interaction. Downstream programmers are likely to get confused and think that the state of the user's credit limit has been set to a gazillion, or needs to be set to a gazillion. Instead, the event should be UserRequestedCreditLimitSetToAGazillion. That accurately describes what the UI observed and is attesting to, and it is more likely to be interpreted correctly by downstream systems.

In the article's example, SeatSelected sound ambiguous to me. Does it only mean the user saw that the seat was available and attempted to reserve it? Or does it mean that the system has successfully reserved the seat for that passenger? Is the update finalized, or is the user partway through a multistep process that they might cancel before confirming? Depending on the answer, we might need to release the user's prior seat for other passengers, or we might need to reserve both seats for a few minutes, pending a confirmation of the change or a timeout of their hold on the new seat. The state of the reservation may or may not need to be updated. (There's nothing wrong with using a name like that in a toy example like the article does, but I want to make the point that event names in real systems need to be much more precise.)

Naming events accurately is the best protection against a downstream programmer misinterpreting them. But you still need to design the system and the events to make sure they can be used as intended, both for triggering behavior and for reporting the state and history of the system. You don't get anything automatically. You can't design a set of events for triggering behavior and expect that you'll be able to tell the state of the system from them, or vice-versa.

Re: The Dual Nature of Events in Event-Driven Architecture

#20

Post by gunnarmorling »

marcosdumay wrote: post #10
williamdclt wrote: post #7

Earlier quoted context omitted.

> each consumer can perform their own targeted lookups as needed that puts you into tricky race condition territory, the data targeted by an event might have changed (or be deleted) between the time it was emitted and the time you're processing it. It's not always a problem, but you have to analyse if it could be every time. It also means that you're losing information on what this event actually represents: looking…

Your event data must not be mutable. That's kind of the first rule of any event-based system. It doesn't really matter the architecture, if you decide to name the things "event", everybody's head will break if you make them mutable. If you decide to add mutation there in some way, you will need to rewrite the event stream, replacing entire events.

It's not about mutability of events, but about mutating the underlying data itself. If the event only says "customer 123 has been updated", and a consumer of that event goes back to the source of the event to query the full state of that customer 123, it may have been updated again (or even deleted) since the event was emitted. Depending on the use case, this may or may not be a problem. If the consumer is only interested in the current state of the data, this typically is acceptable, but if it is needed in the complete history of changes, it is not.
Post reply on HN