Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Dual Nature of Events in Event-Driven Architecture

reactivesystems.eu

Post reply on HN
Print view View original

31–40 of 66 posts

Re: The Dual Nature of Events in Event-Driven Architecture

#31

Post by Hilift »

kaba0 wrote: post #18

I am going on a bit of a tangent here, but I always wondered, are those of you who use absolutely huge event-driven architectures, have you ever got yourself into a loop? I can't help but worry about such, as event systems are fundamentally Turing-complete, and with a complex enough system it doesn't seem too hard to accidentally send an event because A, which will eventually, multiple other events later again cases…

Debugging Windows events is challenging. I once created a reproducible deadlock using events. I had a Windows application with a notebook computer base, so I had to provide for and handle the power save/resume event for a service in the application. This was simple, just ensure that the service stopped and didn't get into a funky state, and restarted cleanly after resume. During testing, I accidentally discovered the power save event was going into a loop or hanging. Fortunately, Microsoft has a tool for this (DebugDiag - C#), and to my amazement landed right on the problem area. This event (like most) can and was firing multiple times..., so had to add some extra locks. Also, the power save/resume code worked 100% on a virtual, it was real hardware where the issue manifested.

Re: The Dual Nature of Events in Event-Driven Architecture

#32

Post by phi-go »

marcosdumay wrote: post #10
williamdclt wrote: post #7

Earlier quoted context omitted.

> each consumer can perform their own targeted lookups as needed that puts you into tricky race condition territory, the data targeted by an event might have changed (or be deleted) between the time it was emitted and the time you're processing it. It's not always a problem, but you have to analyse if it could be every time. It also means that you're losing information on what this event actually represents: looking…

Your event data must not be mutable. That's kind of the first rule of any event-based system. It doesn't really matter the architecture, if you decide to name the things "event", everybody's head will break if you make them mutable. If you decide to add mutation there in some way, you will need to rewrite the event stream, replacing entire events.

I understand gp to say that the database data is changed not the data in the event.

Surely, some data needs to change if a password is updated?

Re: The Dual Nature of Events in Event-Driven Architecture

#33

Post by lutzh »

gunnarmorling wrote: post #21
lutzh wrote: post #2

Article observing that events in event-driven architecture are both triggers of actions and carriers of data, and that these roles may conflict in the event design. Submitted by author.

Nice one. I wrote about this a while ago, from a slightly different perspective, focusing on data change events [1]. Making a similar differentiation there between id-only events (which you describe as triggers of action; from a data change feed perspective, that action typically would be a re-select of the current state of the represented record), full events (your carriers of data) and patch events (carriers of dat…

Thanks for your feedback Gunnar, I appreciate it!

Your categorization makes total sense and fits well with what I called the "spectrum". I only mentioned the "id-only" events to show what the one end of the spectrum would look like. What I call the "trigger" events would be what you call "delta" events. I should have written that more clearly.

Interestingly a few people advocated for id-only events as a response to the article. I have some issues with that pattern.. already thinking about a follow-up article to elaborate on that.

Re: The Dual Nature of Events in Event-Driven Architecture

#34

Post by withinboredom »

candiddevmike wrote: post #13
withinboredom wrote: post #11

Earlier quoted context omitted.

The best way to avoid distributed locks and transactions is to manually do the work. For example, instead of doing a distributed lock on two accounts when transferring funds, you might do this (which is the same as a distributed transaction, without the lock): 1. take money from account A 2. if failed, put money back into account A 3. put money into account B 4. if failed, put money back into account A In other words…

> This also requires that you have some kind of mechanism to handle an application crash between 2 and 3, but that is something else entirely Like a distributed transaction or lock. This is the entire problem space, your example above is very naive.

You _can_ use them, but even that won't save you. If you take a distributed lock, and crash, you still have the same issue. What I wrote is essentially a distributed transaction and what happens in a distributed transaction with "read uncommitted" isolation levels. A database that supports this handles all the potential failure cases for you. However, that doesn't magically make the errors disappear or may not be even fully/properly handled (e.g., a node running out of disk space in the middle of the transaction) by your code/database. It isn't naive, it is literally a pseudo-code of what you are proposing.

Re: The Dual Nature of Events in Event-Driven Architecture

#35

Post by withinboredom »

singron wrote: post #26
withinboredom wrote: post #11

Earlier quoted context omitted.

The best way to avoid distributed locks and transactions is to manually do the work. For example, instead of doing a distributed lock on two accounts when transferring funds, you might do this (which is the same as a distributed transaction, without the lock): 1. take money from account A 2. if failed, put money back into account A 3. put money into account B 4. if failed, put money back into account A In other words…

The better version of this is sagas, which is a kind of a simplified distributed transaction. If you do this without actually using sagas, you can really mess this up. E.g. you perform step 2, but fail to record it. When resuming from crash, you perform step 2 again. Now A has too much money in their account.

Sagas are great for this and should be used when able, IMHO. It's still possible to mess it up, as there are basically two guarantees you can make in a distributed system: at-least-once, and at-most-once. Thus, you will either need to accept the possibility of lost messages or be able to make your event consumers idempotent to provide an illusion of exactly-once.

Sagas require careful consideration to make sure you can provide one of these guarantees during a "commit" (the order in which you ACK a message, send resulting messages, and record your own state -- if necessary) as these operations are non-atomic. If you mess it up, you can end up providing the wrong level of guarantee by accident. For example:

1. fire resulting messages

2. store state (which includes ids of processed messages for idempotency)

3. ACK original event

In this case, you guarantee that you will always send results at-least-once if a crash happens between 1&2. Once we get past 2, we provide exactly-once semantics, but we can only guarantee at-least-once. If we change the order to:

1. store state

2. fire messages

3. ACK original message

We now only provide at-most-once semantics. In this case, if we crash between 1&2, when we resume, we will see that we've already handled the current message and not process it again, despite never having sent any result yet. We end up with at-most-once if we swap 1&3 as well.

So, yes, Sagas are great, but still pretty easy to mess up.

Re: The Dual Nature of Events in Event-Driven Architecture

#36

Post by plaguuuuuu »

kaba0 wrote: post #18

I am going on a bit of a tangent here, but I always wondered, are those of you who use absolutely huge event-driven architectures, have you ever got yourself into a loop? I can't help but worry about such, as event systems are fundamentally Turing-complete, and with a complex enough system it doesn't seem too hard to accidentally send an event because A, which will eventually, multiple other events later again cases…

Never seen it, but totally possible.

It always bothers me that the systems I've worked on have their data flows mapped out basically in semi-up-to-date miro diagrams. If that. There's no overarching machine readable and verifiable spec.

Re: The Dual Nature of Events in Event-Driven Architecture

#37

Post by monksy »

bob1029 wrote: post #3

The triggering of the action is a direct consequence of the information an event contains. Whether or not an action is triggered should not be the responsibility of the event. If you are writing events with the intention of having them invoke some specific actions, then you should prefer to invoke those things directly. You should be describing a space of things that have occurred, not commands to be carried out. By…

Say that again louder for people in the back.

Message queues aren't a networking protocol. Anyone can subscribe to consume the events.

Re: The Dual Nature of Events in Event-Driven Architecture

#38

Post by remram »

exabrial wrote: post #17

Well put! We do a lot of event driven architecture with ActiveMQ. We try to stick messaging-as-signalling rather than messaing-as-data-transfer. These are the terms we came up with, I'm sure Martin Fowler or someone else has described it better! So we have SystemA that is completing some processing of something. It's going to toss a message onto a queue that SystemB is listening on. We use an XA to make sure the data…

So... you're not using event-driven architecture. That's fine, but I don't know why it supports this article's weird point about event-driven architecture.

Re: The Dual Nature of Events in Event-Driven Architecture

#39

Post by hcarvalhoalves »

The "produce a trigger event then have the consumer reach back to the producer and fetch data" can be an anti-pattern. You expose yourself to all kinds of race conditions and cache incoherency, then you start trying to fix with cache invalidation or pinning readers, and the result is you can't scale readers well.

If you're a using a message queue, the message should convey necessary information such that, if all messages were replayed, the consumer would reach the same state. Anything other than that, you'll be in a world of pain at scale.

Re: The Dual Nature of Events in Event-Driven Architecture

#40

Post by chipdart »

hcarvalhoalves wrote: post #39

The "produce a trigger event then have the consumer reach back to the producer and fetch data" can be an anti-pattern. You expose yourself to all kinds of race conditions and cache incoherency, then you start trying to fix with cache invalidation or pinning readers, and the result is you can't scale readers well. If you're a using a message queue, the message should convey necessary information such that, if all mess…

> If you're a using a message queue (...)

Events and messages are entirely different things. They might look similar, but their responsibilities are completely different. The scenario you're describing matches the usecases for messages, not events.

Post reply on HN