Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Events: Fat or Thin?

codesimple.blog

Post reply on HN
Print view View original

41–50 of 61 posts

Re: Events: Fat or Thin?

#41

Post by [deleted] »

delusional wrote: post #39
tremon wrote: post #32

Earlier quoted context omitted.

If you allow A's state to lag behind it's own events That's a mischaracterization. A's state is not lagging its emitted events; instead, A's state may have been changed at the time A's event is processed.

The comment I quoted says: > when A's state lags or has moved on ahead of the event That sounds like it can EITHER be ahead or behind. Specifically, I do not understand it as A's state can either lag OR be ahead, not that "lags" is a synonym for "moved ahead"

[deleted]

Re: Events: Fat or Thin?

#42

Post by jayd16 »

SideburnsOfDoom wrote: post #36
andsbf wrote: post #33

When will people settle on the fact that there is no silver bullet? It's always a matter of trade offs according to the system needs.

Because the very real downsides of thin events are not simple or obvious. It may be fine most of the time, but as it usually involves a "get more details" call over http or of some other kind, it has more moving parts, is therefor more prone to failures and slowdowns due to the extra coupling. This can kick in when load goes up or some other issue affects reliability, and cause cascading failure.

But it's a lot less prone to data races and other upsides so it really is contextual to your needs.

I would also push back that it has more moving parts. You'll often need to pull information and getting pushed information as well can duplicate code on both the service side and client. In practice thin events are easier to get right, despite the extra API requests.

I also think the cases where there's some kind of outage but it's an outage such that the information in the event is enough is fairly rare. I would guess it's more rare than outages that also disable event triggering anyhow.

Re: Events: Fat or Thin?

#43

Post by SideburnsOfDoom »

jayd16 wrote: post #42
SideburnsOfDoom wrote: post #36

Earlier quoted context omitted.

Because the very real downsides of thin events are not simple or obvious. It may be fine most of the time, but as it usually involves a "get more details" call over http or of some other kind, it has more moving parts, is therefor more prone to failures and slowdowns due to the extra coupling. This can kick in when load goes up or some other issue affects reliability, and cause cascading failure.

But it's a lot less prone to data races and other upsides so it really is contextual to your needs. I would also push back that it has more moving parts. You'll often need to pull information and getting pushed information as well can duplicate code on both the service side and client. In practice thin events are easier to get right, despite the extra API requests. I also think the cases where there's some kind of ou…

> But it's a lot less prone to data races

What's the reasoning for that statement? My other comments have detailed cases of data races, caused by using thin events, solved by using fat events, so I'm going to push back on that: In my experience, the idea that thin events are "less prone to data races" has not been true at all, and that data race is inherent in the model of "receive an event, go back and get more data over http about it". How would you qualify this as "a lot" - how much is that lot? Citations very much needed.

> I would also push back that it has more moving parts. You'll often need to pull information

"Often" is not always. You seem to be saying that when you don't get the signature benefit of fat events (not having to pull information) ... there's no benefit. Yes, that's a tautology, and also an encouragement to further refine the design until you do get the benefit.

> I also think the cases where there's some kind of outage but it's an outage such that the information in the event is enough

Are you speaking from experience here? Doesn't seem like it. I refer you to point c) here https://news.ycombinator.com/item?id=33392655 It helps reliability to shorten the list of services that your service depends upon being 100% up and responsive.

Re: Events: Fat or Thin?

#44

Post by didip »

I have a different take, why not just send RPC functions with all of their parameters?

Then the payload is guaranteed to be small but still able to handle complex operations.

Re: Events: Fat or Thin?

#45

Post by echelon »

majke wrote: post #31

I'll bite. Neither. Both. Depending on system. When the "state" is large, or changes often, obviously you can't send full state every time - that would be too much for end-nodes to process on every event. Both cpu - deserialization, and bandwidth. Delta is the answer. Delta though is hard, since there always is an inherent race between getting the first full snapshot, and subscribing to updates. On the other hand doi…

There's a third type of event:

- Entire Object.

You send the entire state of the entire object that changed. Irrelevant fields and all.

This makes business logic and migrations easier in dependent services. You can easily roll back to earlier points in time without diffing objects to determine what state changed. You don't have to replay an entire history of events to repopulate caches and databases. You can even send "synthetic" events to reset the state of everything that is listening from a central point of control.

I've dealt with all three types of system, and this is by far the easiest one to work with.

Re: Events: Fat or Thin?

#46

Post by SideburnsOfDoom »

tremon wrote: post #32
delusional wrote: post #10

Earlier quoted context omitted.

b) You're asking for occasional "eventual consistency" trouble when A's state lags or has moved on ahead of the event If you allow A's state to lag behind it's own events, then how are you ever going to create a sane system? Surely A either has to be ahead or at the state that caused the event to emit, or events are pointless.

If you allow A's state to lag behind it's own events That's a mischaracterization. A's state is not lagging its emitted events; instead, A's state may have been changed at the time A's event is processed.

The "own events" was the faulty assumption. it's not always the same service that both emits the events, and is the place to go to over http for data. It "seems logical" to also build that store from listening to events, but it can cause issues as mentioned.

Re: Events: Fat or Thin?

#47

Post by SideburnsOfDoom »

echelon wrote: post #45
majke wrote: post #31

I'll bite. Neither. Both. Depending on system. When the "state" is large, or changes often, obviously you can't send full state every time - that would be too much for end-nodes to process on every event. Both cpu - deserialization, and bandwidth. Delta is the answer. Delta though is hard, since there always is an inherent race between getting the first full snapshot, and subscribing to updates. On the other hand doi…

There's a third type of event: - Entire Object. You send the entire state of the entire object that changed. Irrelevant fields and all. This makes business logic and migrations easier in dependent services. You can easily roll back to earlier points in time without diffing objects to determine what state changed. You don't have to replay an entire history of events to repopulate caches and databases. You can even sen…

How does this differ from a "fat event" ?

Re: Events: Fat or Thin?

#48

Post by insanitybit »

FeepingCreature wrote: post #27
PaulRobinson wrote: post #14

Earlier quoted context omitted.

The bit you seem to be missing is the events are the source of truth, not the databases. Lose your database? Roll up all the events. Got a lot of them? Take snapshots and then roll up from the last trusted snapshot. In true event sourced systems, the databases and stateful systems are artefacts that can be thrown away and rebuilt. The event log is the actual “true” database. Once you design around that, your objectio…

This only works if your events are in a single globally ordered stream or all your code is eventually consistent over every stream it consumes. Specifically, you cannot do the "query a service for the aggregate state" thing this article espouses for thin events, ever.

You can achieve strong eventual consistency with this system.

Re: Events: Fat or Thin?

#49

Post by jayd16 »

SideburnsOfDoom wrote: post #43
jayd16 wrote: post #42

Earlier quoted context omitted.

But it's a lot less prone to data races and other upsides so it really is contextual to your needs. I would also push back that it has more moving parts. You'll often need to pull information and getting pushed information as well can duplicate code on both the service side and client. In practice thin events are easier to get right, despite the extra API requests. I also think the cases where there's some kind of ou…

> But it's a lot less prone to data races What's the reasoning for that statement? My other comments have detailed cases of data races, caused by using thin events, solved by using fat events, so I'm going to push back on that: In my experience, the idea that thin events are "less prone to data races" has not been true at all, and that data race is inherent in the model of "receive an event, go back and get more data…

Data races can occur with fat events because fat events don't usually store a snapshot of the entire world. They're often actually delta events, forcing the consumer to rely on caching (a racey affair) or just fall back to hitting the API anyway.

If you're having issues with thin events causing data inconsistency then the answer should be better data representation in your core API. If you had to solve it by cramming perfectly stale but self consistent data into a fat event, a better solution would be to persist such data and make it queryable, no?

But of course any way can be made to work. Any extra bit of data can be appended to the event details. Any number of separate DB and persistent queues can be maintained and backed up like the business depends on it.

In my experience thin events are just less bug prone because they're more simple. You're not worrying about stale data floating around an event queue or a client cache. You're not guessing what data the consumer needs. Having the event consumer pull data is usually a trivial cost difference.

Fat events have their advantages but I put them under premature optimization and YAGNI.

Re: Events: Fat or Thin?

#50

Post by SideburnsOfDoom »

delusional wrote: post #37
SideburnsOfDoom wrote: post #25

Earlier quoted context omitted.

> I'm genuinely curious how such an architecture would work. Complex systems are the way that they are because they got that way over time. It is not my goal to defend or even characterise a system that I did not create. I am here telling you the issue that I saw: one event consumer, at an edge case, ran substantially behind another, and when they attempted to co-ordinate over http, this failed. And how it was succes…

Ah, so A and C where both subscribed to B, but during A's processing of the event it assumed C had already processed it and tried to look up some state. Is that correctly understood? This sounds more like an architectural deficiency (as you say probably from architectural decay) than a systematic design edge case. I can't quite understand what information A would need to get from C that could be included in the fat e…

> Ah, so A and C where both subscribed to B, but during A's processing of the event it assumed C had already processed it and tried to look up some state. Is that correctly understood?

yes, though you're down the rabbit-hole on this one issue. My point (aside from the fact we actually saw this specific issue and it took a long time to correctly diagnose) is that with thin events followed by a http query call-back, You're asking for occasional "eventual consistency" trouble. Data races will happen occasionally - this is inevitable in the design.

At the tail end of the latency distributions, too fast or too slow, or service A is now having a blip, or you now hit the new version just deployed or whatever, things will go wrong by mis-sequencing in surprising and hard to follow ways (example given that you're fixating on) in complex real systems, and it's a win to avoid that chaos entirely, with fat events.

Post reply on HN