Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Events: Fat or Thin?

codesimple.blog

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Events: Fat or Thin?

#51

Post by SideburnsOfDoom »

jayd16 wrote: post #49
SideburnsOfDoom wrote: post #43

Earlier quoted context omitted.

> But it's a lot less prone to data races What's the reasoning for that statement? My other comments have detailed cases of data races, caused by using thin events, solved by using fat events, so I'm going to push back on that: In my experience, the idea that thin events are "less prone to data races" has not been true at all, and that data race is inherent in the model of "receive an event, go back and get more data…

Data races can occur with fat events because fat events don't usually store a snapshot of the entire world. They're often actually delta events, forcing the consumer to rely on caching (a racey affair) or just fall back to hitting the API anyway. If you're having issues with thin events causing data inconsistency then the answer should be better data representation in your core API. If you had to solve it by cramming…

> fat events because fat events don't usually store a snapshot of the entire world. They're often actually delta events, forcing the consumer to rely on caching (a racey affair) or just fall back to hitting the API anyway.

I think you're saying "I don't like fat events because they're not really fat events", well, sorry that's your experience, I don't think that's a valid criticism of the actual thing at all, just of your poor experience of it.

> Data races can occur with fat events.. hitting the API anyway.

That happens with thin events as a matter of course, right? You stated that thin events are "lot less prone to data races" and now you're saying that they're the same? Where's the fat-event-specific issue that you alluded to? Citation not provided.

> Having the event consumer pull data is usually a trivial cost difference.

As I have stated twice before, it's a non-trivial _reliability_ difference, and that's the key.

Re: Events: Fat or Thin?

#52

Post by morelisp »

manfre wrote: post #20
doctor_eval wrote: post #18

Earlier quoted context omitted.

With an API you can publish a new endpoint (/v1, /v2 etc). It’s normally reasonably easy to maintain an old API even while you add features to the new API, and the runtime penalty is minimal because clients would be expected to call just one version of the API for any given event. (You can also see who’s calling the old API and ask them to change) But this is not true for events. If you change the body such that you…

Supporting multiple versions of an event schema is a solved problem. Apache Avro with a published schema hash in a message header is one solution. https://avro.apache.org/

This lets you identify the version but it doesn't let old clients read the new messages. (Well, for avro and others they still can if the new fields aren't important or the old fields aren't required - but if you can do that you also don't really have a new incompatible version and you don't need the schema hash to begin with.)

The point is that with a pull-based API, I have a fixed number of requests. As clients migrate from /v1 to /v2, load on /v1 goes down and /v2 goes up, and I can adjust resource allocations accordingly to keep the total requirements relatively constant. I can even reimplement /v1 in terms of /v2 internally in many cases and have ~0 operational overhead.

But for an evented system, as soon as just a single client wants v2 I need to publish that, and as long as any client wants v1 I need to publish that. So my outbound "work" (at the very least i/o but probably also DTO conversions and god help you if it's any kind of storage or business logic) is doubled immediately and remains doubled until everything is migrated.

Re: Events: Fat or Thin?

#53

Post by kgeist »

Fat events once overloaded our message broker with OOM under high load and the broker's default behavior was to block all publishers until the queue was emptied (to release memory) - downtime as a result. Another issue was that under high load, if the event queue was too large, handlers would end up processing very stale data resulting in all kinds of broken behavior (from the point of view of the user).

Thin events resulted in DDoS of our service a few times because handlers would call our APIs too frequently to retrieve object state (which was partially mitigated by having separate machines serve incoming traffic and process events).

(A trick we used which worked for both fat and thin events was to add versioning to objects to avoid unnecessary processing).

We also used delta events as well but they had same issues as thin events because handlers usually have to retrieve full object state anyway to have meaningful processing (not always, depends on business logic and the architecture).

There are so many ways to shoot yourself in the foot with all three approaches and I still hesitate a lot when choosing what kind of events to use for the next project.

Re: Events: Fat or Thin?

#54

Post by echelon »

SideburnsOfDoom wrote: post #47
echelon wrote: post #45

Earlier quoted context omitted.

There's a third type of event: - Entire Object. You send the entire state of the entire object that changed. Irrelevant fields and all. This makes business logic and migrations easier in dependent services. You can easily roll back to earlier points in time without diffing objects to determine what state changed. You don't have to replay an entire history of events to repopulate caches and databases. You can even sen…

How does this differ from a "fat event" ?

Thin event (1): person object XYZ changed.

Thin event (2): address object ABC changed.

Delta event (1): person object XYZ's name changed to "Bob"

Delta event (2): address object ABC's address line 2 was deleted and zip code was changed to "12345"

Fat event (1): person object XYZ changed, and here's everything we think person-consuming systems will care about

Fat event (2): address object ABC changed, and here's everything we think address-consuming systems will need

Entire object (1): { person: { token: "XYZ", name: "Bob", email: "bob@bob.com" age: 42, likes: ["ice cream", ...], ... }, updated_at: T, updated_by: U, version: 3, ... }

Entire object (2): { address: { token: "ABC", line_1: "1234 Some Place Rd.", line_2: null, city: "Everywhere", state: "NA", zip: "12345", ... }, updated_at: T, updated_by: U, version: 5, ... }

Re: Events: Fat or Thin?

#55

Post by jayd16 »

SideburnsOfDoom wrote: post #51
jayd16 wrote: post #49

Earlier quoted context omitted.

Data races can occur with fat events because fat events don't usually store a snapshot of the entire world. They're often actually delta events, forcing the consumer to rely on caching (a racey affair) or just fall back to hitting the API anyway. If you're having issues with thin events causing data inconsistency then the answer should be better data representation in your core API. If you had to solve it by cramming…

> fat events because fat events don't usually store a snapshot of the entire world. They're often actually delta events, forcing the consumer to rely on caching (a racey affair) or just fall back to hitting the API anyway. I think you're saying "I don't like fat events because they're not really fat events", well, sorry that's your experience, I don't think that's a valid criticism of the actual thing at all, just of…

>That happens with thin events as a matter of course, right?

No. If you send no data, what is the race? You can structure an API where there are no races. If you concede to API access the API can provide any sort of historical data necessary. The issue is with incomplete event data and assumptions around what the state was when the event was sent.

>As I have stated twice before, it's a non-trivial _reliability_ difference, and that's the key

You can state it until you're blue but the advantage _is_ usually trivial or there would be no debate. It just doesn't bite that often.

I would say the buggy and prone to rot coupling of baked in event data is a bigger concern for most.

That said, I'm making an argument about what is the best bet for dev time and it sounds like you're making an argument about what is best given infinite developer resources, an unchanging APIs, and full knowledge of what data the consumer needs.

Re: Events: Fat or Thin?

#56

Post by SideburnsOfDoom »

jayd16 wrote: post #55
SideburnsOfDoom wrote: post #51

Earlier quoted context omitted.

> fat events because fat events don't usually store a snapshot of the entire world. They're often actually delta events, forcing the consumer to rely on caching (a racey affair) or just fall back to hitting the API anyway. I think you're saying "I don't like fat events because they're not really fat events", well, sorry that's your experience, I don't think that's a valid criticism of the actual thing at all, just of…

>That happens with thin events as a matter of course, right? No. If you send no data, what is the race? You can structure an API where there are no races. If you concede to API access the API can provide any sort of historical data necessary. The issue is with incomplete event data and assumptions around what the state was when the event was sent. >As I have stated twice before, it's a non-trivial _reliability_ diffe…

> No. If you send no data, what is the race? You can structure an API where there are no races.

You seem to be saying that the race is a problem when comparing 2 copies of the same data (yes) and that this is an issue for fat events (no, and misses the entire point).

A vague thin event contains at minimum an event type and an item id, e.g. "SoemthingHappendedToAnOrder id:123456" which is _ahem_ two pieces of data that are sent. Events containing "no data" are not a thing, don't be absolute.

So there's potential for a race or inconsistency when you correlate that with a http api which might or might not have that order. You can't entirely get away from that.

> You can structure an API where there are no races.

I do not think that you understand "the fallacies of Distributed Systems"

https://www.simpleorientedarchitecture.com/8-fallacies-of-di...

> It just doesn't bite that often.

Has not been my experience.

> That said, I'm making an argument about what is the best bet for dev time

Sure, if you want to write as many bugs as possible as fast as possible, go ahead. (yes, I realise that this is mischaracterising hyperbole, but you did same by saying "infinite developer resources" etc above)

Re: Events: Fat or Thin?

#57

Post by mrkeen »

didip wrote: post #44

I have a different take, why not just send RPC functions with all of their parameters? Then the payload is guaranteed to be small but still able to handle complex operations.

If you send to multiple receivers, some of the messages may not make it through. Then you're in the position of deciding whether 'the message was sent' or not.

Re: Events: Fat or Thin?

#58

Post by SideburnsOfDoom »

echelon wrote: post #54
SideburnsOfDoom wrote: post #47

Earlier quoted context omitted.

How does this differ from a "fat event" ?

Thin event (1): person object XYZ changed. Thin event (2): address object ABC changed. Delta event (1): person object XYZ's name changed to "Bob" Delta event (2): address object ABC's address line 2 was deleted and zip code was changed to "12345" Fat event (1): person object XYZ changed, and here's everything we think person-consuming systems will care about Fat event (2): address object ABC changed, and here's every…

since the "fat event" ones are vaguely defined here, they could be arbitrarily close to or far from the "Entire object" cases. How does it differ? Maybe it does not.

Re: Events: Fat or Thin?

#59

Post by drewcoo »

doctor_eval wrote: post #3

> Thin Events add coupling That’s not my experience. In fact I’d say fat events add coupling because they create an invisible (from the emitter) dependency on the event body, which becomes ossified. So I’d say the opposite: thin events reduce coupling. Sure, the receiver might call an API and that creates coupling with the API. But receivers are also free to call or not call any other API they want. What if they don’…

Thank you. Came here to say that.

When I've seen this fat event pattern it's been because different services' responsibilities were not fully separated. And that's tight coupling. Fat events imply tight coupling.

The "thin" pattern described in the article goes like this:

1) service FOO gets an event

2) FOO then has to query BAR (and maybe BAZ and QUUX) to determine the overall state of everything to determine what to do next

And #2 means all of that is kind of "thin" is tightly coupled, too.

I've also personally seen thin events that are not the article's thin strawman.

I sometimes wonder if people understand coupling or design.

Re: Events: Fat or Thin?

#60

Post by echelon »

SideburnsOfDoom wrote: post #58
echelon wrote: post #54

Earlier quoted context omitted.

Thin event (1): person object XYZ changed. Thin event (2): address object ABC changed. Delta event (1): person object XYZ's name changed to "Bob" Delta event (2): address object ABC's address line 2 was deleted and zip code was changed to "12345" Fat event (1): person object XYZ changed, and here's everything we think person-consuming systems will care about Fat event (2): address object ABC changed, and here's every…

since the "fat event" ones are vaguely defined here, they could be arbitrarily close to or far from the "Entire object" cases. How does it differ? Maybe it does not.

Your team decides what parts of the model to expose in its events and it becomes an API in its own right.

You might change the names of fields, move them to places that don't reflect where they live on a nested model, etc. It requires a lot more thought and maintenance.

That isn't to say the choice can't be correct. All of these approaches have pros and cons.

Post reply on HN