Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event Sourcing, CQRS and Micro Services: Real FinTech Example

lukasniessen.medium.com

Post reply on HN
Print view View original

21–30 of 70 posts

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#21

Post by saxenaabhi »

javcasas wrote: post #18
saxenaabhi wrote: post #14

Earlier quoted context omitted.

Not speaking about their case, but I think some cases a "versioned mutable data store" with a event log that lists updates/inserts makes more sense than an "immutable event log" one like kafka. Consider the update_order_item_quantity event in a classic event sourced systems. It's not possible to guarantee that two waiters dispatching two such events at same time when current quantity is 1 would not cause the quantity…

Any modern DB with a WAL (write ahead log) is an immutable event system, where the events are the DB primitives (insert, update, delete...). When you construct your own event system you are constructing a DB with your own primitives (deposit, withdraw, transfer, apply monthly interest...). You have to figure out your transaction semantics. For example, how to reject invalid events.

> Any modern DB with a WAL (write ahead log) is an immutable event system, where the events are the DB primitives (insert, update, delete...).

Agreed, I just wish apart from WAL they also had versioning as first class and their update api required clients to pass the version they have "last seen" to prevent inconsistencies.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#22

Post by ndriscoll »

throwup238 wrote: post #17
javcasas wrote: post #12

Earlier quoted context omitted.

Yeah, that's a big NO. Events are immutable. If an event is wrong, you post an event with an amendment. Then yes, rebuild the app state.

The big caveat here is GDPR and other privacy laws. In some cases you need the ability to scrub the event store completely of PII for legal reasons, even if only to null the relevant fields in the events. Without preemptive defensive coding in your aggregates (whatever you call them) this can quickly blow up in your face.

For finance recordkeeping requirements take precedence over privacy requirements. Audit trail data must be on WORM storage and must not be scrubbable.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#24

Post by kabes »

zsoltkacsandi wrote: post #15
nateroling wrote: post #5

This made me do a double-take. Surely you would never do this, right? It seems to be directly counter to the idea of being able to audit changes: “Event replay: if we want to adjust a past event, for example because it was incorrect, we can just do that and rebuild the app state.”

Instead of modifying the original (and incorrect) event, you can add a manual correction event with the info of who did it and why, and replay the events. This is how we dealt with such corrections with event sourcing.

But you don't need to replay in that case. You just fire the correction event and the rest is taken care of.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#25

Post by kabes »

speed_spread wrote: post #20
nateroling wrote: post #5

This made me do a double-take. Surely you would never do this, right? It seems to be directly counter to the idea of being able to audit changes: “Event replay: if we want to adjust a past event, for example because it was incorrect, we can just do that and rebuild the app state.”

It's poorly phrased but I'm not sure they meant "mutate the past". The keyword is "adjust" which could mean "append a correction".

But then you wouldn't need a replay. So the author really means mutate the past.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#26

Post by chuckadams »

throwup238 wrote: post #2

> their MVP was not auditable and thus not compliant with financial regulations and also not scalable (high usage and fault tolerance). There it is. My automatic response to any questions about event sourcing is “if you have to ask, you don’t need it.” This is one of those situations where the explosion in complexity somewhat makes sense: when you need legally enforced auditability. Event sourcing is a really cool ar…

I'd call it a trade-off. I've done event sourcing for systems that didn't really need it, and the ability to fix things retroactively by just replaying the log through the corrected pipeline of processors was marvelous. It lends itself to functional programming with effects instead of banging on global state wherever and whenever, and that pays dividends. But there's no free lunch, and pipelines could get complex by needing to track "epochs" and keeping bug-compatible behavior for old messages that would be patched downstream. Since the system just built persisted state at the end, I could checkpoint it at a new state and retire the older epochs. Doubtful that would that pass a rigorous audit, but if that's not one of your requirements, you really can do event sourcing halfway and still reap a lot of the benefits.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#27

Post by manoDev »

nateroling wrote: post #5

This made me do a double-take. Surely you would never do this, right? It seems to be directly counter to the idea of being able to audit changes: “Event replay: if we want to adjust a past event, for example because it was incorrect, we can just do that and rebuild the app state.”

No, that definitely happens.

There are two kinds of adjustments: an adjustment transaction (pontual), or re-interpreting what happened (systemic). The event sourcing pattern is useful on both situations.

Sometimes you need to replay events to have a correct report because your interpretation at the time was incorrect or it needs to change for whatever reason (external).

Auditing isn't about not changing anything, but being able to trace back and explain how you arrived at the result. You can have as many "versions" as you want of the final state, though.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#28

Post by kabes »

throwup238 wrote: post #2

> their MVP was not auditable and thus not compliant with financial regulations and also not scalable (high usage and fault tolerance). There it is. My automatic response to any questions about event sourcing is “if you have to ask, you don’t need it.” This is one of those situations where the explosion in complexity somewhat makes sense: when you need legally enforced auditability. Event sourcing is a really cool ar…

This! I've recently was hired to fix a company that tried to build their own niche ERP over the last 10 years and they had totally drowned in the confluent kool-aid. There is very very few projects where event sourcing is the best solution.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#29

Post by manoDev »

saxenaabhi wrote: post #9
herval wrote: post #7

Earlier quoted context omitted.

Do you mean inter-system? Intra-system would mean on the same db, so a simple transaction would do. For inter-system consistency, you’d probably need a reconciliation mechanism or some kind of 2 phase commit

I mean a bank account sending money to another bank account. That would be 2 events(one withdraw and one deposit)? But if I'm downstream consumer consuming the event log and computing the state from that, if for some reasons I receives only first event the state computed would be invalid and not represent the real state of accounts?

Nobody relies on atomic transactions to model money transfers in the real world. You can read up on "clearing" and "settlement" processes to get an idea.
Post reply on HN