Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event Sourcing, CQRS and Micro Services: Real FinTech Example

lukasniessen.medium.com

Post reply on HN
Print view View original

51–60 of 70 posts

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#51

Post by ealexhudson »

coryvirok wrote: post #50

Not sure why there is so much hate on this thread. I found the post well written, insightful, and pragmatic. Having built systems that process billions of events and displayed results, triggered notifications, etc in real time (not RTOS level, I'm talking 1 or 2 seconds of latency) you absolutely need to separate reads and writes. And if you can trust db replication to be fast and reliable, you can indeed skip distri…

I suspect there is a bit of knee-jerk because so often this pattern is misapplied. I actually quite like the example in the article although I'm basically allergic to CQRS in general.

I think your point about write-ahead logging etc is a good one. If you need a decent transactional system, you're probably using a system with some kind of WAL. If you're event sourcing and putting events into something which already implements a WAL, you need to give your head a wobble - why is the same thing being implemented twice? There can be great reasons, but I've seen (a few times) people using a perfectly fine transactional DB of some kind to implement an event store, effectively throwing away all the guarantees of the system underneath.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#52

Post by ricardobeat »

buster wrote: post #47

Event Sourcing seems like massive overkill for the stated problem. The core requirement is simple: "show account balance at any point in time" for regulatory compliance. What specific audit requirements existed beyond point-in-time balance queries? The author dismisses alternatives as "less business-focused" but doesn't justify why temporal tables or structured audit logs couldn't satisfy the actual compliance need.…

> They use running totals with audit trails, which is exactly what temporal tables provide.

In the author's case, they separate writes and reads into different DBs. The read-optimized DB has aggregated balances stored, not events. This is not materially different, and the trade-offs regarding staleness of data will be mostly the same.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#53

Post by ealexhudson »

buster wrote: post #47

Event Sourcing seems like massive overkill for the stated problem. The core requirement is simple: "show account balance at any point in time" for regulatory compliance. What specific audit requirements existed beyond point-in-time balance queries? The author dismisses alternatives as "less business-focused" but doesn't justify why temporal tables or structured audit logs couldn't satisfy the actual compliance need.…

I think they needed to be clearer about what the actual requirement was.

If the requirement is, "Show the balance _as it was_ at that point in time", this system doesn't fulfil it. They even say so in the article: if something is wrong, throw away the state and re-run the events. That's necessarily different behaviour. To do this requirement, you actually have to audit every enquiry and say what you thought the result was, including the various errors/miscalculations.

If the requirement is, "Show the balance as it should have been at that point in time", then it's fine.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#54

Post by coryvirok »

ealexhudson wrote: post #51
coryvirok wrote: post #50

Not sure why there is so much hate on this thread. I found the post well written, insightful, and pragmatic. Having built systems that process billions of events and displayed results, triggered notifications, etc in real time (not RTOS level, I'm talking 1 or 2 seconds of latency) you absolutely need to separate reads and writes. And if you can trust db replication to be fast and reliable, you can indeed skip distri…

I suspect there is a bit of knee-jerk because so often this pattern is misapplied. I actually quite like the example in the article although I'm basically allergic to CQRS in general. I think your point about write-ahead logging etc is a good one. If you need a decent transactional system, you're probably using a system with some kind of WAL. If you're event sourcing and putting events into something which already im…

For sure. Event logs in a transactional dbs are weird. I was surprised that they weren't using something like kafka for this.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#55

Post by Netcob »

javcasas wrote: post #19
throwup238 wrote: post #17

Earlier quoted context omitted.

The big caveat here is GDPR and other privacy laws. In some cases you need the ability to scrub the event store completely of PII for legal reasons, even if only to null the relevant fields in the events. Without preemptive defensive coding in your aggregates (whatever you call them) this can quickly blow up in your face.

What I have read about it is: encrypt PII with a client-depending key, do not post the key to the event system. When an erasure request comes in, delete the corresponding key. Now the data cannot be decrypted anymore for that client.

That's what I said too, and the answer was "No, just because it cannot be decrypted today does not mean it cannot be decrypted in the future. The data must be deleted"

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#56

Post by torben-friis »

Semi related question: does anyone have experience introducing proper financial data handling (ledgers or other alternatives) in a fintech _after the fact_ ?

As in, fixing things during a scaleup phase when business has been working for a while and the original improvised systems are breaking, but you can’t stop business to repair.

Currently undergoing a similar project and would really appreciate any resource thrown my way, both purely technical and/or for interfacing with accounting people with no hybrid roles to bridge the domain gap.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#57

Post by refset »

manoDev wrote: post #27
nateroling wrote: post #5

This made me do a double-take. Surely you would never do this, right? It seems to be directly counter to the idea of being able to audit changes: “Event replay: if we want to adjust a past event, for example because it was incorrect, we can just do that and rebuild the app state.”

No, that definitely happens. There are two kinds of adjustments: an adjustment transaction (pontual), or re-interpreting what happened (systemic). The event sourcing pattern is useful on both situations. Sometimes you need to replay events to have a correct report because your interpretation at the time was incorrect or it needs to change for whatever reason (external). Auditing isn't about not changing anything, but…

Aka 'bitemporal' - https://tidyfirst.substack.com/p/eventual-business-consisten...

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#58

Post by mrkeen »

nateroling wrote: post #5

This made me do a double-take. Surely you would never do this, right? It seems to be directly counter to the idea of being able to audit changes: “Event replay: if we want to adjust a past event, for example because it was incorrect, we can just do that and rebuild the app state.”

It would be outside of the normal exceptional cases, yes.

Like buggy data that crashes the system.

If you have the old events there, you can "measure twice, cut once", in the sense that you can keep re-running your old events and compare them to the new events under unit-test conditions, and be absolutely sure that your history re-writing won't break anything else.

It's not for just doing a refund or something.

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#59

Post by saxenaabhi »

torben-friis wrote: post #56

Semi related question: does anyone have experience introducing proper financial data handling (ledgers or other alternatives) in a fintech _after the fact_ ? As in, fixing things during a scaleup phase when business has been working for a while and the original improvised systems are breaking, but you can’t stop business to repair. Currently undergoing a similar project and would really appreciate any resource thrown…

Isn't it simple? Create events from past relational data and add them to the log?

Re: Event Sourcing, CQRS and Micro Services: Real FinTech Example

#60

Post by mrkeen »

mirekrusin wrote: post #48
r1cka wrote: post #33

Earlier quoted context omitted.

The argument I've always heard for this was issues with code, not the event. If for a period of time you have a bug in your code, with event sourcing, you can fix the bug and replay all the events to correct current projections of state.

What if your correction renders subsequent events nonsensical?

There is a very real chance of this happening, and two choices.

One - bake whatever happens into your system permanently, like 99% of all apps, and disallow corrections.

Two - keep the events around so that you can check and re-check your corrections before you check in new code or data.

Post reply on HN