Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Reactive Monolith – How to Move from CRUD to Event Sourcing

wix.engineering

Post reply on HN
Print view View original

81–90 of 112 posts

Re: The Reactive Monolith – How to Move from CRUD to Event Sourcing

#81

Post by dtech »

zaptheimpaler wrote: post #67

I used to be really excited about event sourcing but yeah, its usually just over-engineering. It appeals to the nerd in me because its a powerful & clean model - events are immutable, your entire database can theoretically be reconstructed at any point in time by just replaying an event log up to time X. In the ideal form its sort of the highest fidelity version of data storage, throwing nothing away, supporting all…

For non-trivial amounts of data you should combine event sourcing with snapshots - i.e. a somewhat up-to-date materialized view/DB table - so you don't have to start for 0. At that point you can delete older events or move them to cold storage.

A valid question then is: "what do you gain over just using the table"? You gain a well-described model of your business domain, with very clear actions about what should happen on which real-world event, and an audit log. Whether that's worth it depend on your use case.

Secondary benefits include easily being able to share events for new apps or use-cases, business analysts really like them, and making it unlikely things go wrong because data is in an inconsistent state.

Re: The Reactive Monolith – How to Move from CRUD to Event Sourcing

#83

Post by lmm »

holoduke wrote: post #59
lmm wrote: post #45

Earlier quoted context omitted.

Prevent lost writes from when two people read-modify-write at the same time. Be able to answer questions like "how did this end up like this" or "what happened to the change I made on xx/yy". Easier sharding, better performance, and avoiding deadlocks, because you're not worrying about database-level transactions any more. Much easier data migrations that you can do in a gradual, rollbackable way. Clear data provenan…

Could you elaborate how you solve transaction, double writes , deadlocks. I mean the consumers do still process messages in parallel right? Or do you force a serial pipeline to avoid handling messages at the same time? I thought that EV is more about creating entire separate entities/codebase which have zero dependencies.

> I mean the consumers do still process messages in parallel right? Or do you force a serial pipeline to avoid handling messages at the same time?

Generally I follow the Kafka approach of, essentially, shards that are each processed serially. So unrelated messages may be processed out of order, but related messages will be processed serially.

If you don't have a good enough key, or you need to join two streams, then you have to use CRDTs or equivalent. This can be hard, and you will have bugs, but since you retain the original events, you can always fix things up and recover the correct data, whereas when you get an SQL datastore into an inconsistent state you're generally SOL.

Re: The Reactive Monolith – How to Move from CRUD to Event Sourcing

#84

Post by dtech »

codeflo wrote: post #65
vegai_ wrote: post #56

Earlier quoted context omitted.

Two examples of systems that use a variant of event sourcing: - relational database systems (their journals and async replication in particular) - git Perhaps not so shit. Perhaps a bit practical. It's just a bad fit for some problems and architectologists like to say it's good for everything for obvious reasons. And it's should be pretty obvious that architecture doesn't matter at all if it's implemented poorly.

We have to be very precise here I think. Git doesn't really store file changes as events, rather, it's a long chain of state snapshots, in something like a persistent data structure. Sure, the history is all there, but so is the current state in its most efficient form. DB transaction logs might be considered event sourcing by some definition, but their use is very different. It's purely a technical trick. As a conse…

Event sourcing with snapshotting and deleting old events - like DB transaction logs - is still event sourcing. In fact I'd say it's the only way ES makes practical sense.

Re: The Reactive Monolith – How to Move from CRUD to Event Sourcing

#85

Post by kitd »

zaptheimpaler wrote: post #67

I used to be really excited about event sourcing but yeah, its usually just over-engineering. It appeals to the nerd in me because its a powerful & clean model - events are immutable, your entire database can theoretically be reconstructed at any point in time by just replaying an event log up to time X. In the ideal form its sort of the highest fidelity version of data storage, throwing nothing away, supporting all…

ES is the C in CQRS though. If you're using it for the Q too, then it's no wonder you'll quickly get in a bad way. It's specifically not designed for that.

Re: The Reactive Monolith – How to Move from CRUD to Event Sourcing

#86

Post by discreteevent »

holoduke wrote: post #29

I would like to see answers to the question: why move from Crud to event sourcing. Because I have seen at least 5 moderate projects trying to integrate ES. They all failed in the sense that either people didn't understand the code anymore, huge integration times, performance issues and even complete project cancellation because of all people walking away

I worked on a system years ago before event sourcing was even a term. We built the system on top of a relational database but stored every change and had a generic schema. We did this because we specifically had requirements where we needed to be able to quickly generate diffs between states of the system (differences in time and differences in version). So it was a natural solution to this problem and the benefits for us outweighed the costs. Of course we only used this approach for the part of our system that had these requirements.

Re: The Reactive Monolith – How to Move from CRUD to Event Sourcing

#87

Post by rswail »

Event sourcing (and CQRS in general) is often compared (as comments here are doing) to "simple" CRUD and why add the complexity.

The difference is that event sourcing, aggregates and CQRS force you to actually identify the entities and the external and internal events that cause them to change.

It forces you to do the analysis that you should do anyway, but is often lost or forgotten when it devolves to "simple" CRUD. You end up adding change logging, histories, materialized view updating etc etc, all of which come "for free" if you do the work to establish what you're trying to build.

WALs are not the same as a business level entity/event stream, WALs are event streams for the database, not the business logic you're using the database for.

Yes, it is more complex, but, in combination with techniques like domain driven design and creating ubiquitous languages, your business logic will end up actually being simpler and closer to reality.

Re: The Reactive Monolith – How to Move from CRUD to Event Sourcing

#88

Post by _3u10 »

nivertech wrote: post #72
_3u10 wrote: post #3

And then you realize that the CRUD db you moved away from operates on event sourcing (WAL) anyway. And thus you’re just doing inner platform effect.

Correct, and many applications are using WAL via CDC (Change Data Capture) to gain some of the benefits of the CQRS/ES. The problem is that the CRUD is only recording "What" change/mutation was done, but not "Why" by "Whom", and "When" it was done. With tailing WAL / CDC you can also capture the "When". This can be partially mitigated by adding audit fields like: Why (reason for change): created_bc ("because") / crea…

Yup a few fields gets you a long way, though I would recommend an audit log updated by triggers, rather than audit fields.

For example on my CRUD rails app, we use audit triggers in combination with setting a postgres local config variable with the username, the audit triggers pull the user info from the variables and record them into the DB. With a little more work I could probably also pull a backtrace of what line of code caused the trigger. Best part is because its all in the DB, if we rollback a transaction, the audit entries dont commit either.

Because this stuff gets set in the application controller as an around filter it's all completely transparent to the devs. They just make ActiveRecord calls and the database records the queries, user, etc that made the change as well as storing all the changed fields into the audit log. Even if it's one of those update all commands.

Re: The Reactive Monolith – How to Move from CRUD to Event Sourcing

#89

Post by rswail »

zaptheimpaler wrote: post #67

I used to be really excited about event sourcing but yeah, its usually just over-engineering. It appeals to the nerd in me because its a powerful & clean model - events are immutable, your entire database can theoretically be reconstructed at any point in time by just replaying an event log up to time X. In the ideal form its sort of the highest fidelity version of data storage, throwing nothing away, supporting all…

Materialized views are what you should be using, not reconstructing the state from the event log every time. That gives you access to SQL and everything to do with it.

And in reality, the event log does live forever in the real world outside of your system. Attributes of your aggregates from last year are still valid for events related to last year, even if they're now deprecated or no longer in use.

CQRS/ES is about system design that evolves. It evolves in a much cleaner and easier way if you follow it.

Is it perfect? No. There are some gnarly problems related to, for example, GDPR's "right to be forgotten", but that needs to be solved across database backups as well.

Re: The Reactive Monolith – How to Move from CRUD to Event Sourcing

#90

Post by rswail »

Flankk wrote: post #33
holoduke wrote: post #29

I would like to see answers to the question: why move from Crud to event sourcing. Because I have seen at least 5 moderate projects trying to integrate ES. They all failed in the sense that either people didn't understand the code anymore, huge integration times, performance issues and even complete project cancellation because of all people walking away

It's not practical. That's why all the arguments for it involve overstating and hyperbole. The results speak for themselves. It's shit.

That's bullshit. I've deployed a large CQRS/ES system that processes hundreds of thousands of business events each day.

The entities/aggregates have clearly defined state machines and the commands/events that change their state are clearly documented and simple.

The system is easy to understand, the business people understand it because we used their words to describe it.

We were able to model it and workshop it with them by literally having them each take on the role of one of the aggregates and pass paper "commands" and "events" between each other.

CQRS/ES is actually the way the world works. If you can't imagine a bunch of bureaucrats from Brazil, all sitting in a large room, passing files and paper forms between each other, then you're not understanding the actual work.

Post reply on HN