Why Are People into Event Sourcing? • Hacker News Board

Re: Why Are People into Event Sourcing?

#51

Post by karmajunkie » Mon, Oct 03, 2016, 7:57 PM UTC

dragonwriter wrote: post #22

GundersenM wrote: post #20

Having been part of a project to rewrite a monolith e-commerce site into an event-sourced, domain driven, CQRS system, let me tell you in which situation that is not possible: when you already have data. Remember that in a DDD, ES, CQRS system, the event store is the single source of truth. If you already have data in a relational database, then the existing data is the source of truth. You can't have two sources of…

Conceptually, that's not really true: you just transform the pre-ES state into one or more events (in an basic accounting system, which is pretty much the simplest ES system, long-predating the name for the model, this is just creating "starting balance" entries as transactions.) In practice, that can be challenging, but it doesn't seem fundamentally more challenging than any other legacy data conversion effort.

I'd say the data isn't the problem. Its all the code that WRITES data that's the problem. finding every nook and cranny in your apps that do this, and very frequently across logical boundaries, is a challenging exercise at a minimum, even if you build projections that reproduced the old database state.

Re: Why Are People into Event Sourcing?

#52

Post by porker » Mon, Oct 03, 2016, 8:07 PM UTC

karmajunkie wrote: post #49

grandalf wrote: post #25

There is a lot that could be done to make event sourcing easier to work with... Imagine tooling that allowed an event stream to be used to create state for testing modules, crudlike helpers to allow crud-familiar developers to think that way at first, and workflows based on snapshots, rewind, etc. I think a model that used events that correlated to graph deltas rather than crud deltas would be the cat's ass, and many…

> Imagine tooling that allowed an event stream to be used to create state for testing modules, crudlike helpers to allow crud-familiar developers to think that way at first, and workflows based on snapshots, rewind, etc. i know where you're going with this, and i honestly believe its a terrible idea (not to be discouraging or rude—just experienced.) if your event streams contain mostly CRUD (possibly ANY) then you're…

> if your event streams contain mostly CRUD (possibly ANY) then you're most likely applying it incorrectly. Its not just a version history of your data.

Thanks for that. I'd made that mistake: I have a system which now needs to become distributed (a copy of it goes offline for a couple of weeks, and has to merge back into the main datastore) and keep a history of changes. It's currently CRUD backed by MySQL, and I'd latched onto event sourcing as what I'd need.

> The event type itself is data, which provides context and semantics over and above the notion of writes and deletes.

OK, going to have to get my head around that :)

Re: Why Are People into Event Sourcing?

#53

Post by zarkov99 » Mon, Oct 03, 2016, 8:14 PM UTC

I have been working with this sort of patterns for a while but I have yet to find good texts exploring the topic. Does anyone have book or paper recommendations for event sourcing? The stuff I have seen is mostly programmers reporting on something that worked on their particular domain. I am, looking for something more rigorous and comprehensive.

Re: Why Are People into Event Sourcing?

#54

Post by dragonwriter » Mon, Oct 03, 2016, 8:19 PM UTC

karmajunkie wrote: post #49

grandalf wrote: post #25

There is a lot that could be done to make event sourcing easier to work with... Imagine tooling that allowed an event stream to be used to create state for testing modules, crudlike helpers to allow crud-familiar developers to think that way at first, and workflows based on snapshots, rewind, etc. I think a model that used events that correlated to graph deltas rather than crud deltas would be the cat's ass, and many…

> Imagine tooling that allowed an event stream to be used to create state for testing modules, crudlike helpers to allow crud-familiar developers to think that way at first, and workflows based on snapshots, rewind, etc. i know where you're going with this, and i honestly believe its a terrible idea (not to be discouraging or rude—just experienced.) if your event streams contain mostly CRUD (possibly ANY) then you're…

> if your event streams contain mostly CRUD (possibly ANY) then you're most likely applying it incorrectly. Its not just a version history of your data. The event type itself is data, which provides context and semantics over and above the notion of writes and deletes.

Right. A good way to think about this is that as with rows in an RDBMS, events in an ES system are facts, and just as tables in an RDBMS define a category of facts with a particular shape, event-types in ES do the same thing. The difference is that whereas in an RDBMS the facts represented by rows can be general (and are often, in many designs, facts about the current state of the world), events are facts about a specific occurrence in the world rather than the state of the world (and the "state of the world" is an aggregate function of the collection of events.)

Re: Why Are People into Event Sourcing?

#55

Post by blowski » Mon, Oct 03, 2016, 8:31 PM UTC

btown wrote: post #4

Event sourcing isn't nearly as common knowledge among new programmers as the CRUD-one-row-per-entity pattern, and it really should be. I liken it to introducing version control for your data; when immutable updates are your canonical source, no matter how much the system behind them changes, or the business requirements change, and no matter how many teams are deriving different things from them in parallel, they can…

The CRUD one-row-per-pattern is common because it's enough for most projects. It works well with ORMs so you can build quickly and securely. And most of the time, performance isn't an issue and having a history of an entity is unnecessary.

I'm worried that event sourcing is going to become this year's over-applied design pattern with libraries in every language for every database with blog posts that recommend it be used on every project.

It's a good idea, very useful - in the right hands on the right projects. But it makes sense that junior devs normally use CRUD because that's normally the right solution. At least until better tools come along.

Re: Why Are People into Event Sourcing?

#56

Post by SEJeff » Mon, Oct 03, 2016, 8:32 PM UTC

Two must-read documents for those who want to learn more about this method of building reactive applications:

https://engineering.linkedin.com/distributed-systems/log-wha...

http://martinfowler.com/eaaDev/EventSourcing.html

Note that Martin's blog is what inspired the event bus in https://home-assistant.io, an open source home automation project I occasionally contribute to.

Re: Why Are People into Event Sourcing?

#57

Post by grandalf » Mon, Oct 03, 2016, 9:18 PM UTC

karmajunkie wrote: post #49

grandalf wrote: post #25

There is a lot that could be done to make event sourcing easier to work with... Imagine tooling that allowed an event stream to be used to create state for testing modules, crudlike helpers to allow crud-familiar developers to think that way at first, and workflows based on snapshots, rewind, etc. I think a model that used events that correlated to graph deltas rather than crud deltas would be the cat's ass, and many…

> Imagine tooling that allowed an event stream to be used to create state for testing modules, crudlike helpers to allow crud-familiar developers to think that way at first, and workflows based on snapshots, rewind, etc. i know where you're going with this, and i honestly believe its a terrible idea (not to be discouraging or rude—just experienced.) if your event streams contain mostly CRUD (possibly ANY) then you're…

I was thinking of making CRUD a specific event type that had meaning only in the context of changes to an instance of some schema. Of course, the most interesting events will not be CRUD oriented, but does this mean it's a mistake to include them at all, particularly if interacting with other systems that do use a CRUD metaphor for interaction and state must be synchronized?

Re: Why Are People into Event Sourcing?

#58

Post by karmajunkie » Mon, Oct 03, 2016, 9:37 PM UTC

grandalf wrote: post #57

karmajunkie wrote: post #49

Earlier quoted context omitted.

> Imagine tooling that allowed an event stream to be used to create state for testing modules, crudlike helpers to allow crud-familiar developers to think that way at first, and workflows based on snapshots, rewind, etc. i know where you're going with this, and i honestly believe its a terrible idea (not to be discouraging or rude—just experienced.) if your event streams contain mostly CRUD (possibly ANY) then you're…

I was thinking of making CRUD a specific event type that had meaning only in the context of changes to an instance of some schema. Of course, the most interesting events will not be CRUD oriented, but does this mean it's a mistake to include them at all, particularly if interacting with other systems that do use a CRUD metaphor for interaction and state must be synchronized?

I don't know if I'd be strong enough in my words to call it a "mistake" (there are few absolutes) but its a big stinky code smell, and it likely falls into that bucket of things you're going to regret one day sooner than you realize.

The thing is, DDD isn't easy. You're going to end up with someone on your team at some point that's inexperienced with it, and having a bunch of CRUD events in the schema already is going to entice them to add a few more because its easy and they're already there and oh hey what's a few more amongst friends?

Something else I've learned about this stuff is that because it works best in stable domains, it is also going to work best in domain services that have low code churn. Once you get it right, it tends to stay that way. But if you try to model with non-domain events, you're dramatically increasing the surface area that is subject to churn.

YMMV but I wouldn't do it.

Re: Why Are People into Event Sourcing?

#59

Post by EdSharkey » Mon, Oct 03, 2016, 9:40 PM UTC

Here's the term I wish was unfashionable with the kids: reshaping.

Did you spot all those command-to-query-to-event-to-log-to-storage data type conversions in those pretty diagrams? That's a whole bunch of needless reshaping of data as it flows through the system.

For each one of those data transformations to be successful, there has to be accurate communications between people and bug free code written in the data conversion and routing of messages through the system. All those moving parts make changing the system extremely painful, lotsa ripple effects - and every time you have to make a change to your events, you'd have a data migration project for any running event streams.

Naming things is hard too, and there's a lot more naming of entities needed in a CQRS-ES system.

I like all the promised benefits of a CQRS and ES, but I can't imagine a case where I'd take the risk of attempting it on anything but a toy project. Perhaps if I was on the version 5 rewrite project for an insanely profitable system where the requirements and design are completely understood up-front. I would need to grok some canonical example of a large, well-architected, well-implemented representative system before I would ever attempt to implement one.

Are there any non-toy examples of successful CQRS-ES with open source available to read? Did those projects go over-budget, and by how much? Would the authors of those examples still recommend the architecture now that they've gone through the experience?

Re: Why Are People into Event Sourcing?

#60

Post by karmajunkie » Mon, Oct 03, 2016, 9:44 PM UTC

porker wrote: post #52

karmajunkie wrote: post #49

Earlier quoted context omitted.

> Imagine tooling that allowed an event stream to be used to create state for testing modules, crudlike helpers to allow crud-familiar developers to think that way at first, and workflows based on snapshots, rewind, etc. i know where you're going with this, and i honestly believe its a terrible idea (not to be discouraging or rude—just experienced.) if your event streams contain mostly CRUD (possibly ANY) then you're…

> if your event streams contain mostly CRUD (possibly ANY) then you're most likely applying it incorrectly. Its not just a version history of your data. Thanks for that. I'd made that mistake: I have a system which now needs to become distributed (a copy of it goes offline for a couple of weeks, and has to merge back into the main datastore) and keep a history of changes. It's currently CRUD backed by MySQL, and I'd…

See the presentation by Greg Young I linked to in another comment. One thing he talks about towards the end is the application for occasionally connected clients, which is one I'm looking to tackle myself in the next several months. ES may very well serve your needs, but I'd take a step back from the CRUD model and really think about the domain model. Something I do a lot of times when I'm looking at moving away from CRUD in an app is creating a bunch of domain-based command classes, which take over the persistence job, and move the app layer towards talking only to those commands. Still CRUD under the covers, but now there's an abstraction layer above it, and as you get a better delineation of boundaries you start to see where the services will fall out of it and what areas might benefit from ES the most.

[1]https://www.youtube.com/watch?v=LDW0QWie21s