Why Are People into Event Sourcing? • Hacker News Board

Re: Why Are People into Event Sourcing?

#31

Post by taeric » Mon, Oct 03, 2016, 6:01 PM UTC

rreppel wrote: post #13

taeric wrote: post #3

As someone that has fallen for the "event sourcing" promise before, the article does a decent job explaining the promise. Not sure if it will be the next article, but the actual task of delivering on this work is where things break. Hard. The vast majority of the things you will ever program are pretty much guaranteed from one statement to the next. Hard boundaries, where things can fail, are often decently understoo…

Agree with the potential for complexity. Here's how we've dealt with this (on a so far / so good basis): Didn't seem necessary to go asynchronous and beef up on heavy infrastructure, so we went with simple in-thread, in memory message buses to start with. I think a lot of the perceived complexity of building event sourced systems comes because people start with the heavy plumbing instead of going YAGNI in order to ge…

This seems solid advice. I guess my main questions are:

* If you are in-memory and in-process, why bother with the events in the first place? (Simpler put, why not go with the simpler process based solutions?)

* If you are not testing distributed, how do you know you will be able to distribute?

In particular, there is a very large chance that you will have the same difficulty in replacing an in-memory/process solution that you would have had with a naive one.

And don't underestimate the amount of manpower you can get with success. Nor the amount of features that will not help get success.

Re: Why Are People into Event Sourcing?

#32

Post by jsprogrammer » Mon, Oct 03, 2016, 6:02 PM UTC

grandalf wrote: post #25

There is a lot that could be done to make event sourcing easier to work with... Imagine tooling that allowed an event stream to be used to create state for testing modules, crudlike helpers to allow crud-familiar developers to think that way at first, and workflows based on snapshots, rewind, etc. I think a model that used events that correlated to graph deltas rather than crud deltas would be the cat's ass, and many…

Only issue with building it is that it probably already exists somewhere (though, we may not be able to access it).

Re: Why Are People into Event Sourcing?

#33

Post by GundersenM » Mon, Oct 03, 2016, 6:02 PM UTC

dragonwriter wrote: post #29

GundersenM wrote: post #27

Earlier quoted context omitted.

Sure, if the existing DB is simple, that is straight forward, but remember that likely this is a monolith that is so bad that even management have agreed that it needs to be rewritten. Likely there are lots of DB tables with foreign keys and relations (sometimes documented and enforced, most often not). This means you can't really convert the entire database into an event sourced system, as that means converting all…

> Sure, if the existing DB is simple, that is straight forward, but remember that likely this is a monolith that is so bad that even management have agreed that it needs to be rewritten. Yeah, but that's not a "converting legacy data to ES" problem, that's a "converting legacy data to any non-broken thing" problem. > This means you can't really convert the entire database into an event sourced system, as that means c…

Yup, agree, this is the problem of having a legacy monolith RDBMS that needs to be rewritten and split apart. It's tempting to throw every new fancy technology at the problem when that is suddenly an option, but it's better to focus on the goal of splitting it apart only. If you have split it out, and it's now simple to convert to ES CQRS, then you are probably in a situation where you don't need to do that, as it works quite well.

Re: Why Are People into Event Sourcing?

#34

Post by taeric » Mon, Oct 03, 2016, 6:06 PM UTC

biot wrote: post #26

kazagistar wrote: post #18

I've tried working out how to move to an event sourcing system, but I always struggle with locking behavior. Do you just have to invent your own locking mechanisms on top of event sourcing?

Combine this with the actor model (using Akka or similar) gives you guaranteed "one message at a time" processing and you don't have to deal with locks.

I question any amount of guarantees around "one message" anything. There might be this guarantee per actor, but you have no such guarantee per system. And, assuming a real system, this will be a problem.

So, you get to pick, "at most once" or "at least once." And then you need to build your system to act accordingly.

Re: Why Are People into Event Sourcing?

#35

Post by hew » Mon, Oct 03, 2016, 6:16 PM UTC

taeric wrote: post #34

biot wrote: post #26

Earlier quoted context omitted.

Combine this with the actor model (using Akka or similar) gives you guaranteed "one message at a time" processing and you don't have to deal with locks.

I question any amount of guarantees around "one message" anything. There might be this guarantee per actor, but you have no such guarantee per system. And, assuming a real system, this will be a problem. So, you get to pick, "at most once" or "at least once." And then you need to build your system to act accordingly.

Slightly shooting from the hip here (as I'm still learning).

Low-ish volume- design your system such that data flows [at the relevant crucial points] through a single actor to ensure proper concurrency.

High volume- trickier but I think same idea in principle. First thought that comes to mind here is the new GenStage stuff in Elixir.

Re: Why Are People into Event Sourcing?

#36

Post by taeric » Mon, Oct 03, 2016, 6:22 PM UTC

hew wrote: post #35

taeric wrote: post #34

Earlier quoted context omitted.

I question any amount of guarantees around "one message" anything. There might be this guarantee per actor, but you have no such guarantee per system. And, assuming a real system, this will be a problem. So, you get to pick, "at most once" or "at least once." And then you need to build your system to act accordingly.

Slightly shooting from the hip here (as I'm still learning). Low-ish volume- design your system such that data flows [at the relevant crucial points] through a single actor to ensure proper concurrency. High volume- trickier but I think same idea in principle. First thought that comes to mind here is the new GenStage stuff in Elixir.

I'm not sure what you are suggesting. There is no getting around "at most/at least once." You can shift the posts, some, but at some point that is your choice. This is a good read on the problem: http://bravenewgeek.com/you-cannot-have-exactly-once-deliver...

Re: Why Are People into Event Sourcing?

#37

Post by kazagistar » Mon, Oct 03, 2016, 6:27 PM UTC

jessaustin wrote: post #19

kazagistar wrote: post #18

I've tried working out how to move to an event sourcing system, but I always struggle with locking behavior. Do you just have to invent your own locking mechanisms on top of event sourcing?

ISTM event sourcing actually avoids many locking problems, since it's essentially "write-only". Of course every event write should be atomic, but that seems easier than making updates atomic?

When a certain set of events occurs (the files arrive etc) I want to kick off one and only one batch processor task. This is accomplished with a transaction and a write lock in an sql database, but when trying to use event sourcing it ends up requiring a 2 step "intent to run" event before running or some out of band synchronization.

Re: Why Are People into Event Sourcing?

#38

Post by jnbiche » Mon, Oct 03, 2016, 6:32 PM UTC

kazagistar wrote: post #18

I've tried working out how to move to an event sourcing system, but I always struggle with locking behavior. Do you just have to invent your own locking mechanisms on top of event sourcing?

Read about how LMAX achieved 6 million transactions per second using a ring buffer-based concurrency architecture called disruptor, all on a single thread and without locks. Event sourcing plays a big role in their architecture [0].

0. http://martinfowler.com/articles/lmax.html

Re: Why Are People into Event Sourcing?

#39

Post by jnbiche » Mon, Oct 03, 2016, 6:38 PM UTC

kazagistar wrote: post #37

jessaustin wrote: post #19

Earlier quoted context omitted.

ISTM event sourcing actually avoids many locking problems, since it's essentially "write-only". Of course every event write should be atomic, but that seems easier than making updates atomic?

When a certain set of events occurs (the files arrive etc) I want to kick off one and only one batch processor task. This is accomplished with a transaction and a write lock in an sql database, but when trying to use event sourcing it ends up requiring a 2 step "intent to run" event before running or some out of band synchronization.

I'm not sure you can achieve good event sourcing performance using a regular database engine. Better to view it like writing logs.

If you really must expose an SQL API, perhaps you could read the journal on another thread or process and then make changes to the db based on the incoming "diffs" that the threads determines from the journal?

Re: Why Are People into Event Sourcing?

#40

Post by dragonwriter » Mon, Oct 03, 2016, 6:38 PM UTC

GundersenM wrote: post #33

dragonwriter wrote: post #29

Earlier quoted context omitted.

> Sure, if the existing DB is simple, that is straight forward, but remember that likely this is a monolith that is so bad that even management have agreed that it needs to be rewritten. Yeah, but that's not a "converting legacy data to ES" problem, that's a "converting legacy data to any non-broken thing" problem. > This means you can't really convert the entire database into an event sourced system, as that means c…

Yup, agree, this is the problem of having a legacy monolith RDBMS that needs to be rewritten and split apart. It's tempting to throw every new fancy technology at the problem when that is suddenly an option, but it's better to focus on the goal of splitting it apart only. If you have split it out, and it's now simple to convert to ES CQRS, then you are probably in a situation where you don't need to do that, as it wo…

> It's tempting to throw every new fancy technology at the problem when that is suddenly an option, but it's better to focus on the goal of splitting it apart only.

Splitting it apart involves:

(1) Dividing the data and functionality into a legacy component and a new-implementation component,

(2) Making changes to the DB and application code for the legacy component,

(3) Implementing the new-implementation component.

In a monolith that you are breaking apart, the reusability of legacy code for the new-implementation component is likely to be low (you'll actually likely have to do extensive changes to the larger "legacy component" as well, but the reusability should be somewhat higher there.)

You have to use some technology for the new implementation component, and what you should aim for is whatever is the best fit for the job, whether it is similar to what existed before or not.

> If you have split it out, and it's now simple to convert to ES CQRS, then you are probably in a situation where you don't need to do that, as it works quite well.

I disagree. The hard part of converting to ES/CQRS for the components that are broken out ("new implementation" components, not the "legacy" reduced-monolith) is done in the analysis phase of what you are breaking out. Once that is done, implementation in a ES/CQRS manner is fairly straightforward, since defining the events that the component will handle is a core part of analysis, as is defining the impacts those events have on stored, reportable data (the query side of CQRS).