Event Sourcing is Hard • Hacker News Board

Re: Event Sourcing is Hard

#121

Post by bruth » Mon, Feb 04, 2019, 11:59 AM UTC

Event sourcing is often described in the context of a rich domain model where every event has slightly different semantics with respect to the domain. As a result, there is often a need to adapt these event types over time as the domain changes. This could involve revising an existing event type (which is fine for adding new attributes for capturing more information) or creating a new event type to model some that has changed in the domain. If you simply incorrectly modeled the events for the domain, then you will have the same challenges of migrating a poorly designed database schema.

The arguments that Git and Datomic are both event sourced systems are good examples of successful application of this pattern. However, these are poor examples when it comes to event sourcing where the events are "domain events". In both Git and Datomic, the data model of the event are pre-defined. With Git you have a changeset and with Datomic you have datoms, both of which are composable by design and where every changeset and datom is equivalent (from a structural standpoint).

Applying event sourcing to an arbitrary domain model means every event is both semantically and possibly structurally different. That is, how event of type X affects the state compared to an event of type Y is different, where as applying two different datoms or changesets in Datomic or Git, respectively, change the state in the same way.

So I think there are two fundamental challenges faced when using event sourcing with a domain model. First is that the type and structure of events need to adapt to the domain (business, organization, etc). Datoms and changesets never change.. they are fixed and therefore don't have that challenge. Second, and related, is that as new events are introduced or existing ones are adapted, etc. the code that processes those events are unique as well (not just the model of the event itself). This challenge is exacerbated if there are multiple downstream consumers of this event stream where now there is "coupling" on the event/data side of things and likely semantic coupling as well.

Again, this is not something you run into with Datomic or Git simply because the data model of the events in those systems (the fundamental unit being left-folded) has a fixed data model and semantics.

Re: Event Sourcing is Hard

#122

Post by photonios » Mon, Feb 04, 2019, 12:08 PM UTC

koliber wrote: post #99

BlackFly wrote: post #94

Earlier quoted context omitted.

> Save all the events and you can always get to the latest state This isn't actually true and good event sourcing guides will point out why this isn't true. Event based systems are naturally racey, on a rerun of a stream of events the order in which the events is processed may change and you might therefore get a different result than the first time. For example, if you have 1 item remaining in inventory but two peop…

Preventing race conditions in message-queue based architectures is a neat challenge. Depending on your problem domain, there are some approaches that work really well. It is likely that these don't work for all cases though. The approach I've seen work well is to divide your messages into categories, such that events that belong to the same category MUST be handles in order, while events in different categories can b…

We dealt with a similar problem where I work.

At first we had ideas to do exactly what you described. We're using RabbitMQ, so partitioning is not built-in like with Kafka. We did some experiments with RabbitMQ's consistent hash exchange. Although the plugin does properly partition, we'd need to build our own stuff for rebalancing the partitions over workers as workers join and leave the pool, which happens often in our case because we sometimes process massive spikes and automatically scale up.

In the end we went with something more stupid. Each event gets a timestamp and we acquire a lock in Redis when we start processing the event. If the lock is already acquired, the event is rejected and put back into the queue with a delay. Before processing an event (and after acquiring the lock) we make sure that the timestamp of the event is newer than what we already processed. If it's older, we drop the event.

This approach trades some efficiency for correctness. Depending on your problem domain, this can work very well.

Re: Event Sourcing is Hard

#123

Post by BlackFly » Mon, Feb 04, 2019, 12:12 PM UTC

dtech wrote: post #98

BlackFly wrote: post #94

Earlier quoted context omitted.

> Save all the events and you can always get to the latest state This isn't actually true and good event sourcing guides will point out why this isn't true. Event based systems are naturally racey, on a rerun of a stream of events the order in which the events is processed may change and you might therefore get a different result than the first time. For example, if you have 1 item remaining in inventory but two peop…

There should be 1 point in the system that said "user x has ordered item a". That then becomes an event. The command from user x and y to order item a might be a race condition to see which command gets processed first, but once it has been turned into an event it's immutable and without race conditions. You should never replay commands.

Event systems do not require deduplication and/or strong ordering on the ingress. You can also solve the problem on the consumption side and there are often reasons to do this.

Plus you should be more forgiving of examples. Let us strengthen the example by saying that the inventory system lied and told us there were 2 items when there was only one. So you issue 2 commands, but when the events arrive at the fully automated warehouse, one randomly receives a null pointer exception and auto-heals into a refund, but which one?

Re: Event Sourcing is Hard

#124

Post by bonesss » Mon, Feb 04, 2019, 12:25 PM UTC

justjico wrote: post #120

jacques_chester wrote: post #23

Earlier quoted context omitted.

Kafka-oriented streaming folks talk about stream-table duality; the idea that one form can be expressed as the other. There is usually a little lip service paid to this idea before some heavy hints that actually , the stream is the true reality are dropped. My own view is that there are dimensions for any data of interest, expressing some ability to show an evolution of it. Frequently that dimension is time, or can b…

I believe this is where CQRS plays nice with “event sourcing”. You write all your events in one model, but can read them in multiple ones, that is if you can tolerate some read latency... and most systems are usually ok with that.

CQRS also alleviates a lot of the pain people experience around event sourcing and distributed systems with event evolution and manipulation. There are many cases where it's inappropriate for consuming services to be aware of the internal representation of events. Giving them a 'snapshot-centric' view of the data can be a simplification in both directions.

Re: Event Sourcing is Hard

#125

Post by bonesss » Mon, Feb 04, 2019, 12:29 PM UTC

ploxiln wrote: post #24

6t6t6t6 wrote: post #13

Earlier quoted context omitted.

EventSourcing is not a Framework, but a concept. The idea is to store not the current state of your app, but the transitions (events) that derive into the current state. Think about how git stores your source code as a series of commits. In theory it is a beautiful idea; in the real world, it is hard to implement.

In fact, git stores a full snapshot of your entire repo with every commit. It does not store diffs from the previous commit. When you do "git show " it generates a diff from the parent commit on the fly. There's a huge optimization though: it uses a content-addressed blob store, where everything is referenced by the sha1 of its contents. So if a file's contents is exactly the same between two commits, it ends up usin…

Gits bi-modal nature is a wonderful representation of a sanely architected Event Sourced system. When needed it can create a delta-by-delta view of the world for processing, but for most things most of the time it shows a file-centric view of the world.

IMO a well-factored event sourced system isn't going to feel 'event sourced' for most integrated components, APIs, and services because it's working predominately with snapshots and materialized views. For compex process logic, or post-facto analysis, the event core keeps all the changes available for processing.

Done right it should feel like a massive win/win. Done wrong and it's going to feel hard in all the wrong places :)

Re: Event Sourcing is Hard

#126

Post by bonesss » Mon, Feb 04, 2019, 12:33 PM UTC

hueving wrote: post #82

AndrewKemendo wrote: post #4

The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of. Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking. How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

The basic idea of event sourcing is to store the actions rather than the end state of the action. This allows actions to interleave and systems calculate the endstate from all of the actions. Think of ATMs. They don't update the balance of your bank account directly. They just record a debit against it and then the sum of all of your credits and debits is your balance. This avoids it having to have some kind of lock…

Databases that try to only reflect the entire truth as-is are like everyone working from a shared whiteboard.

Databases setup to event source, ie reflect the truth as it happened at each step, are like everyone working from a shared spreadsheet with change tracking.

It's an entire extra dimension that makes reconciliating problems of conflicting actions in disparate systems possible.

Re: Event Sourcing is Hard

#127

Post by fma » Mon, Feb 04, 2019, 12:36 PM UTC

kilburn wrote: post #103

bitcharmer wrote: post #83

Earlier quoted context omitted.

I agree that it's hard, however doable and pays benefits if you know what you're doing. I worked on 3 successful implementations for finance sector and we could replay a few million messages per second. Have a look at how we achieved that in LMAX: https://martinfowler.com/articles/lmax.html Sorry to say it, but clearly you must have been doing something wrong or employing event sourcing where it does not belong.

> LMAX's in-memory structures are persistent across input events, so if there is an error it's important to not leave that memory in an inconsistent state. However there's no automated rollback facility. As a consequence the LMAX team puts a lot of attention into ensuring the input events are fully valid before doing any mutation of the in-memory persistent state. They have found that testing is a key tool in flushin…

I honestly can't imagine bug free software, even in critical software development. Luckily for me I've worked on important apps, but if there's an issue there is time to trace and repair the data...not a system that has 6 million orders per second.

Re: Event Sourcing is Hard

#128

Post by jen20 » Mon, Feb 04, 2019, 12:58 PM UTC

coldtea wrote: post #119

olalonde wrote: post #19

Earlier quoted context omitted.

Things like Redux or Bitcoin are more or less event sourced systems. It's basically the idea of deriving your application state from a series of business events and storing those as a single source of truth. It's very appealing in theory but as the article explains, it's a bit more complicated in practice (e.g. dealing with consistency).

So somewhat akin to the Command pattern?

Somewhat but with an important difference: commands may fail. Events may not, since they are facts.

Although in 2019 one has to wonder.

Re: Event Sourcing is Hard

#129

Post by vemv » Mon, Feb 04, 2019, 1:19 PM UTC

willvarfar wrote: post #76

ilikehurdles wrote: post #56

Datomic is, at its core, an event sourced datastore, and it works really well. I don’t think event sourcing is something that should be solved in application space — you wouldn’t write a database from scratch for your products yet implementing an in-house event sourcing system is for some reason more acceptable. All of these stories of failure are stories of teams bandaiding event sourcing on top of other databases n…

I was thinking about Datomic when I read this, and yet came to exactly the opposite conclusion :) Datomic may be something like 'event sourcing' internally, but it works so well because it abstracts and hides the naked overreach that the article pounds against. Datomic users think of it as a DB, not a stream of events. So I don't think that Datomic users are following the 'event sourcing' model.

Fair point... but, events in the style of "User address changed" are very repetitive, wasteful, fragile etc.

So stopping seeing user-address-change as (primarily) an event is a win. One always can fall back later to perceiving user-address-change as an event.

For "real" events (e.g. "Transaction fraud detected") possibly I wouldn't use Datomic at all even if it was my primary DB. There are more suitable pub-sub systems.

And that results in a neat separation of concerns: data changes in one store, actual business-valuable events in the other.

Re: Event Sourcing is Hard

#130

Post by erpellan » Mon, Feb 04, 2019, 1:19 PM UTC

BlackFly wrote: post #94

manigandham wrote: post #33

Event Sourcing = everything that happens is an event. Save all the events and you can always get to the latest state, as well as what things look like at any time in the past. What an "event" is depends on your business domain and the granularity of processing. It's very common in enterprise apps with complex workflows (like payment processing or manufacturing). Good fit for functional programming techniques and make…

> Save all the events and you can always get to the latest state This isn't actually true and good event sourcing guides will point out why this isn't true. Event based systems are naturally racey, on a rerun of a stream of events the order in which the events is processed may change and you might therefore get a different result than the first time. For example, if you have 1 item remaining in inventory but two peop…

Event systems are not racey. An event is historical and immutable. The event source is a list of things that categorically happened. Commands can race each other, but the outcome (OrderPlaced vs OrderFailedBecauseItemOutOfStock) is definitive.

A common confusion is between commands (which can fail) and events (which have already happened). Replaying events should always result in the same state because they only reflect things that actually happened.

You can't replay commands because they have side effects.