Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What they don’t tell you about event sourcing

medium.com

Post reply on HN
Print view View original

41–50 of 81 posts

Re: What they don’t tell you about event sourcing

#41

Post by tomerbd »

theptip wrote: post #40

I don't see much discussion of event-sourcing simply using a SQL database (i.e. skipping the CQRS part). This would allow you to keep your CP (strongly-consistent) semantics. While this clearly wouldn't work in high-volume cases (i.e. where you _actually_ need CQRS), it seems like this would be the simplest option for many systems. I see a lot of articles advocating for immediately jumping into CQRS, which seems like…

Don't you still need a queue in your example? do you lock the whole table when you insert a new record to the table? can you elaborate how this solves consistency?

Re: What they don’t tell you about event sourcing

#42

Post by jessaustin »

LandR wrote: post #5
ulcica wrote: post #2

They didn't even tell me what event sourcing is.

As I understant it: Consider you have a database where you store the account balance. If you want to update the account balance you might update the row for that customer, e.g. tblAccounts ------------ | AccountHolderId | AccountBalance | update tblAccounts set AccountBalance = @NewAccountBalance; In an EventSource database instead you wouldn't update the AccountBalance column. You would store something like: Account…

One would expect all DB-based live accounting systems to store balances as a series of transactions? Even if you have a view that sums over that or a regular process to collapse old transactions, there's really no other sensible way to do it. Concurrent updates to the same row will always cause problems.

...Having written the above, I just examined the ERD for the COTS customer system in the office I'm in now. It stores not just balances but aged balances directly in the main customer table. Good grief.

Re: What they don’t tell you about event sourcing

#43

Post by [deleted] »

theptip wrote: post #40

I don't see much discussion of event-sourcing simply using a SQL database (i.e. skipping the CQRS part). This would allow you to keep your CP (strongly-consistent) semantics. While this clearly wouldn't work in high-volume cases (i.e. where you _actually_ need CQRS), it seems like this would be the simplest option for many systems. I see a lot of articles advocating for immediately jumping into CQRS, which seems like…

[deleted]

Re: What they don’t tell you about event sourcing

#44

Post by jmull »

vorpalhex wrote: post #32
jmull wrote: post #30

Earlier quoted context omitted.

I don't think you necessarily have to have all this in a useful CQRS system. E.g., here's a real-world example of a general pattern in which CQRS is pretty simple and useful: A walking/running app which tracks distance, time, and other relevant information over the course of a workout. It collects a series of events like "location changed at time X", "user started workout", "user paused workout" etc., over the course…

That's not "Command Query Responsibility Segregation" (CQRS). That's modeling your data as a time series - which is a totally valid and perfectly useful model in many cases, but has nothing to do with the architectural pattern known as CQRS. Martin Fowler gives the following simple definition of CQRS: > At its heart is the notion that you can use a different model to update information than the model you use to read…

But what is the distinction between what Fowler describes and what I describe? There's not really a contradiction with what you describe either.

I have a distinct write store and read store, with very different models. As you say, the write store is the source of truth. Since the read store is updated synchronously with the write store there's no need for a queue between them. Indeed there are also multiple projections for different client needs (e.g. the pace chart, vs. workout progress), though in this case I don't generally need them until after the sequence of events is complete, which simplifies things.

Maybe we're just have a pointless debate on semantics, in which case, never mind.

It's just that I see this as a quite valuable pattern without necessarily bringing distributed stores into it. Indeed, part of its value is that you can start simple and later extend it to a scaling distributed system without disrupting the whole pattern.

Re: What they don’t tell you about event sourcing

#45

Post by tonyhb »

joefreeman wrote: post #10

Good article. I've spent the last year migrating to an event sourced system, so thought I'd share some thoughts. On the eventual consistency point, I've found you can get quite far with having the read model managing the race condition. This probably doesn't work everywhere, but in our system, multiple users can accept an invitation, so we have something like `InvitationAccepted{invitation_id, user_id}`. It's possibl…

I've been planning out an event sourcing like system for our healthtech startup, and you're definitely right re namespacing and versioning.

I explicitly keep a version in the event - which is a date. It's similar in how Stripe versions their API. We're also planning to handle the events in the same way as stripe's API: each event has side effects; the side effects may change depending on the version and each version has its own application logic (cascading, so you can have 2018-08-01 run all of 2018-07-30 plus its own changes).

This lets us replay events as they happened, run an event using two different versions and perform only the diffs etc.

Our system is probably not a typical CQRS/event sourcing setup.

The event system itself idempotent: you take an event with all input data necessary to run the event (form data, necessary current state), so the system can run independently. This means that every event is typed such that the input data dictates what is necessary.

The event handler validates the event, returning errors if necessary.

Then, the event handler runs all side effects and returns operations to perform: update model X attributes to Y, insert new record Z.

In effect, we go:

User -> API -> (generate event) -> Event Handler -> (error|response) -> save event and side effects in a transaction.

This means our DB is a cache of the event and all previous data, so we're not really event sourcing — we're audit trailing.

The main benefits are:

1. Medical records are complex and we always need audit trails.

2. If a doctor submits a prescription, we can show all side effects that happened for visibility (ie. this triggered a lab task, push notification, sent this message). We can verify this in the UI and see what happened for each patient without relying on assumptions.

3. Engineers know that the API produces events and can look up exactly the side effects that happen when an event occurs (we're using Rails for the API logic right now and this isn't always obvious).

4. We can ensure that we validate when an event happens based on input and current state without complex code, catching edge cases.

5. We can choose to save the event and side effects or not. This lets us "preview" actions or "replay" actions without actually changing any world state (you toggle a "test" flag in the event which also means the event handlers know not to trigger outside side effects).

6. The "side effects" response from the event handler can be sent to a websocket observable and consumed by frontends, ensuring that the doctor UI always has an up to date version of patient data.

Random thoughts:

- It's really just a framework for the logic of an application controller that's typed and ensures everything is consistent. Plus, similar to Stripe, it allows us to version events and write migrations/upgrade paths etc.

- What about conflicts? We have a plan to use hashes of the previous data to ensure consistency with medical records: if you're modifying fields A, B, C, you send over a hash of the previous data for A, B, C alongside the request. If the event handler can't verify the hash the data must've changed in the meantime.

----

We're producing events now but the handlers aren't yet in place, so this is currently still being planned. Essentially, we're using the API as authentication, authorization, routing/HTTP management, transaction/database management while the event/controller logic is being placed into a structured framework to ingest form data, current state and produce output.

Re: What they don’t tell you about event sourcing

#46

Post by theptip »

tomerbd wrote: post #41
theptip wrote: post #40

I don't see much discussion of event-sourcing simply using a SQL database (i.e. skipping the CQRS part). This would allow you to keep your CP (strongly-consistent) semantics. While this clearly wouldn't work in high-volume cases (i.e. where you _actually_ need CQRS), it seems like this would be the simplest option for many systems. I see a lot of articles advocating for immediately jumping into CQRS, which seems like…

Don't you still need a queue in your example? do you lock the whole table when you insert a new record to the table? can you elaborate how this solves consistency?

I don't think you need a separate queue; if you have an "Events" table then you can just write everything there.

It solves the consistency problem because you can create your event inside a transaction, which will rollback if another event touching the same source is created simultaneously.

E.g. if you have these incompatible events in a ledger:

CreditAccount(account_id=123, amount=100)

DebitAccount(account_id=123, amount=60)

DebitAccount(account_id=123, amount=60)

You'd want one of the debit transactions to fail, assuming you want to preserve the invariant that the account's balance is always positive. You could put the `account_id` UUID as an `Event.source` field, which would allow you to lock the table for rows matching that UUID.

Re: What they don’t tell you about event sourcing

#47

Post by Yokohiii »

lscharen wrote: post #16

The last section on Operational Flexibility and the inability to change the event history raises a very good point. Like most of the issues, the solution requires experience to know when you are at the Goldilocks point (Just Right). This specific issue has a lot in common with managing database migrations in django or any other migration system. The ideal situation is to create migrations that can always be rolled ba…

Idk why people are saying that ES data is always immutable. They can be by default sure, but if a facility is useful to change the history, why not?

Re: What they don’t tell you about event sourcing

#49

Post by Yokohiii »

theptip wrote: post #46
tomerbd wrote: post #41

Earlier quoted context omitted.

Don't you still need a queue in your example? do you lock the whole table when you insert a new record to the table? can you elaborate how this solves consistency?

I don't think you need a separate queue; if you have an "Events" table then you can just write everything there. It solves the consistency problem because you can create your event inside a transaction, which will rollback if another event touching the same source is created simultaneously. E.g. if you have these incompatible events in a ledger: CreditAccount(account_id=123, amount=100) DebitAccount(account_id=123, a…

If your idea in the example is that the second "debit" is created by another transaction while your transaction is in progress, then this will not work out. Firstly it requires a dirty read, which is nothing I would rely on in a transaction. Secondly, if the dirty read works, assuming the outcome of several rows is just a read operation, which forces you to rollback on the client and still leaves a window for inconsistencies if you decide to commit. Maybe SELECT .. FOR UPDATE can do a trick here, but that is like giving up life.

To round this up: RDBMS are bad for queue like read semantics. All you can do is polling. Which is even worse if you end up being lock heavy.

Re: What they don’t tell you about event sourcing

#50

Post by fusiongyro »

spitfire wrote: post #35
lscharen wrote: post #16

The last section on Operational Flexibility and the inability to change the event history raises a very good point. Like most of the issues, the solution requires experience to know when you are at the Goldilocks point (Just Right). This specific issue has a lot in common with managing database migrations in django or any other migration system. The ideal situation is to create migrations that can always be rolled ba…

Basically implement double entry accounting. I think ideally you want something like Rich Hickey speaks about when he speaks of Datomic. An append only database. You can see what the previous values for that row were, along with schema changes.

I worry about GDPR with respect to Kafka and other append-only databases.
Post reply on HN