Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turning the database inside-out (2015)

martin.kleppmann.com

Post reply on HN
Print view View original

21–30 of 97 posts

Re: Turning the database inside-out (2015)

#21

Post by eyads »

joshlemer wrote: post #7

The thing I always get stuck on with these techniques is, how do you handle transactions which perform validations/enforce invariants on data when you’re just writing writes to a log and computing materialized views down the line? How can you do essentially, an “add item to shopping cart” if for example, users can only have max 10 items and so you need to validate that there aren’t already 10 items in the cart?

TLDR; You evaluate your preconditions and invariants before the event is published, using the current state of the aggregate.

Here's how that looks like in a DDD world:

* An aggregate is responsible for encapsulating business rules and emitting events

* An aggregate is responsible for maintaining the validity of its own state (ensuring invariants are valid)

* When a command/request is received, the aggregate first rehydrates its current state by replaying all previous events

* The aggregate then validates the command against its business rules using the current state

* Only if validation passes does the aggregate emit the new event

* If validation fails, the command is rejected (e.g., throws CartMaxLimitReached error)

Example flow: Command "AddItemToCart" arrives

>> System loads CartAggregate by replaying all its events

>> CartAggregate checks its invariants (current items count >> If valid: emits "ItemAddedToCart" event. If invalid: throws CartMaxLimitReached error

Re: Turning the database inside-out (2015)

#22

Post by dagss »

joshlemer wrote: post #7

The thing I always get stuck on with these techniques is, how do you handle transactions which perform validations/enforce invariants on data when you’re just writing writes to a log and computing materialized views down the line? How can you do essentially, an “add item to shopping cart” if for example, users can only have max 10 items and so you need to validate that there aren’t already 10 items in the cart?

All systems I have worked on like this has some concept of a version number for each entity / aggregate.

So you get that the account_balance was 100$ on version 10 of the account, and write an event that deducts 10$ on version 11.

If another writer did the same at exact same time, they would write an event to deduct 100$ at version 11. There will be a conflict and only one version will win.

This is exactly like any optimistic concurrency control also without event as the primary storage.

Didn't check if the system linked to supports this, I guess it might not? But this primitive seems quite crucial to me.

Re: Turning the database inside-out (2015)

#23

Post by dagss »

cwalv wrote: post #9
joshlemer wrote: post #7

The thing I always get stuck on with these techniques is, how do you handle transactions which perform validations/enforce invariants on data when you’re just writing writes to a log and computing materialized views down the line? How can you do essentially, an “add item to shopping cart” if for example, users can only have max 10 items and so you need to validate that there aren’t already 10 items in the cart?

You write the 'add item' event regardless, and when building the 'cart' view you handle the limit.

Well, assume the non-overdraftable bank account example instead then, what do you do then?

Re: Turning the database inside-out (2015)

#24

Post by Joker_vD »

> Databases are global, shared, mutable state. [...] However, most self-respecting developers have got rid of mutable global variables in their code long ago. So why do we tolerate databases as they are?

Because the world itself is a global, shared, mutable state (which, incidentally, is also a single source of truth) and databases were invented to mirror it (well, relevant parts of it) 1-to-1, or close to it. This style of "we use the database as a proxy of the physical world itself" is still pretty common, see e.g. the example with the shopping cart somewhere else in these comments.

Re: Turning the database inside-out (2015)

#27

Post by swiftcoder »

joshlemer wrote: post #7

The thing I always get stuck on with these techniques is, how do you handle transactions which perform validations/enforce invariants on data when you’re just writing writes to a log and computing materialized views down the line? How can you do essentially, an “add item to shopping cart” if for example, users can only have max 10 items and so you need to validate that there aren’t already 10 items in the cart?

I assume that shopping cart limit is a made-up example, but I'm curious what preconditions are you actually enforcing in the real world via DB transaction rollback?

Re: Turning the database inside-out (2015)

#28

Post by agumonkey »

fungiblecog wrote: post #25

Datomic

My first thought. And then I realized that the talk is from 2015, so I wonder if there was cross pollination between people at the time (datomic was released in 2012 according to wikipedia, so it's plausible it predates Martin's ideas but I don't know)

https://news.ycombinator.com/item?id=20937215

Re: Turning the database inside-out (2015)

#29

Post by agumonkey »

dagss wrote: post #5

We did this style on top of plain MSSQL. Each event would have a SQL table which is the primary storage. Then we have workers that listens to new data in tables and updates projections we needed. (Sometimes DB triggers but mostly async workers.) The main issue is "listening to new data in a SQL table". I wrote this code to achieve it in MSSQL (perhaps it is somehow built into postgres?): https://github.com/vippsas/ms…

I would love to know if other people in the industry (beside hickey/datomic) use the immutable log/stream + integrators. From my small experience in enterprise app: auditability and time travelling are always bolted on good old sql tables/snapshots after the fact and the pain is already baked in.

Re: Turning the database inside-out (2015)

#30

Post by reubenmorais »

agumonkey wrote: post #29
dagss wrote: post #5

We did this style on top of plain MSSQL. Each event would have a SQL table which is the primary storage. Then we have workers that listens to new data in tables and updates projections we needed. (Sometimes DB triggers but mostly async workers.) The main issue is "listening to new data in a SQL table". I wrote this code to achieve it in MSSQL (perhaps it is somehow built into postgres?): https://github.com/vippsas/ms…

I would love to know if other people in the industry (beside hickey/datomic) use the immutable log/stream + integrators. From my small experience in enterprise app: auditability and time travelling are always bolted on good old sql tables/snapshots after the fact and the pain is already baked in.

LinkedIn supposedly: https://engineering.linkedin.com/distributed-systems/log-wha...
Post reply on HN