Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turning the database inside-out (2015)

martin.kleppmann.com

Post reply on HN
Print view View original

91–97 of 97 posts

Re: Turning the database inside-out (2015)

#91

Post by mrkeen »

Izkata wrote: post #88
mrkeen wrote: post #61

Earlier quoted context omitted.

> I worked with a feature that used this approach once. You work with many features that use this approach. https://www.dbi-services.com/blog/oracle-log-writer-and-write-ahead-logging/ https://git-scm.com/docs/git-reflog https://dev.mysql.com/blog-archive/mysql-8-0-new-lock-free-scalable-wal-design/ https://www.postgresql.org/docs/current/wal-intro.html https://www.sqlite.org/wal.html https://www.amazon.science/blog/…

From this: > was cumbersome and extremely slow. I'm pretty sure those sit somewhere in between the two sides GP was describing: All of your links have "current state" as the interface and changes are logged as they're applied. The system they worked with apparently has the log as a first-class citizen and no "current state" interface, while what they wanted was a "current state" interface with snapshots.

Understood, but when we look at all the databases and source-control products, we're standing on the customer side. We get to experience stuff that just works, because the people on the development side opted for write-AHEAD-log and/or copy-on-write, as opposed to write-afterward-log or write-but-store-backups-in-case-something-goes-wrong.

When we're on the developer side of things, we get to choose whether to CRUD or to go eventy. If we want to build systems as good as those DBs, we should do what they did (events) rather than what they say (insert/update/delete). It's unfortunate that there's not much upstream tech to help us with this (it's pretty much just kafka) and that people who choose to use kafka get accused of 'resume-driven-development', etc.

>> Querying current state, which was 99% of the usecases, was cumbersome and extremely slow.

It's an easy fix if the events are there. Just cache the current state. It sounds glib but it's one of the first-principles of event-sourcing that makes me choose it over CRUD. If your events are immutable, they can be shared. If they can be shared, they can be folded into a fast-current-state-view. If they are instead mutable (or non-existent)), you can't share them unless you also have a plan on how to sync them (which no-one gets around to and probably overlaps on some classical impossibility result like two-generals.)

Re: Turning the database inside-out (2015)

#92

Post by mrkeen »

zbentley wrote: post #87
mrkeen wrote: post #58

Earlier quoted context omitted.

Two points: * Technically the data is probably there, but I really don't think you want back-up ops invoked by your REST call to /getUserHistory/. Is it even possible to mix old data and new data within the same SQL expression? * The DB is still a god object at the centre of your system. It doesn't give you consistency across partner systems and end users. If a partner sends the event CustomerBanned(2025-02-04, 1234)…

> If you just blindly write the event, then you always know that fact about customer 1234 in any future query. Unless the write times out or the DB is down for maintenance when the event arrives. Sure, you could block acknowledgement of the event until the event log receives it, but can your DB handle synchronous write volume from however many people are out there sending events? If your RPC servers listening for eve…

> Down that road lies "let's put every event in a fast message bus with higher insert volume and availability than the database, and feed that into the DB asynchronously", hence Kafka and friends.

Yes, this is the way. Which is why it's not

>> Isn't this essentially how a modern transactional database works anyway

and needed a comment.

Re: Turning the database inside-out (2015)

#93

Post by alecco »

kragen wrote: post #84
alecco wrote: post #82

Earlier quoted context omitted.

Event sourcing is a PITA, from experience. Something like Datomic makes a lot more sense: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...

Thanks! Does the article accurately describe the PITA in your experience? Because it seems to say that it's separable from the core architectural principle of event sourcing.

In my experience most event-sourcing was implemented as storing versions of objects (it came from the OOP camp). All the consistency checks had to be done manually in imperative code across countless classes. Many large investment banks use it. And then all the actual DB stuff has to be exported to SQL or other actual database engines to be processed properly.

Re: Turning the database inside-out (2015)

#94

Post by mrkeen »

arialdomartini wrote: post #90
jdkoeck wrote: post #86

Earlier quoted context omitted.

Git is still event sourced, it’s just there is only one kind of event (a commit), and its payload is the whole state ¯\_(ツ)_/¯

Eh eh, this is an interesting point of view, but it’s really not like this. Take the case of the event of “deleting a file”. There has been an interesting discussion between Linus and the orher developers, when Git was being d initially esigned: some of them wanted to capture and track this event. Linus firmly rejected the whole idea of track events, providing very solid arguments http://web.archive.org/web/202001170…

jdkoeck is right.

Storing-events-and-calculating-state is the dual of storing-state-and-calculating-diffs. It's an implementation detail, and if it works well, who cares?

I personally use git as an event log.

When I commit something, I write out a description of what I changed, not the current state of the whole repository (even if that's what a git hash is supposed to represent).

When I pull or push, I send and receive only diffs, not the state of the multi-GB repo.

When I rebase, I grab the latest diffs that went into master and put them under my feature branch diffs.

When I run 'git show {commit}', I see a diff, not a state.

It doesn't matter which way it is in any case: the evergreen debate about fast-mutations vs slow-immutations is that it's easier and faster to just mutate someone's bank balance in-place, rather than the slow way of appending transactions, or just quickly delete a bad transaction rather than slowly append a refund event to the log.

Taking that debate over to git land, "it would be faster" to just mutate the state of the codebase in some single, central location rather than the "slow way of sending commits" back and forth - regardless of whether those commits are technically states or diffs.

Re: Turning the database inside-out (2015)

#95

Post by Joker_vD »

mwgalloway wrote: post #56
Joker_vD wrote: post #45

Earlier quoted context omitted.

While the world can tentatively be conceived to be like that, it is not nearly "just as easily". Lots of those "immutable facts" can't be realistically discovered: e.g. good luck recovering "this blackboard had a drawing of a cat and a bird until five minutes ago when it was cleared with a wet sponge" if you weren't there in time. The approach with mutable state fosters this destructible nature of many things upon yo…

Ok, but why should I even care about what was on this hypothetical blackboard? Do you have any real world examples with business or technical significance where this "cope with it, somehow" approach to mutability is a clear win?

You've just stated it, no? If you only care what is on that blackboard right now, not what was there — why even bother storing knowledge about its previous states?

Re: Turning the database inside-out (2015)

#96

Post by kragen »

alecco wrote: post #93
kragen wrote: post #84

Earlier quoted context omitted.

Thanks! Does the article accurately describe the PITA in your experience? Because it seems to say that it's separable from the core architectural principle of event sourcing.

In my experience most event-sourcing was implemented as storing versions of objects (it came from the OOP camp). All the consistency checks had to be done manually in imperative code across countless classes. Many large investment banks use it. And then all the actual DB stuff has to be exported to SQL or other actual database engines to be processed properly.

No true Scotsman! No true Scotsman!

(Which is to say, that sure isn't what I thought "event sourcing" was.)

Re: Turning the database inside-out (2015)

#97

Post by eyads »

cwalv wrote: post #9
joshlemer wrote: post #7

The thing I always get stuck on with these techniques is, how do you handle transactions which perform validations/enforce invariants on data when you’re just writing writes to a log and computing materialized views down the line? How can you do essentially, an “add item to shopping cart” if for example, users can only have max 10 items and so you need to validate that there aren’t already 10 items in the cart?

You write the 'add item' event regardless, and when building the 'cart' view you handle the limit.

add_item is not an event, rather a command/ request that is yet to be validated. item_added is the event = a fact that was 'allowed to happen' by the system.

Keeping commands in a persistent store is a matter of choice but not necessary. I've seen people doing command sourcing and calling it event sourcing.

Post reply on HN