Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turning the database inside-out (2015)

martin.kleppmann.com

Post reply on HN
Print view View original

81–90 of 97 posts

Re: Turning the database inside-out (2015)

#81

Post by arialdomartini »

agentultra wrote: post #66

This is also known as event sourcing [0] and is a common pattern used inside of databases, in git, in lots of popular software. I don't generally recommend it for every application as the tooling is not as well integrated as it is in an RDBMS and the data model doesn't fit every use-case. However, if you have a system that needs to know "when" something happened in an on-going process, it can be a very handy architec…

I saw other times Git being related to event sourcing, but the argument is wrong.

Most of the VCSs before Git (RCS, CVS, SVN) used to store deltas and to rebuild the state reapplying them.

The very reason why Git took them by the storm is exactly because, on the contrary, Git does not store deltas but snapshots. Each commit is not there collection of the occurred chances but a complete snapshot of the whole project. Git is very efficient in reusing the blob objects to save space, but it’s still a whole snapshot. The occurred changes are not stored, and they are calculated on demand.

The very opposite of event sourcing, where it’s the state to be calculated and the occurred changes / events to be stored.

Git is really the demonstration that for code versioning state sourcing is way more efficient than event sourcing.

Re: Turning the database inside-out (2015)

#83

Post by dang »

Related:

Turning the database inside out (2014) [video] - https://news.ycombinator.com/item?id=41664271 - Sept 2024 (1 comment)

Turning the database inside-out with Apache Samza (2015) - https://news.ycombinator.com/item?id=13581096 - Feb 2017 (30 comments)

Turning the database inside-out with Apache Samza - https://news.ycombinator.com/item?id=9145197 - March 2015 (64 comments)

Re: Turning the database inside-out (2015)

#84

Post by kragen »

alecco wrote: post #82
kragen wrote: post #78

It seems like event sourcing keeps gaining mindshare.

Event sourcing is a PITA, from experience. Something like Datomic makes a lot more sense: https://vvvvalvalval.github.io/posts/2018-11-12-datomic-even...

Thanks! Does the article accurately describe the PITA in your experience? Because it seems to say that it's separable from the core architectural principle of event sourcing.

Re: Turning the database inside-out (2015)

#85

Post by zbentley »

timacles wrote: post #68
mike_hearn wrote: post #50

Earlier quoted context omitted.

Depends which industry. If you look at a lot of non-tech industry then they'll use a commercial DB with all those features in place already, rather than hacking up their own data layer. A few years ago I spent some time in the enterprise finance space, and learned some unfashionable tech you don't see talked about on Hacker News much. It left me with a new appreciation for what goes on there. A staggering amount of t…

Who besides Oracle offers this stuff though? Yeah Oracle has a bunch of nice features, it also costs a gajillion dollars that no one besides a large enterprise can afford.

Debezium is popular in this space, though it does bring more tools into the CDC/CQRS stack: https://debezium.io/

Re: Turning the database inside-out (2015)

#86

Post by jdkoeck »

arialdomartini wrote: post #81
agentultra wrote: post #66

This is also known as event sourcing [0] and is a common pattern used inside of databases, in git, in lots of popular software. I don't generally recommend it for every application as the tooling is not as well integrated as it is in an RDBMS and the data model doesn't fit every use-case. However, if you have a system that needs to know "when" something happened in an on-going process, it can be a very handy architec…

I saw other times Git being related to event sourcing, but the argument is wrong. Most of the VCSs before Git (RCS, CVS, SVN) used to store deltas and to rebuild the state reapplying them. The very reason why Git took them by the storm is exactly because, on the contrary, Git does not store deltas but snapshots. Each commit is not there collection of the occurred chances but a complete snapshot of the whole project.…

Git is still event sourced, it’s just there is only one kind of event (a commit), and its payload is the whole state ¯\_(ツ)_/¯

Re: Turning the database inside-out (2015)

#87

Post by zbentley »

mrkeen wrote: post #58
chikere232 wrote: post #36

Isn't this essentially how a modern transactional database works anyway? All mutations end up in the Write Ahead Log (WAL) and you can replicate or back up that to be able to recover the state at a point in time?

Two points: * Technically the data is probably there, but I really don't think you want back-up ops invoked by your REST call to /getUserHistory/. Is it even possible to mix old data and new data within the same SQL expression? * The DB is still a god object at the centre of your system. It doesn't give you consistency across partner systems and end users. If a partner sends the event CustomerBanned(2025-02-04, 1234)…

> If you just blindly write the event, then you always know that fact about customer 1234 in any future query.

Unless the write times out or the DB is down for maintenance when the event arrives.

Sure, you could block acknowledgement of the event until the event log receives it, but can your DB handle synchronous write volume from however many people are out there sending events? If your RPC servers listening for events are unavailable, do you trust event senders to retry when they're back?

Down that road lies "let's put every event in a fast message bus with higher insert volume and availability than the database, and feed that into the DB asynchronously", hence Kafka and friends.

Re: Turning the database inside-out (2015)

#88

Post by Izkata »

mrkeen wrote: post #61
Swizec wrote: post #34

Earlier quoted context omitted.

> if you explicitly model time as a dimension it can instead be seen as a sequence of transitions from immutable state to immutable state, an accumulation of events over time, which fits the log abstraction perfectly I worked with a feature that used this approach once. It even made sense for the feature (an immutable history log of patient chart data). It was absolute hell to work with. Querying current state, which…

> I worked with a feature that used this approach once. You work with many features that use this approach. https://www.dbi-services.com/blog/oracle-log-writer-and-write-ahead-logging/ https://git-scm.com/docs/git-reflog https://dev.mysql.com/blog-archive/mysql-8-0-new-lock-free-scalable-wal-design/ https://www.postgresql.org/docs/current/wal-intro.html https://www.sqlite.org/wal.html https://www.amazon.science/blog/…

From this:

> was cumbersome and extremely slow.

I'm pretty sure those sit somewhere in between the two sides GP was describing: All of your links have "current state" as the interface and changes are logged as they're applied. The system they worked with apparently has the log as a first-class citizen and no "current state" interface, while what they wanted was a "current state" interface with snapshots.

Re: Turning the database inside-out (2015)

#89

Post by valiant55 »

dagss wrote: post #12
hans_castorp wrote: post #6

Earlier quoted context omitted.

> perhaps it is somehow built into postgres? Postgres has a built-in listen/notify mechanism. The problem with that is, that it doesn't guarantee delivery and if no process is listening, notifications will be lost. Most solutions that need something like that use "logical decoding" these days. That's the built-in change data capture exposed as a public API as part of the logical replication.

Yes, listen/notify is something very different. We would often write new projections that consumes events from years back and until today. You want sequence numbers that indicate the event's position in a partitioned log. Something like "int identity" except that the int is assigned during commit, so that you have guarantee that if you see IDs 5 and 7, then 6 will never show up, so that each consumer can store a curs…

I'm not aware of any one minute minimum delay for CDC. We currently are running an on prem SQL Server -> CDC -> Debezium -> Azure Event Hub -> Azure Function App back to on prem SQL Server and that has 5-10 second delay from source system transaction commit till update/insert.

Re: Turning the database inside-out (2015)

#90

Post by arialdomartini »

jdkoeck wrote: post #86
arialdomartini wrote: post #81

Earlier quoted context omitted.

I saw other times Git being related to event sourcing, but the argument is wrong. Most of the VCSs before Git (RCS, CVS, SVN) used to store deltas and to rebuild the state reapplying them. The very reason why Git took them by the storm is exactly because, on the contrary, Git does not store deltas but snapshots. Each commit is not there collection of the occurred chances but a complete snapshot of the whole project.…

Git is still event sourced, it’s just there is only one kind of event (a commit), and its payload is the whole state ¯\_(ツ)_/¯

Eh eh, this is an interesting point of view, but it’s really not like this.

Take the case of the event of “deleting a file”. There has been an interesting discussion between Linus and the orher developers, when Git was being d initially esigned: some of them wanted to capture and track this event. Linus firmly rejected the whole idea of track events, providing very solid arguments

http://web.archive.org/web/20200117061404/http://www.gelato....

Post reply on HN