Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every System is a Log: Avoiding coordination in distributed applications

restate.dev

Post reply on HN
Print view View original

91–100 of 157 posts

Re: Every System is a Log: Avoiding coordination in distributed applications

#91

Post by Animats »

This is a basic concept in accounting. The general ledger is an immutable log of transactions. Other accounting documents are constructed from the general ledger, and can, if necessary, be rebuilt from it. This is the accepted way to do money-related things.

Synchronization is called "reconcilation" in accounting terminology.

The computer concept is that we have a current state, and changes to it come in. The database with the current state is authoritative. This is not suitable for handling money.

The real question is, do you really care what happened last month? Last year? If yes, a log-based approach is appropriate.

Re: Every System is a Log: Avoiding coordination in distributed applications

#92

Post by inopinatus »

Animats wrote: post #91

This is a basic concept in accounting. The general ledger is an immutable log of transactions. Other accounting documents are constructed from the general ledger, and can, if necessary, be rebuilt from it. This is the accepted way to do money-related things. Synchronization is called "reconcilation" in accounting terminology. The computer concept is that we have a current state, and changes to it come in. The databas…

I’ve always concurred with the Helland/Kleppman observation mentioned viz. that the transaction log of a typical RDBMS is the canonical form and all the rows & tables merely projections.

It’s curious that over those projections, we then build event stores for CQRS/ES systems, ledgers etc, with their own projections mediated by application code.

But look underneath too. The journaled filesystem on which the database resides also has a log representation, and under that, a modern SSD is using an adaptive log structure to balance block writes.

It’s been a long time since we wrote an application event stream linearly straight to media, and although I appreciate the separate concerns that each of these layers addresses, I’d probably struggle to justify them all from first principles to even a slightly more Socratic version of myself.

Re: Every System is a Log: Avoiding coordination in distributed applications

#93

Post by xuancanh »

shikhar wrote: post #57

This post makes a great case for how universal logs are in data systems. It was strange to me that there was no log-as-service with the qualities that make it suitable for building higher-level systems like durable execution: conditional appends (as called out by the post!), support very large numbers of logs, allow pushing high throughputs with strict ordering, and just generally provide a simple serverless experien…

> It was strange to me that there was no log-as-service with the qualities that make it suitable for building higher-level systems like durable execution

There are several services like that, but they are mostly kept behind the scene as a competitive advantage when building distributed systems. AWS uses it behind the scene for many services, as mentioned here by Marc Brooker https://brooker.co.za/blog/2024/04/25/memorydb.html. Facebook has similar systems like LogDevice https://logdevice.io/, and recently Delos https://research.facebook.com/publications/log-structured-pr...

Re: Every System is a Log: Avoiding coordination in distributed applications

#94

Post by globular-toast »

inopinatus wrote: post #92
Animats wrote: post #91

This is a basic concept in accounting. The general ledger is an immutable log of transactions. Other accounting documents are constructed from the general ledger, and can, if necessary, be rebuilt from it. This is the accepted way to do money-related things. Synchronization is called "reconcilation" in accounting terminology. The computer concept is that we have a current state, and changes to it come in. The databas…

I’ve always concurred with the Helland/Kleppman observation mentioned viz. that the transaction log of a typical RDBMS is the canonical form and all the rows & tables merely projections. It’s curious that over those projections, we then build event stores for CQRS/ES systems, ledgers etc, with their own projections mediated by application code. But look underneath too. The journaled filesystem on which the database r…

> It’s curious that over those projections, we then build event stores for CQRS/ES systems, ledgers etc, with their own projections mediated by application code.

The database only supports CRUD. So while the CDC stream is the truth, it's very low level. We build higher-level event types (as in event sourcing) for the same reason we build any higher-level abstraction: it gives us a language in which to talk about business rules. Kleppmann makes this point in his book and it was something of an aha moment for me.

Re: Every System is a Log: Avoiding coordination in distributed applications

#95

Post by shikhar »

xuancanh wrote: post #93
shikhar wrote: post #57

This post makes a great case for how universal logs are in data systems. It was strange to me that there was no log-as-service with the qualities that make it suitable for building higher-level systems like durable execution: conditional appends (as called out by the post!), support very large numbers of logs, allow pushing high throughputs with strict ordering, and just generally provide a simple serverless experien…

> It was strange to me that there was no log-as-service with the qualities that make it suitable for building higher-level systems like durable execution There are several services like that, but they are mostly kept behind the scene as a competitive advantage when building distributed systems. AWS uses it behind the scene for many services, as mentioned here by Marc Brooker https://brooker.co.za/blog/2024/04/25/memo…

Indeed. We are trying to democratize that secret sauce. Since it is backed by object storage, the latencies are not what AWS enjoys with its internal Journal service, but we intend to get there with a NVMe-based tier later. In the meantime there is an existing large market for event streaming where a "truly serverless" (https://erikbern.com/2021/04/19/software-infrastructure-2.0-...) API has been missing.

Re: Every System is a Log: Avoiding coordination in distributed applications

#96

Post by withinboredom »

globular-toast wrote: post #94
inopinatus wrote: post #92

Earlier quoted context omitted.

I’ve always concurred with the Helland/Kleppman observation mentioned viz. that the transaction log of a typical RDBMS is the canonical form and all the rows & tables merely projections. It’s curious that over those projections, we then build event stores for CQRS/ES systems, ledgers etc, with their own projections mediated by application code. But look underneath too. The journaled filesystem on which the database r…

> It’s curious that over those projections, we then build event stores for CQRS/ES systems, ledgers etc, with their own projections mediated by application code. The database only supports CRUD. So while the CDC stream is the truth, it's very low level. We build higher-level event types (as in event sourcing) for the same reason we build any higher-level abstraction: it gives us a language in which to talk about busi…

I'm sorry; but have you ever actually used a database before? A database supports FAR more than "only CRUD". Some really simple examples are CTEs, SELECT ... INTO (or INSERT ... SELECT for some dialects), triggers, views, etc.

Re: Every System is a Log: Avoiding coordination in distributed applications

#97

Post by ianburrell »

hinkley wrote: post #87
shikhar wrote: post #57

This post makes a great case for how universal logs are in data systems. It was strange to me that there was no log-as-service with the qualities that make it suitable for building higher-level systems like durable execution: conditional appends (as called out by the post!), support very large numbers of logs, allow pushing high throughputs with strict ordering, and just generally provide a simple serverless experien…

I just want a recognized standard format for write ahead logs. Start with replicating data between OLTP and OLAP databases with minimal glue code, and start moving other systems to a similar structure, like Kafka, then new things we haven’t thought of yet.

The structure for the write head logs needs to different between systems. For Postgres, the WAL is a record of writes with new blocks. It can't be used without knowing the Postgres disk format. I don't think it can be used to construct logical changes.

Using a standard format, converting things into logical data, would be significantly slower. It is important that WAL be fast because it is the bottleneck in transactions. It would make more sense to have a separate change streaming service.

Re: Every System is a Log: Avoiding coordination in distributed applications

#98

Post by withinboredom »

XorNot wrote: post #22
trollbridge wrote: post #11

I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does not matter. - Users of the log are peers. No client / server distinction. - When appending a log entry, you can send a copy of the append to all your peers. - You can ask your peers to refresh the latest log entries. - When creating a new entry, it is a very go…

How do you know summary entries are valid if order doesn't matter? I.e. "we have 10 customers as of this date" can become immediately invalid if a new entry is appended afterwards with a date before that summary entry (i.e. because it was on a peer which hadn't yet sent it)

Realistically, you never store summaries in the log. Instead, you store what it took to calculate them. So you won't store "we have 10 customers on this date with this range" but instead store "we found these 10 customers on this date with this range". This assumes you can store infinite sized lists in your log, but realistically, this is never a concern if you can keep your time windows small enough. Then, you periodically do a reconciliation and log corrections (look for entries not summarized -- easily done via a bloom filter which can tell you what entries are definitely NOT in your set) over a longer period.

For example, we had a 28-day reconciliation period at one company I worked at (and handled over 120 million events per day). If you appended an event earlier than 28 days prior, it was simply ignored. This very rarely happened, but allowed us to fix bugs with events for up to 28 days.

Re: Every System is a Log: Avoiding coordination in distributed applications

#99

Post by ianburrell »

sewen wrote: post #43
hcarvalhoalves wrote: post #41

Earlier quoted context omitted.

I believe you want "one log" in the logical sense. In theory, you could have "one log" per user or group of users, or whatever sharding technique makes sense for multi tenancy model. It can also be "one log" per bounded context - e.g. the entire payment pipeline in one log.

Yes, exactly right. One log per logical entity, here "payment ID". The way our open source project implements that is with a partitioned log and indexes at key-granularity, so it is like virtually a log per key.

How do you deal with one side of transaction getting lost? Traditional double entry accounting puts entry for each side of the transaction. It will cause problems if one side is lost or delayed.

How do you handle transactions in general? It is pretty common to have multiple changes that need to be applied together. In a single log, can write the changes and then write a record that says the changes are committed. This is how databases work.

Re: Every System is a Log: Avoiding coordination in distributed applications

#100

Post by ianburrell »

inopinatus wrote: post #92
Animats wrote: post #91

This is a basic concept in accounting. The general ledger is an immutable log of transactions. Other accounting documents are constructed from the general ledger, and can, if necessary, be rebuilt from it. This is the accepted way to do money-related things. Synchronization is called "reconcilation" in accounting terminology. The computer concept is that we have a current state, and changes to it come in. The databas…

I’ve always concurred with the Helland/Kleppman observation mentioned viz. that the transaction log of a typical RDBMS is the canonical form and all the rows & tables merely projections. It’s curious that over those projections, we then build event stores for CQRS/ES systems, ledgers etc, with their own projections mediated by application code. But look underneath too. The journaled filesystem on which the database r…

The table data in database is the canonical form. You can delete the transaction logs, and temporarily lose some reliability. It is very common to delete the transaction logs when not needed. When databases are backed up, they either dump the logical data or take snapshot of the data. Then can take stream of transaction logs for syncing or backup until the next checkpoint.

I'm pretty sure journalled filesystem recycle the journal. There are log-structured filesystem but they aren't used much beyond low-level flash.

Post reply on HN