Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every System is a Log: Avoiding coordination in distributed applications

restate.dev

Post reply on HN
Print view View original

121–130 of 157 posts

Re: Every System is a Log: Avoiding coordination in distributed applications

#121

Post by trollbridge »

random3 wrote: post #115
trollbridge wrote: post #11

I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does not matter. - Users of the log are peers. No client / server distinction. - When appending a log entry, you can send a copy of the append to all your peers. - You can ask your peers to refresh the latest log entries. - When creating a new entry, it is a very go…

if order does not matter how do you implement deletes?

You don’t do deletes. The log is append only.

If you want to get rid of an entry, you create a new entry that negates the existing entry.

Re: Every System is a Log: Avoiding coordination in distributed applications

#122

Post by trollbridge »

Kinrany wrote: post #23
trollbridge wrote: post #11

I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does not matter. - Users of the log are peers. No client / server distinction. - When appending a log entry, you can send a copy of the append to all your peers. - You can ask your peers to refresh the latest log entries. - When creating a new entry, it is a very go…

> The order of log entries does not matter. This is surprising, Kafka-like logs are all strictly ordered.

The reason I made ordering not matter is so that multiple peers don’t have to worry about keeping the exact same order when appending to each other’s logs.

The log entries do have timestamps on them. You can sort by timestamp, but a peer has the right to append an new entry that’s older than the latest timestamp.

Re: Every System is a Log: Avoiding coordination in distributed applications

#123

Post by trollbridge »

log4shell wrote: post #34
trollbridge wrote: post #11

I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does not matter. - Users of the log are peers. No client / server distinction. - When appending a log entry, you can send a copy of the append to all your peers. - You can ask your peers to refresh the latest log entries. - When creating a new entry, it is a very go…

Calling a WAL a ledger, why? Ledger sounds fancier but why would it be a ledger in this case?

We called it a ledger since we stored financial data and basically used the “ledger” format from plain text accounting, initially.

Re: Every System is a Log: Avoiding coordination in distributed applications

#124

Post by trollbridge »

glitchc wrote: post #53
trollbridge wrote: post #11

I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does not matter. - Users of the log are peers. No client / server distinction. - When appending a log entry, you can send a copy of the append to all your peers. - You can ask your peers to refresh the latest log entries. - When creating a new entry, it is a very go…

How do you manage log size for high-transaction systems?

It’s not terribly high transaction, yet. If it becomes that way, I would partition the one by entry type so all the high transaction stuff gets stuffed in a particular log, and then ensure it can be summarised regularly.

Re: Every System is a Log: Avoiding coordination in distributed applications

#125

Post by trollbridge »

grahamj wrote: post #24
trollbridge wrote: post #11

I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does not matter. - Users of the log are peers. No client / server distinction. - When appending a log entry, you can send a copy of the append to all your peers. - You can ask your peers to refresh the latest log entries. - When creating a new entry, it is a very go…

The lack of ordering is surprising. Without that you can’t stream without a buffer.

The idea is that all peers eventually have the same collection of log entries, with an agreed-upon sort method. Currently that sort method is time stamps.

When it’s time to have identical replicas, a log entry must be appended that says the log is closed. Once all peers have this, the log is now immutable and can be treated as such. An example of this (for a payroll system) is a collection of entries for time clock in and outs, payroll taxes withheld, direct deposits to employees, etc when it’s time to do a payroll. The log is closed and that data can never be altered, but now the data can be relied on by someone else.

Conceptually this is surprisingly similar to the type of internal logs a typical database like PostgreSQL keeps.

Re: Every System is a Log: Avoiding coordination in distributed applications

#126

Post by calvinmorrison »

Animats wrote: post #91

This is a basic concept in accounting. The general ledger is an immutable log of transactions. Other accounting documents are constructed from the general ledger, and can, if necessary, be rebuilt from it. This is the accepted way to do money-related things. Synchronization is called "reconcilation" in accounting terminology. The computer concept is that we have a current state, and changes to it come in. The databas…

You over estimate ERP and accounting systems.

Re: Every System is a Log: Avoiding coordination in distributed applications

#127

Post by hem777 »

EGreg wrote: post #5

Since we’re on the subject of logs and embarassingly parallel distributed systems, I know someone who’s also in NYC who’s been building a project exactly along these lines. It’s called gossiplog and it uses Prolly trees to make some interesting results. https://www.npmjs.com/package/@canvas-js/gossiplog Joel Gustafson started this stuff at MIT and used to work at Protocol Labs. It’s very straightforward. By any chanc…

There’s also OrbitDB https://github.com/orbitdb/orbitdb which to my understanding has been a pioneer for p2p logs, databases and CRDTs.

Re: Every System is a Log: Avoiding coordination in distributed applications

#128

Post by jiggawatts »

inopinatus wrote: post #103
ianburrell wrote: post #100

Earlier quoted context omitted.

The table data in database is the canonical form. You can delete the transaction logs, and temporarily lose some reliability. It is very common to delete the transaction logs when not needed. When databases are backed up, they either dump the logical data or take snapshot of the data. Then can take stream of transaction logs for syncing or backup until the next checkpoint. I'm pretty sure journalled filesystem recycl…

Sorry, this is mistaking the operational for the fundamental. If a transaction log is replayed, then an identical set of relations will be obtained. Ergo, the log is the prime form of the database. It’s that simple.

Conversely, given a database, you can't (in general) reconstruct the specific transaction log that resulted in it. You can reconstruct some log, but it's not uniquely defined and is missing a lot of potentially relevant information.

Re: Every System is a Log: Avoiding coordination in distributed applications

#129

Post by funcDropShadow »

Animats wrote: post #91

This is a basic concept in accounting. The general ledger is an immutable log of transactions. Other accounting documents are constructed from the general ledger, and can, if necessary, be rebuilt from it. This is the accepted way to do money-related things. Synchronization is called "reconcilation" in accounting terminology. The computer concept is that we have a current state, and changes to it come in. The databas…

Could you recommend some resource to understand this view of accounting better?

Re: Every System is a Log: Avoiding coordination in distributed applications

#130

Post by baq »

calvinmorrison wrote: post #126
Animats wrote: post #91

This is a basic concept in accounting. The general ledger is an immutable log of transactions. Other accounting documents are constructed from the general ledger, and can, if necessary, be rebuilt from it. This is the accepted way to do money-related things. Synchronization is called "reconcilation" in accounting terminology. The computer concept is that we have a current state, and changes to it come in. The databas…

You over estimate ERP and accounting systems.

That was basic accounting when computers were people with pencils and paper.
Post reply on HN