Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every System is a Log: Avoiding coordination in distributed applications

restate.dev

Post reply on HN
Print view View original

11–20 of 157 posts

Re: Every System is a Log: Avoiding coordination in distributed applications

#11

Post by trollbridge »

I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles:

- The order of log entries does not matter.

- Users of the log are peers. No client / server distinction.

- When appending a log entry, you can send a copy of the append to all your peers.

- You can ask your peers to refresh the latest log entries.

- When creating a new entry, it is a very good idea to have a nonce field. (I use nano IDs for this purpose along with a timestamp, which is probabilistically unique.)

- If you want to do database style queries of the data, load all the log entries into an in memory database and query away.

- You can append a log entry containing a summary of all log entries you have so far. For example: you’ve been given 10 new customer entries. You can create a log entry of “We have 10 customers as of this date.”

- When creating new entries, prepare the entry or list of entries in memory, allow the user to edit/revise them as a draft, then when they click “Save”, they are in the permanent record.

- To fix a mistake in an entry, create a new entry that “negates” that entry.

A lot of parallelism / concurrency problems just go away with this design.

Re: Every System is a Log: Avoiding coordination in distributed applications

#12

Post by jaseemabid »

A notable example of a large-scale app built with a very similar architecture is ATproto/Bluesky[1].

"ATProto for Distributed Systems Engineers" describes how updates from the users end up in their own small databases (called PDS) and then a replicated log. What we traditionally think of as an API server (called a view server in ATProto) is simply one among the many materializations of this log.

I personally find this model of thinking about dataflow in large-scale apps pretty neat and easy to understand. The parallels are unsurprising since both the Restate blog and ATProto docs link to the same blog post by Martin Kleppmann.

This arch seems to be working really well for Bluesky, as they clearly aced through multiple 10x events very recently.

[1]: https://atproto.com/articles/atproto-for-distsys-engineers

Re: Every System is a Log: Avoiding coordination in distributed applications

#13

Post by vdm »

EGreg wrote: post #5

Since we’re on the subject of logs and embarassingly parallel distributed systems, I know someone who’s also in NYC who’s been building a project exactly along these lines. It’s called gossiplog and it uses Prolly trees to make some interesting results. https://www.npmjs.com/package/@canvas-js/gossiplog Joel Gustafson started this stuff at MIT and used to work at Protocol Labs. It’s very straightforward. By any chanc…

Thank you @EGreg for sharing this.

Re: Every System is a Log: Avoiding coordination in distributed applications

#14

Post by azmy »

davexunit wrote: post #6

My takeaway from this article is that the proposed solution for distributed app coordination is a shared, centralized log. What did I miss?

IMHO the article is not mainly about the implementation of the Log, but rather leveraging on the idea of the log to build reliable and fault tolerant applications. The implementation of the log itself can be either centralised or decentralised.

Re: Every System is a Log: Avoiding coordination in distributed applications

#15

Post by trollbridge »

whoiskatrin wrote: post #3

whats your take on handling log compaction to prevent unbounded growth, especially in systems with high write throughput?

I have a “summarise” log entry: the current log’s contents that will be relevant to the future are summarised. For example, if it’s FY2023’s financial transactions, we compute the final balances at the end of the year. We then close the log, and write an entry to it of “no more log entries after this are valid”.

We then copy the summary transactions to a new log, and compress and archive the old log.

You can identify high throughput and low throughput types of log entries and segregate them into different log streams. For example, the “new customer/change customer info” stream probably gets way less traffic than the “customer has logged in” stream. The former is also harder to summarise. Put the hard to summarise but low volume stuff in its own log.

Re: Every System is a Log: Avoiding coordination in distributed applications

#16

Post by sewen »

davexunit wrote: post #6

My takeaway from this article is that the proposed solution for distributed app coordination is a shared, centralized log. What did I miss?

That gist is correct - I would add that the log needs a few specific properties and conceptually be the shared log for state, communication, execution scheduling.

The next step is the, how do you make this usable in practice...

Re: Every System is a Log: Avoiding coordination in distributed applications

#17

Post by [deleted] »

vdm wrote: post #13
EGreg wrote: post #5

Since we’re on the subject of logs and embarassingly parallel distributed systems, I know someone who’s also in NYC who’s been building a project exactly along these lines. It’s called gossiplog and it uses Prolly trees to make some interesting results. https://www.npmjs.com/package/@canvas-js/gossiplog Joel Gustafson started this stuff at MIT and used to work at Protocol Labs. It’s very straightforward. By any chanc…

Thank you @EGreg for sharing this.

[deleted]

Re: Every System is a Log: Avoiding coordination in distributed applications

#19

Post by EGreg »

vdm wrote: post #13
EGreg wrote: post #5

Since we’re on the subject of logs and embarassingly parallel distributed systems, I know someone who’s also in NYC who’s been building a project exactly along these lines. It’s called gossiplog and it uses Prolly trees to make some interesting results. https://www.npmjs.com/package/@canvas-js/gossiplog Joel Gustafson started this stuff at MIT and used to work at Protocol Labs. It’s very straightforward. By any chanc…

Thank you @EGreg for sharing this.

Def. I geek out on this stuff, as I am building my own distributed systems. I have had discussions with a lot of people in the space, like Leslie Lamport, Petar Maymounkov etc.

You might like this interview: https://www.youtube.com/watch?v=JWrRqUkJpMQ

This is what I’m working on now: https://intercoin.org/intercloud.pdf

Re: Every System is a Log: Avoiding coordination in distributed applications

#20

Post by EGreg »

sewen wrote: post #8
EGreg wrote: post #5

Since we’re on the subject of logs and embarassingly parallel distributed systems, I know someone who’s also in NYC who’s been building a project exactly along these lines. It’s called gossiplog and it uses Prolly trees to make some interesting results. https://www.npmjs.com/package/@canvas-js/gossiplog Joel Gustafson started this stuff at MIT and used to work at Protocol Labs. It’s very straightforward. By any chanc…

Never encountered it before, but it looks cool. I think they are trying to solve a related problem. "We can consolidate the work by making a generic log that has networking and syncing built-in. This can be used by developers to make automatically-decentralized apps without writing a single line of networking code." At a first glance, I would say that Gossiplog is a bit more low level, targeting developers of databas…

It’s part of his higher-level framework called Canvas.

Check this out: https://joelgustafson.com/posts/2024-09-30/introduction-to-c...

And this: https://github.com/canvasxyz/canvas

Post reply on HN