Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every System is a Log: Avoiding coordination in distributed applications

restate.dev

Post reply on HN
Print view View original

51–60 of 157 posts

Re: Every System is a Log: Avoiding coordination in distributed applications

#51

Post by sewen »

Some clarification on what "one log" means here:

- It means using one log across different concerns like state a, communication with b, lock c. Often that is in the scope of a single entity (payment, user, session, etc.) and thus the scope for the one log is still small. You would have a lot of independent logs still, for separate payments.

- It does _not_ mean that one should share the same log (and partition) for all the entities in your app, like necessarily funneling all users, payments, etc. through the same log. That goes actually beyond the proposal here - has some benefits of its own, but have a hard time scaling.

Re: Every System is a Log: Avoiding coordination in distributed applications

#52

Post by sewen »

mrkeen wrote: post #50
bruce343434 wrote: post #27

> If everything’s in one log, there’s nothing to coordinate # On the contrary. Everything becomes coordinated. The entire "log" becomes a giant ass mutex lock. Good luck scaling it.

I think the author is motte-and-baileying between: Literally one log - which does indeed reduce your coordination headache, but is susceptible to your "giant ass mutex" comment, and One log per ... - which brings the coordination problems right back into existence.

I can see where some of that could be written more clearly. To elaborate:

- We mean using one log across different concerns like state a, communication with b, lock c. Often that is in the scope of a single entity (payment, user, session, etc.) and thus the scope for the one log is still small, and it reduces coordination headache for coordinating between the systems. You would have a lot of independent logs still, for separate payments.

- It does _not_ mean that one should share the same log (and partition) for all the entities in your app, like necessarily funneling all users, payments, etc. through the same log. What would be needed if you try and do some multi-key-distributed transaction processing. That goes actually beyond the proposal here, and has some benefits of its own, but have a hard time scaling.

Re: Every System is a Log: Avoiding coordination in distributed applications

#53

Post by glitchc »

trollbridge wrote: post #11

I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does not matter. - Users of the log are peers. No client / server distinction. - When appending a log entry, you can send a copy of the append to all your peers. - You can ask your peers to refresh the latest log entries. - When creating a new entry, it is a very go…

How do you manage log size for high-transaction systems?

Re: Every System is a Log: Avoiding coordination in distributed applications

#54

Post by magicalhippo »

sewen wrote: post #51

Some clarification on what "one log" means here: - It means using one log across different concerns like state a, communication with b, lock c. Often that is in the scope of a single entity (payment, user, session, etc.) and thus the scope for the one log is still small. You would have a lot of independent logs still, for separate payments. - It does _not_ mean that one should share the same log (and partition) for a…

Interesting read, not my area but I think I got the gist of it.

In your Restate example of the "processPayment" function, how do you handle errors of the "accountService" call? Like, what if it times out or returns a server error?

Do you store the error result and the caller of "processPayment" has to re-trigger the payment, in order to generate a new log?

Re: Every System is a Log: Avoiding coordination in distributed applications

#55

Post by gjtorikian »

daxfohl wrote: post #49

Haven't formed thoughts on the content yet, but happy to see a company launching something non-AI for a change.

My startup, Yetto (http://www.yetto.app) is building a better way for support professionals to do their job. (Shameless plug but we always gotta hustle.)

We, too, are weighed down by how much space AI-focused companies are taking.

Re: Every System is a Log: Avoiding coordination in distributed applications

#56

Post by jaseemabid »

daxfohl wrote: post #44
kasey_junk wrote: post #37

Earlier quoted context omitted.

I love them so much that I’ve noodled with building a programming language optimized for using them. Things like types that encode what events are legal in a log, first class support for data versions, fast file read and writes, etc

How do you do GDPR takedowns?

Immutable append only persistent log doesn't imply store everything _forever_.

If you want to remove something you could add a tombstone record (like Cassandra) and eventually remove the original entry during routine maintenance operations like repacking into a more efficient format, archival into cold storage, TTL handling etc.

Re: Every System is a Log: Avoiding coordination in distributed applications

#57

Post by shikhar »

This post makes a great case for how universal logs are in data systems. It was strange to me that there was no log-as-service with the qualities that make it suitable for building higher-level systems like durable execution: conditional appends (as called out by the post!), support very large numbers of logs, allow pushing high throughputs with strict ordering, and just generally provide a simple serverless experience like object storage. This led to https://s2.dev/ which is now available in preview.

It was interesting to learn how Restate links events for a key, with key-level logical logs multiplexed over partitioned physical logs. I imagine this is implemented with a leader per physical log, so you can consistently maintain an index. A log service supporting conditional appends allows such a leader to act like the log is local to it, despite offering replicated durability.

Leadership can be an important optimization for most systems, but shared logs also allow for multi-writer systems pretty easily. We blogged about this pattern https://s2.dev/blog/kv-store

Re: Every System is a Log: Avoiding coordination in distributed applications

#58

Post by mrkeen »

sewen wrote: post #52
mrkeen wrote: post #50

Earlier quoted context omitted.

I think the author is motte-and-baileying between: Literally one log - which does indeed reduce your coordination headache, but is susceptible to your "giant ass mutex" comment, and One log per ... - which brings the coordination problems right back into existence.

I can see where some of that could be written more clearly. To elaborate: - We mean using one log across different concerns like state a, communication with b, lock c. Often that is in the scope of a single entity (payment, user, session, etc.) and thus the scope for the one log is still small, and it reduces coordination headache for coordinating between the systems. You would have a lot of independent logs still, f…

> thus the scope for the one log is still small, and it reduces coordination headache for coordinating between the systems.

This does not follow.

Neither the dev (nor the product seller) is in control of what real-world effects have a causal relationship.

If my actions can be independently processed from your actions, then we don't need to coordinate, and micrologs will perform faster than a monolog.

If my actions cannot be independently processed from your actions, then we do need to coordinate, unless we use a monolog, which kills performance.

Either way, we don't get to choose the level of independence. If the actions are dependent, we either coordinate or or mess up causality.

Re: Every System is a Log: Avoiding coordination in distributed applications

#59

Post by UltraSane »

daxfohl wrote: post #44
kasey_junk wrote: post #37

Earlier quoted context omitted.

I love them so much that I’ve noodled with building a programming language optimized for using them. Things like types that encode what events are legal in a log, first class support for data versions, fast file read and writes, etc

How do you do GDPR takedowns?

Best method is to encrypt data with a per user key so you if you delete the encryption key that customer's data cannot be read. You can create a new customer key at a fixed time interval to make deletion more granular.

Re: Every System is a Log: Avoiding coordination in distributed applications

#60

Post by UltraSane »

pjc50 wrote: post #25

> Having a single place (the one log) that forces a linear history of events as the ground truth and owns the decision of who can add to that ground truth, means we don’t have to coordinate much any more. Well, yes, but then you've backed into CAP again because you only have one log.

couldn't the log be synchronously replicated to multiple servers to increase A?
Post reply on HN