Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every System is a Log: Avoiding coordination in distributed applications

restate.dev

Post reply on HN
Print view View original

61–70 of 157 posts

Re: Every System is a Log: Avoiding coordination in distributed applications

#61

Post by stsffap »

magicalhippo wrote: post #54
sewen wrote: post #51

Some clarification on what "one log" means here: - It means using one log across different concerns like state a, communication with b, lock c. Often that is in the scope of a single entity (payment, user, session, etc.) and thus the scope for the one log is still small. You would have a lot of independent logs still, for separate payments. - It does _not_ mean that one should share the same log (and partition) for a…

Interesting read, not my area but I think I got the gist of it. In your Restate example of the "processPayment" function, how do you handle errors of the "accountService" call? Like, what if it times out or returns a server error? Do you store the error result and the caller of "processPayment" has to re-trigger the payment, in order to generate a new log?

By default, failing ctx.run() calls (like the accountService call) will be retried indefinitely until they succeed unless you have configured a retry policy for them. In the case of a configured retry policy where you have exhausted the number of retry attempts, Restate will mark this call as terminally failed and record it in its log as such and return it to the caller.

Re: Every System is a Log: Avoiding coordination in distributed applications

#62

Post by logsr »

shikhar wrote: post #57

This post makes a great case for how universal logs are in data systems. It was strange to me that there was no log-as-service with the qualities that make it suitable for building higher-level systems like durable execution: conditional appends (as called out by the post!), support very large numbers of logs, allow pushing high throughputs with strict ordering, and just generally provide a simple serverless experien…

> log as a service

very exciting. this is the future. i am working on a very similar concept. every database is a log at its core, so the log, which is the highest performance part of the system, is buried behind many layers of much lower performing cruft. edge persistence with log-per-user application patterns opens up so many possibilities.

Re: Every System is a Log: Avoiding coordination in distributed applications

#63

Post by hansonkd »

gjtorikian wrote: post #55
daxfohl wrote: post #49

Haven't formed thoughts on the content yet, but happy to see a company launching something non-AI for a change.

My startup, Yetto ( http://www.yetto.app ) is building a better way for support professionals to do their job. (Shameless plug but we always gotta hustle.) We, too, are weighed down by how much space AI-focused companies are taking.

TBH looking at helpdesk software in 2025, I would expect new ones to be built AI first. It would be hard for me to consider one without at least some sort of LLMs helping with triage or at classifications of tickets, etc.

Re: Every System is a Log: Avoiding coordination in distributed applications

#64

Post by jamamp »

I wonder how this compares, conceptually, to Temporal? While Temporal doesn't talk about a single centralized log, I feel the output is the same: your event handlers become durable and can be retried without re-executing certain actions with outside systems. Both Restate and Temporal feel, as a developer coding these event handlers, like a framework where they handle a lot of the "has this action been performed yet?" and such for you.

Though to be fair I've only read Temporal docs, and this Restate blog post, without much experience in either. Temporal may not have as much on the distributed locking (or concept of) side of things that Restate does, in this post.

Re: Every System is a Log: Avoiding coordination in distributed applications

#65

Post by logsr »

xnorswap wrote: post #33

It sounds like they have just re-discovered Distributed Transactions with a Distributed Transaction Coordinator. But DTs have a huge problem: What happens if the owner of the lock netsplits? Either the DTC waits (potentially forever?) for the owner of the lock to get back in touch and release the lock, or a timeout is applied and now the owner of the lock (who may be unaware of the netsplit) will be out of sync with…

they point out that they are adding conditional append to a log, which gives the universality of a log data structure and a mechanism for implementing lock-free/wait-free concurrency algorithms. it is a complete toolkit for building distributed systems because you can build anything else on top of those primitives.

Re: Every System is a Log: Avoiding coordination in distributed applications

#66

Post by sewen »

stsffap wrote: post #61
magicalhippo wrote: post #54

Earlier quoted context omitted.

Interesting read, not my area but I think I got the gist of it. In your Restate example of the "processPayment" function, how do you handle errors of the "accountService" call? Like, what if it times out or returns a server error? Do you store the error result and the caller of "processPayment" has to re-trigger the payment, in order to generate a new log?

By default, failing ctx.run() calls (like the accountService call) will be retried indefinitely until they succeed unless you have configured a retry policy for them. In the case of a configured retry policy where you have exhausted the number of retry attempts, Restate will mark this call as terminally failed and record it in its log as such and return it to the caller.

You can catch these errors and handle them in a common try/catch manner, and because the results of `ctx.run` are recorded in the log, this is deterministic and reliable

Re: Every System is a Log: Avoiding coordination in distributed applications

#67

Post by sewen »

jamamp wrote: post #64

I wonder how this compares, conceptually, to Temporal? While Temporal doesn't talk about a single centralized log, I feel the output is the same: your event handlers become durable and can be retried without re-executing certain actions with outside systems. Both Restate and Temporal feel, as a developer coding these event handlers, like a framework where they handle a lot of the "has this action been performed yet?"…

Temporal is related, but I would say it is a subset of this.

If you only consider appending results of steps of a handler, then you have something like Temporal.

This here uses the log also for RPC between services, for state that outlives an individual handler execution (state that outlives a workflow, in Temporal's terms).

Re: Every System is a Log: Avoiding coordination in distributed applications

#68

Post by logsr »

clayg wrote: post #32
pjc50 wrote: post #25

> Having a single place (the one log) that forces a linear history of events as the ground truth and owns the decision of who can add to that ground truth, means we don’t have to coordinate much any more. Well, yes, but then you've backed into CAP again because you only have one log.

But can't any log be implemented as a CRDT? Was that not implied in the post? I didn't read it that close...

the "conflict-free" in crdt is like miller high life being the "champagne of beers." it really means conflicts ignored and conflicting items discarded by established rules, which works well for some use cases, but for many does not.

Re: Every System is a Log: Avoiding coordination in distributed applications

#69

Post by gk1 »

hansonkd wrote: post #63
gjtorikian wrote: post #55

Earlier quoted context omitted.

My startup, Yetto ( http://www.yetto.app ) is building a better way for support professionals to do their job. (Shameless plug but we always gotta hustle.) We, too, are weighed down by how much space AI-focused companies are taking.

TBH looking at helpdesk software in 2025, I would expect new ones to be built AI first. It would be hard for me to consider one without at least some sort of LLMs helping with triage or at classifications of tickets, etc.

See https://www.fixify.com/, although they are too modest about mentioning the AI they're using under the hood.
Post reply on HN