Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every System is a Log: Avoiding coordination in distributed applications

restate.dev

Post reply on HN
Print view View original

141–150 of 157 posts

Re: Every System is a Log: Avoiding coordination in distributed applications

#141

Post by formerly_proven »

inopinatus wrote: post #92
Animats wrote: post #91

This is a basic concept in accounting. The general ledger is an immutable log of transactions. Other accounting documents are constructed from the general ledger, and can, if necessary, be rebuilt from it. This is the accepted way to do money-related things. Synchronization is called "reconcilation" in accounting terminology. The computer concept is that we have a current state, and changes to it come in. The databas…

I’ve always concurred with the Helland/Kleppman observation mentioned viz. that the transaction log of a typical RDBMS is the canonical form and all the rows & tables merely projections. It’s curious that over those projections, we then build event stores for CQRS/ES systems, ledgers etc, with their own projections mediated by application code. But look underneath too. The journaled filesystem on which the database r…

This is similar to the observation that memory semantics in most any powerful machine since the late 60s are implemented using messaging, and then applications go ahead and build messaging out of memory semantics. Or the more general observation that every layer of information exchange tends towards implementing packet switching if there's sufficient budget (power/performance/cost) to support doing so.

Re: Every System is a Log: Avoiding coordination in distributed applications

#142

Post by withinboredom »

hinkley wrote: post #119
withinboredom wrote: post #101

Earlier quoted context omitted.

So many people I work with don't "get" distributed systems and how they interplay and cause problems. Most people don't even know that the ORDER you take potentially competing (distributed) locks even matters -- which is super important if you have different teams taking the same locks in different services! The article is well written, but they still have a lot of problems to solve.

I went too far the other way. Concurrent things just fit my brain so well that I created systems that made my coworkers have to ask for help. One that still sticks in my mind after all these years wanted to ask me to knock it off but lacked the technical chops to make it a demand. But I could read between the lines. He was part of my process of coming around to interpreting all questions as feedback. There’s about 20…

> The hangover is going to be spectacular when it hits.

I'm honestly looking forward to it. We constantly deal with abstractions until they break and we are forced to dive into the concrete. That can be painful, but it (usually) results in a better world to live in.

Re: Every System is a Log: Avoiding coordination in distributed applications

#143

Post by drewbug01 »

ris wrote: post #138

> Restate is open source and you can download it at... https://github.com/restatedev/restate/blob/main/LICENSE#L1 > Business Source License 1.1 https://spdx.org/licenses/BUSL-1.1.html > The Business Source License (this document, or the “License”) is not an Open Source license. Suggest exploring e.g. https://github.com/dbos-inc/dbos-transact-py

I definitely don't consider the BSL to be a true "open source" license. However, I think that if the Additional Use Grant is written clearly and unambiguously, it may well be a reasonable compromise.

That said - the Additional Use Grant for Restate is fairly ambiguous:

> Additional Use Grant: You may make use of the Licensed Work, provided that you may not use the Licensed Work for an Application Platform Service.

> An “Application Platform Service” is a commercial offering that allows third parties (other than your employees and contractors) to access the functionality of the Licensed Work by registering new services (directly or indirectly) with the Licensed Work for the purpose of invoking such services through the Licensed Work.

What does it mean to "access the functionality"? What does it mean to "register [a] new service" ? How do you determine the "purpose" of invoking such services?

I understand that the intent is to prevent someone from launching a competing "Restate-as-a-Service" business; but broadly written clauses like this scare me off. If I want to build a totally different kind of business that heavily _relies_ on Restate, I would worry that these ambiguous, broad definitions could easily be used against me.

I really wish that companies adopting the BSL would put much more clarity into the license text. I know there's likely very good business reasons not to - you may constrain yourself in the future by doing so - but in my mind it would be the "right" thing to do. You'd get people actually using and contributing back to your software, much like they would under a more permissive license.

Re: Every System is a Log: Avoiding coordination in distributed applications

#144

Post by wlonkly »

yftsui wrote: post #118

The diagrams are remarkably neat, has a feeling of both excalidraw and draw.io, anyone know what tool was used create those?

Not sure what made the diagrams, but I think you've correctly identified the Excalidraw handwriting font. There's two versions, Virgil[1] and Excalifont[2], both under the MIT license.

I think the monospaced font is Comic Mono.[3]

[1] https://plus.excalidraw.com/virgil

[2] https://plus.excalidraw.com/excalifont

[3] https://dtinth.github.io/comic-mono-font/

Re: Every System is a Log: Avoiding coordination in distributed applications

#145

Post by globular-toast »

withinboredom wrote: post #96
globular-toast wrote: post #94

Earlier quoted context omitted.

> It’s curious that over those projections, we then build event stores for CQRS/ES systems, ledgers etc, with their own projections mediated by application code. The database only supports CRUD. So while the CDC stream is the truth, it's very low level. We build higher-level event types (as in event sourcing) for the same reason we build any higher-level abstraction: it gives us a language in which to talk about busi…

I'm sorry; but have you ever actually used a database before? A database supports FAR more than "only CRUD". Some really simple examples are CTEs, SELECT ... INTO (or INSERT ... SELECT for some dialects), triggers, views, etc.

Have you completely missed the context of this thread? We're talking about the transaction log aka CDC. This consists purely of CRUD operations because that's all the database understands. Maybe read the books cited?

Re: Every System is a Log: Avoiding coordination in distributed applications

#146

Post by log4shell »

trollbridge wrote: post #123
log4shell wrote: post #34

Earlier quoted context omitted.

Calling a WAL a ledger, why? Ledger sounds fancier but why would it be a ledger in this case?

We called it a ledger since we stored financial data and basically used the “ledger” format from plain text accounting, initially.

That answers my question, thanks! Ordering does not matter part has me curious too, I will read other comment and come back to you l.

Re: Every System is a Log: Avoiding coordination in distributed applications

#147

Post by hinkley »

withinboredom wrote: post #142
hinkley wrote: post #119

Earlier quoted context omitted.

I went too far the other way. Concurrent things just fit my brain so well that I created systems that made my coworkers have to ask for help. One that still sticks in my mind after all these years wanted to ask me to knock it off but lacked the technical chops to make it a demand. But I could read between the lines. He was part of my process of coming around to interpreting all questions as feedback. There’s about 20…

> The hangover is going to be spectacular when it hits. I'm honestly looking forward to it. We constantly deal with abstractions until they break and we are forced to dive into the concrete. That can be painful, but it (usually) results in a better world to live in.

Cloud will come back in your lifetime and maybe mine. Everything in software is cycles and epicycles. Hyperscaler hardware is basically a supercomputer without the fancy proprietary control software, which is third party now.

Re: Every System is a Log: Avoiding coordination in distributed applications

#148

Post by gjtorikian »

hansonkd wrote: post #63
gjtorikian wrote: post #55

Earlier quoted context omitted.

My startup, Yetto ( http://www.yetto.app ) is building a better way for support professionals to do their job. (Shameless plug but we always gotta hustle.) We, too, are weighed down by how much space AI-focused companies are taking.

TBH looking at helpdesk software in 2025, I would expect new ones to be built AI first. It would be hard for me to consider one without at least some sort of LLMs helping with triage or at classifications of tickets, etc.

I never said we didn’t have AI :) Just that the vast majority of apps out there push AI agents, when a human will do just fine. We should be making the harder problems easier, not the easy problems faster.

We support parsing your entire help docs into a DAG, so support professionals can have easy access to the most relevant existing docs to answer an end user’s problem. We also provide summarization of incoming tickets so that support teams can understand quickly what the problem is.

I’ve toyed with doing sentiment analysis, but our experience with customers in the support space is that they want the AI to give them information and then get out of the way.

Re: Every System is a Log: Avoiding coordination in distributed applications

#149

Post by jamamp »

sewen wrote: post #67
jamamp wrote: post #64

I wonder how this compares, conceptually, to Temporal? While Temporal doesn't talk about a single centralized log, I feel the output is the same: your event handlers become durable and can be retried without re-executing certain actions with outside systems. Both Restate and Temporal feel, as a developer coding these event handlers, like a framework where they handle a lot of the "has this action been performed yet?"…

Temporal is related, but I would say it is a subset of this. If you only consider appending results of steps of a handler, then you have something like Temporal. This here uses the log also for RPC between services, for state that outlives an individual handler execution (state that outlives a workflow, in Temporal's terms).

That makes a lot of sense, thank you! Extending out to other operations and not just event handlers/workflows would be neat.
Post reply on HN