Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a modern durable execution engine from first principles

restate.dev

Post reply on HN
Print view View original

1–10 of 29 posts

Re: Building a modern durable execution engine from first principles

#4

Post by p10jkle »

oulipo wrote: post #3

Interesting, how does it compare to Inngest and DBOS?

Hey, I work on Restate. There are lots of differences throughout the architecture and the developer experience, but the one most relevant to this article is that Restate is itself a self-contained distributed stream-processing engine, which it uses to offer extremely low latency durable execution with strong consistency across AZs/regions. Other products tend to layer on top of other stores, which will inherit the good things and the bad things about those stores when it comes to throughput/latency/multi-region/consistency.

We are putting a lot of work into high throughput, low latency, distributed use cases, hence some of the decisions in this article. We felt that this necessitated a new database.

Re: Building a modern durable execution engine from first principles

#6

Post by ALLTaken »

p10jkle wrote: post #4
oulipo wrote: post #3

Interesting, how does it compare to Inngest and DBOS?

Hey, I work on Restate. There are lots of differences throughout the architecture and the developer experience, but the one most relevant to this article is that Restate is itself a self-contained distributed stream-processing engine, which it uses to offer extremely low latency durable execution with strong consistency across AZs/regions. Other products tend to layer on top of other stores, which will inherit the go…

Hi,

I'm building a distributed application based on Hypergraphs, because the data being processed is mostly re-executable in different ways.

It's so refreshing to read this, I was also sitting down many nights and was thinking up about the same problem that you guys solved. I'm so glad about this!

Would it be possible to plug other storage engines into Restate? The data-structure that needs to be persisted allows multiple-path execution and instant re-ordering without indexing requirements.

I'm mostly programming in Julia and would love to see some little support for it too =)

Great work guys!

Re: Building a modern durable execution engine from first principles

#8

Post by dang »

yubblegum wrote: post #7
dang wrote: post #5

One of the authors worked on Apache Flink but is too modest to include that interesting detail! So I'm adding it here. Hopefully he won't mind.

They mention it in their about page: https://restate.dev/about

Thanks—sounds like it was more than one of them!

Re: Building a modern durable execution engine from first principles

#9

Post by sewen »

ALLTaken wrote: post #6
p10jkle wrote: post #4

Earlier quoted context omitted.

Hey, I work on Restate. There are lots of differences throughout the architecture and the developer experience, but the one most relevant to this article is that Restate is itself a self-contained distributed stream-processing engine, which it uses to offer extremely low latency durable execution with strong consistency across AZs/regions. Other products tend to layer on top of other stores, which will inherit the go…

Hi, I'm building a distributed application based on Hypergraphs, because the data being processed is mostly re-executable in different ways. It's so refreshing to read this, I was also sitting down many nights and was thinking up about the same problem that you guys solved. I'm so glad about this! Would it be possible to plug other storage engines into Restate? The data-structure that needs to be persisted allows mul…

Thank you for the kind words!

The storage engine is pretty tightly integrated with the log, but the programming model allows you to attach quasi arbitrary state to keys.

So see whether this fits your use case, would be great to better understand the data and structure you are working with. Do you have a link where we could look at this?

Re: Building a modern durable execution engine from first principles

#10

Post by sewen »

dang wrote: post #5

One of the authors worked on Apache Flink but is too modest to include that interesting detail! So I'm adding it here. Hopefully he won't mind.

All of the Restate co-founders com from various stages of Apache Flink.

Restate is in many ways a mirror image to Flink. Both are event-streaming architectures, but otherwise make a lot of contrary design choices.

(This is not really helpful to understand what Restate does for you, but it is an interesting tid bit about the design.)

       Flink     |   Restate
  -------------------------------
                 |
    analytics    |  transactions
                 |
  coarse-grained |  fine-grained
    snapshots    | quorum replication
                 |
   throughput-   |  latency-sensitive
    optimized    |  
                 |
  app and Flink- |  disaggregated code
  share process  |   and framework
                 |
      Java       |      Rust
the list goes on...
Post reply on HN