Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Waltz: A Distributed Write-Ahead Log

wecode.wepay.com

Post reply on HN
Print view View original

41–46 of 46 posts

Re: Waltz: A Distributed Write-Ahead Log

#41

Post by manigandham »

atombender wrote: post #40
manigandham wrote: post #39

Earlier quoted context omitted.

Redis is an in-memory data store that has different options for persistence (snapshots + oplog) but it's not designed to persist every operation immediately. All data structures are covered including Streams. Redis keeps the entire working set in RAM so it'll start dropping writes or freeze if you run out of memory. This is where the simplicity and speed comes from and is a fundamental limitation. There's a simple re…

Thanks. That confirms pretty much the picture I had in my head of how Redis works these days. It's frustrating that there's no obvious middle ground this and Kafka and Pulsar, both of which are memory-hungry JVM apps with multiple external dependencies. Both require ZooKeeper; Pulsar also requires BookKeeper. None of these components are operationally simple. I'm a fan of NATS itself, but NATS Streaming's clustering…

There are other options. RabbitMQ [1] is the usual although clustering has always been fragile.

If you dont care about open-source then there are plenty of other options like AMPS [2] or Solace [3]. The latter has a free edition.

1. https://www.rabbitmq.com/

2. http://www.crankuptheamps.com/

3. https://solace.com/

Re: Waltz: A Distributed Write-Ahead Log

#42

Post by manigandham »

dmix wrote: post #31
manigandham wrote: post #21

Earlier quoted context omitted.

Liftbridge is an experiment and has no real production users. NATS Streaming is already fine for lightweight single-host usage. If you really need to scale to multiple servers then I recommend skipping NATS Streaming and going straight to Kafka or Apache Pulsar.

Have any more lighter-weight recommendations that fit the queuing and persistence category?

RabbitMQ is another option. Clustering is fragile, but it's fine if you just need a single-node: https://www.rabbitmq.com/

There are also proprietary systems like Solace: https://solace.com/

Other than that, there are embedded queuing/log libraries or you can just dump the messages into a key/value store or write them out to a file yourself.

Re: Waltz: A Distributed Write-Ahead Log

#43

Post by atombender »

manigandham wrote: post #41
atombender wrote: post #40

Earlier quoted context omitted.

Thanks. That confirms pretty much the picture I had in my head of how Redis works these days. It's frustrating that there's no obvious middle ground this and Kafka and Pulsar, both of which are memory-hungry JVM apps with multiple external dependencies. Both require ZooKeeper; Pulsar also requires BookKeeper. None of these components are operationally simple. I'm a fan of NATS itself, but NATS Streaming's clustering…

There are other options. RabbitMQ [1] is the usual although clustering has always been fragile. If you dont care about open-source then there are plenty of other options like AMPS [2] or Solace [3]. The latter has a free edition. 1. https://www.rabbitmq.com/ 2. http://www.crankuptheamps.com/ 3. https://solace.com/

I wouldn't put RabbitMQ in this category — it is a classical message broker, not a log. Once you've consumed a message, it's gone, unless you have set it up so ACKed messages are funneled into another queue, but that stuff is finicky and doesn't patch over the fact that underneath it's designed for mutable, ephemeral queues. In particular, you can't peek back into the queue to find older items. You have zero visibility into the contents of the queuem, and you certainly can't treat it as a dependable database.

And, as you say, fragile. I've run RMQ in production for years and I would be very happy if I could throw it out. It's the least well-behaved component in any stack I've used it in. Even Elasticsearch (shudder) is better at not losing data. Not just the clustering, either. Even for a persistent queue, RMQ will start to chug RAM for any message that is delivered but not yet ACKed, for example, making it dangerous for apps that want to batch large groups of messages for efficiency. (It seems to me that it was not designed for that at all, but for one-by-one consumption, which is of course much slower.)

I'm looking for a mature distributed log that is clustered and lightweight. Kafka except, say, written in Go.

Re: Waltz: A Distributed Write-Ahead Log

#44

Post by vadikkiev »

georgewfraser wrote: post #26

This design seems to be an example of a deterministic database system. There's an excellent review of deterministic databases here: http://www.cs.umd.edu/~abadi/papers/abadi-cacm2018.pdf The core concept of all deterministic databases is simple: if your database is deterministic, multiple geographically-distributed replicas can execute the same transaction log independently, and they will all reach the same state. Th…

I ask for your help! Hello everyone, I'd like to get 2 karma point so that I would have 2 karma points in total, so that I can prove my HackerNews account into my Keybase account. Unfortunately I have no idea how can I gain karma, since I haven't used HackerNews that much. Can someone help me. Thanks.

Re: Waltz: A Distributed Write-Ahead Log

#45

Post by manigandham »

atombender wrote: post #43
manigandham wrote: post #41

Earlier quoted context omitted.

There are other options. RabbitMQ [1] is the usual although clustering has always been fragile. If you dont care about open-source then there are plenty of other options like AMPS [2] or Solace [3]. The latter has a free edition. 1. https://www.rabbitmq.com/ 2. http://www.crankuptheamps.com/ 3. https://solace.com/

I wouldn't put RabbitMQ in this category — it is a classical message broker, not a log. Once you've consumed a message, it's gone, unless you have set it up so ACKed messages are funneled into another queue, but that stuff is finicky and doesn't patch over the fact that underneath it's designed for mutable, ephemeral queues. In particular, you can't peek back into the queue to find older items. You have zero visibili…

Try out Solace then. It's fast and supports HA for the free edition with throughput limits.

Re: Waltz: A Distributed Write-Ahead Log

#46

Post by matchagreentea »

georgewfraser wrote: post #26

This design seems to be an example of a deterministic database system. There's an excellent review of deterministic databases here: http://www.cs.umd.edu/~abadi/papers/abadi-cacm2018.pdf The core concept of all deterministic databases is simple: if your database is deterministic, multiple geographically-distributed replicas can execute the same transaction log independently, and they will all reach the same state. Th…

The blog post didn't fully specify, but the source code suggests that the locks are tracked per-partition rather than managed centrally.
Post reply on HN