Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IceFireDB: Distributed disk storage database based on Raft and Redis protocol

github.com

Post reply on HN
Print view View original

61–70 of 71 posts

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#61

Post by sitkack »

jandrewrogers wrote: post #27
tyingq wrote: post #2

SET: 253232.12 requests per second GET: 2130875.50 requests per second The 10:1 throughput ratio for GET vs SET is interesting. Redis being in-memory, the rates there are pretty close to the same for read/write. Is a 10:1 ratio typical for a storage backed distributed kv store? Edit: Looks like CockroachDb has roughly a 3:1 ratio, similar for YugabyteDB: https://www.cockroachlabs.com/docs/stable/performance.html http…

Comparisons of read/write ratios has to account for several differences in design and implementation. Representative benchmarks are difficult. Things that can make a difference: Databases have subtly different definitions of "durability", so they aren't always doing semantically equivalent operations. Write throughput sometimes scales with the number of clients and it is not possible to saturate the server with a sin…

It depends, yes but ... (not discounting any of the above).

One sees a lot of 3:1 in practice due to the replication factor. If you have 3 copies of the data and the client can read from any node, you get 3x the read performance as having to have a quorum write on two out of three nodes.

To the GP, for a rough swag of what is possible out of given hardware, a combination of FIO and ACT (measures IO latency under a fixed load) is a good start.

https://fio.readthedocs.io/en/latest/fio_doc.html

https://github.com/aerospike/act

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#62

Post by rubiquity »

jimsimmons wrote: post #57
rubiquity wrote: post #51

Earlier quoted context omitted.

Not necessarily. As long as your quorums always overlap you can have weaker commit requirements. See the FPaxos paper and some of Heidi Howard’s blogs for more on this. https://fpaxos.github.io/

FPaxos requires commit quorum and subsequent promise request quorum to intersect. It doesn't bring any significant thruput benefits

I didn't say FPaxos brought throughput benefits. I pointed out that you can still have a correct implementation without requiring majority quorums.

But since you brought it up, FPaxos can have both lower latency and higher throughput than MultiPaxos/Raft[0] ;)

0 - https://dl.acm.org/doi/pdf/10.1145/3299869.3319893

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#63

Post by rubiquity »

jimsimmons wrote: post #58
rubiquity wrote: post #52

Earlier quoted context omitted.

It reads like you and GP are talking about two different things. You’re talking about Raft specifics and the GP seems to be talking about a Vertical Paxos like setup where Raft is used for configuration and the data path uses another replication algorithm such as Primary-Backup or Chain Replication.

Hmmm that sorta makes sense I guess. Sorta because raft is a replication algorithm. If you don't use raft in datapath you don't get any of its guarantees

Most data paths don't need consensus for replication. You can implement ACID Transactions on top of many other replication algorithms for example. Ultimately the choice comes down to read/write ratios. Chain Replication has much better read throughout than Paxos and Raft.

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#64

Post by tomnipotent »

sitkack wrote: post #59
3np wrote: post #34

Earlier quoted context omitted.

Oooh that means I can form it to do Redis instead right? Because that could be a nice way to to Redis clustering

Do you need Redis or the Redis protocol? Aerospike and ScyllaDB both support a subset of the Redis API and run as a durable cluster. Both have been tested with Jepsen.

> Do you need Redis or the Redis protocol?

Only if you want to be a drop-in replacement and take advantage of existing compatible libraries. With the recent tea around the official Elasticsearch Python library, it becomes a more interesting question.

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#65

Post by jimsimmons »

rubiquity wrote: post #62
jimsimmons wrote: post #57

Earlier quoted context omitted.

FPaxos requires commit quorum and subsequent promise request quorum to intersect. It doesn't bring any significant thruput benefits

I didn't say FPaxos brought throughput benefits. I pointed out that you can still have a correct implementation without requiring majority quorums. But since you brought it up, FPaxos can have both lower latency and higher throughput than MultiPaxos/Raft[0] ;) 0 - https://dl.acm.org/doi/pdf/10.1145/3299869.3319893

Great link. Need to look into what WPaxos is.

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#66

Post by jimsimmons »

rubiquity wrote: post #63
jimsimmons wrote: post #58

Earlier quoted context omitted.

Hmmm that sorta makes sense I guess. Sorta because raft is a replication algorithm. If you don't use raft in datapath you don't get any of its guarantees

Most data paths don't need consensus for replication. You can implement ACID Transactions on top of many other replication algorithms for example. Ultimately the choice comes down to read/write ratios. Chain Replication has much better read throughout than Paxos and Raft.

I see. What's the catch though. Sounds like free lunch. Is there some gotcha with partition tolerance?

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#67

Post by caffeine »

asadawadia wrote: post #49
hughrr wrote: post #47

Earlier quoted context omitted.

Easy you push the configuration to every target. Then they are logically consistent. We did this for 30 years fine before someone invented this stack on deployments much larger then the average consul or vault deployment these days. I had something running 15,000 dynamic rps on Apache about 15 years ago. People are blinded from simplicity by complexity. Eventually complexity owns you. You can only own simplicity. At…

>Easy you push the configuration to every target. Then they are logically consistent. What does this mean? This doesn't mean anything? Are you saying to push the DB to the client?

I think he’s saying you have a cronjob with an rsync to every client

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#68

Post by rubiquity »

jimsimmons wrote: post #66
rubiquity wrote: post #63

Earlier quoted context omitted.

Most data paths don't need consensus for replication. You can implement ACID Transactions on top of many other replication algorithms for example. Ultimately the choice comes down to read/write ratios. Chain Replication has much better read throughout than Paxos and Raft.

I see. What's the catch though. Sounds like free lunch. Is there some gotcha with partition tolerance?

It’s the case of 2F+1 versus F+1. Paxos/Raft offer fault tolerance where as other replication algorithms don’t. If you have a node failure in Primary-Backup or Chain for example you need to reconfigure before committing more writes. However in practice and certain environments reconfiguration can be faster than or the same as a leader failure in Paxos that requires running Phase 1 again.

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#69

Post by yencabulator »

cortesoft wrote: post #50
hughrr wrote: post #47

Earlier quoted context omitted.

Easy you push the configuration to every target. Then they are logically consistent. We did this for 30 years fine before someone invented this stack on deployments much larger then the average consul or vault deployment these days. I had something running 15,000 dynamic rps on Apache about 15 years ago. People are blinded from simplicity by complexity. Eventually complexity owns you. You can only own simplicity. At…

People need distributed consistent data for a lot of things besides configuration.

People say they do, but then they proceed to read stale data from replicas all the time...

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#70

Post by jimsimmons »

rubiquity wrote: post #68
jimsimmons wrote: post #66

Earlier quoted context omitted.

I see. What's the catch though. Sounds like free lunch. Is there some gotcha with partition tolerance?

It’s the case of 2F+1 versus F+1. Paxos/Raft offer fault tolerance where as other replication algorithms don’t. If you have a node failure in Primary-Backup or Chain for example you need to reconfigure before committing more writes. However in practice and certain environments reconfiguration can be faster than or the same as a leader failure in Paxos that requires running Phase 1 again.

Got it, makes sense. To the extent this can be formalized, I feel this is a much better alternative than consensus based approaches
Post reply on HN