Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IceFireDB: Distributed disk storage database based on Raft and Redis protocol

github.com

Post reply on HN
Print view View original

51–60 of 71 posts

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#51

Post by rubiquity »

refenestrator wrote: post #43
stingraycharles wrote: post #42

Earlier quoted context omitted.

Yes but those can happen at the convenience of the particular node, not necessarily as a globally chrckpointed fsync()

Then you're sacrificing consistency guarantees. If less than a majority have committed a write, it could be lost while the cluster still has a quorum up. Waiting to report success until a majority have committed allows you to make guarantees with a straight face.. "it will probably be committed in the near future" is not the same thing.

Not necessarily. As long as your quorums always overlap you can have weaker commit requirements. See the FPaxos paper and some of Heidi Howard’s blogs for more on this.

https://fpaxos.github.io/

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#52

Post by rubiquity »

jimsimmons wrote: post #36
toolz wrote: post #16

Earlier quoted context omitted.

Typically people use raft for leader election which in turn can coordinate writes. I don't think the writes are being fsync'd in the raft logs here. At least I wouldn't expect that behavior.

You don't understand Raft. Quorum has to fsync for commit

It reads like you and GP are talking about two different things. You’re talking about Raft specifics and the GP seems to be talking about a Vertical Paxos like setup where Raft is used for configuration and the data path uses another replication algorithm such as Primary-Backup or Chain Replication.

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#53

Post by rubiquity »

hughrr wrote: post #33
cortesoft wrote: post #31

Earlier quoted context omitted.

What is a better consensus protocol to use?

Look higher up the problem domain and solve it without requiring a consensus protocol.

Not sure why you’re being down voted as this is definitely the path forward. The industry and research have both explored the polar opposites of weak consistency protocols and linearizibility and consensus on the other end. Understanding your domain and knowing how you can step down to something weaker than consensus will be critical for future applications.

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#54

Post by valzam »

spookylettuce wrote: post #17

Quasi-related: what are some good hosted alternatives to AWS dynamodb / GCloud Firestore that are a) fast b) affordable at scale c) have a good local dev experience? A hosted disk based redis protocol compliant capable of sub TB size datasets would be a dream for me.

I am not sure what weight you assign to each of your requirements but DynamoDB has official Docker containers that you can use for local development. I don't find it different than developing against postgres etc. If you have tried it what problems have you encountered that make you wish for a better experience?

https://hub.docker.com/r/amazon/dynamodb-local

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#55

Post by spookylettuce »

valzam wrote: post #54
spookylettuce wrote: post #17

Quasi-related: what are some good hosted alternatives to AWS dynamodb / GCloud Firestore that are a) fast b) affordable at scale c) have a good local dev experience? A hosted disk based redis protocol compliant capable of sub TB size datasets would be a dream for me.

I am not sure what weight you assign to each of your requirements but DynamoDB has official Docker containers that you can use for local development. I don't find it different than developing against postgres etc. If you have tried it what problems have you encountered that make you wish for a better experience? https://hub.docker.com/r/amazon/dynamodb-local

I actually really like Dynamo, just hate throwing my dollars at Bezos.

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#56

Post by chucky_z »

hughrr wrote: post #48
zetsurin wrote: post #46

Earlier quoted context omitted.

Wouldn't mind knowing what versions you were using/how long ago this was. We haven't seen anything like this in > 3 years. Possible a barely supported plugin caused issues?

This is post 1.0 and without any plugins for consul. Vault has been a persistent point of failure. It’s a good idea in principle but it’s not got a good ROI

Huh. I've been running Consul since 0.7 or so, and the only time any of this happened was my own fault.

Most of the problems I've had with Vault have been around it's Terraform provider which they've improved enough that it's not an issue anymore.

I think the only thing about Raft that folks don't realize is how disk hungry it gets, if you want fast write performance you gotta make dang sure all those fsyncs can keep up. Our largest Consul cluster today runs on storage-heavy boxes as it does ~500Mb/s of writes pretty much 24/7.

I use Consul w/ Vault today instead of the internal storage for Vault just cause Consul has really nice monitoring around some stuff that Vault doesn't (path-based stuff for the most part), I think the internal storage is a really good option for 90% of use-cases.

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#57

Post by jimsimmons »

rubiquity wrote: post #51
refenestrator wrote: post #43

Earlier quoted context omitted.

Then you're sacrificing consistency guarantees. If less than a majority have committed a write, it could be lost while the cluster still has a quorum up. Waiting to report success until a majority have committed allows you to make guarantees with a straight face.. "it will probably be committed in the near future" is not the same thing.

Not necessarily. As long as your quorums always overlap you can have weaker commit requirements. See the FPaxos paper and some of Heidi Howard’s blogs for more on this. https://fpaxos.github.io/

FPaxos requires commit quorum and subsequent promise request quorum to intersect. It doesn't bring any significant thruput benefits

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#58

Post by jimsimmons »

rubiquity wrote: post #52
jimsimmons wrote: post #36

Earlier quoted context omitted.

You don't understand Raft. Quorum has to fsync for commit

It reads like you and GP are talking about two different things. You’re talking about Raft specifics and the GP seems to be talking about a Vertical Paxos like setup where Raft is used for configuration and the data path uses another replication algorithm such as Primary-Backup or Chain Replication.

Hmmm that sorta makes sense I guess. Sorta because raft is a replication algorithm. If you don't use raft in datapath you don't get any of its guarantees

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#59

Post by sitkack »

3np wrote: post #34
tomnipotent wrote: post #28

Earlier quoted context omitted.

A database without any code, actually. It's less than a few hundred lines of Go that just wraps two other databases (syndtr/goleveldb and ledisdb/ledisdb) with a third library (tidwall/uhaha) that provides a Raft API.

Oooh that means I can form it to do Redis instead right? Because that could be a nice way to to Redis clustering

Do you need Redis or the Redis protocol?

Aerospike and ScyllaDB both support a subset of the Redis API and run as a durable cluster. Both have been tested with Jepsen.

Re: IceFireDB: Distributed disk storage database based on Raft and Redis protocol

#60

Post by sitkack »

spookylettuce wrote: post #17

Quasi-related: what are some good hosted alternatives to AWS dynamodb / GCloud Firestore that are a) fast b) affordable at scale c) have a good local dev experience? A hosted disk based redis protocol compliant capable of sub TB size datasets would be a dream for me.

ScyllaDB, https://www.scylladb.com/alternator/
Post reply on HN