CopyCat: Protocol-agnostic implementation of the Raft consensus algorithm
1–10 of 17 posts
Re: CopyCat: Protocol-agnostic implementation of the Raft consensus algorithm
#2I am writing a program which has to durably store large number of small files. My current implementation stores these on the local disk as files in a particular folder structure. I am considering designing some approach to make this program run on a cluster of machines for high-availability, fault-tolerance & scalability reasons.
One approach I can think of is to run a distributed document database and use that for storage. I don't need most features of such database products in the foreseeable future, so I fear that they will add operational overhead for not much benefit.
Another approach I can think of is to run my processing nodes against a network file system, and rely on that to do the replication.
Yet another approach I am considering is to use something like CopyCat to implement the file replication in my application code. Is this a good use case for CopyCat?
Re: CopyCat: Protocol-agnostic implementation of the Raft consensus algorithm
#3I am writing a program which has to durably store large number of small files. My current implementation stores these on the local disk as files in a particular folder structure. I am considering designing some approach to make this program run on a cluster of machines for high-availability, fault-tolerance & scalability reasons. One approach I can think of is to run a distributed document database and use that for s…
Have you considered using a pre-existing distributed filesystem like Ceph or even HDFS instead?
Re: CopyCat: Protocol-agnostic implementation of the Raft consensus algorithm
#4I am writing a program which has to durably store large number of small files. My current implementation stores these on the local disk as files in a particular folder structure. I am considering designing some approach to make this program run on a cluster of machines for high-availability, fault-tolerance & scalability reasons. One approach I can think of is to run a distributed document database and use that for s…
Have you considered using a pre-existing distributed filesystem like Ceph or even HDFS instead?
I did consider both. HDFS, from what I read, is designed for storing large files - my files are about 10Kb in size each.
I am aware of Ceph but have not tried installing it to see how easy/hard it is to setup. Also, although this is not a hard requirement, I'd like to be able to support Windows; from what I have read so far, Ceph does not support Windows.
Re: CopyCat: Protocol-agnostic implementation of the Raft consensus algorithm
#5I am writing a program which has to durably store large number of small files. My current implementation stores these on the local disk as files in a particular folder structure. I am considering designing some approach to make this program run on a cluster of machines for high-availability, fault-tolerance & scalability reasons. One approach I can think of is to run a distributed document database and use that for s…
CopyCat looks really good for HA and fault-tolerance. Not so sure about its scalability since all writes go through the single leader node. It's more appropriate for the use of maintaining metadata of a distributed system, rather than maintaining the data themselves.
For your requirement, can you use S3 or something similar?
Re: CopyCat: Protocol-agnostic implementation of the Raft consensus algorithm
#6Earlier quoted context omitted.
Have you considered using a pre-existing distributed filesystem like Ceph or even HDFS instead?
I did consider both. HDFS, from what I read, is designed for storing large files - my files are about 10Kb in size each. I am aware of Ceph but have not tried installing it to see how easy/hard it is to setup. Also, although this is not a hard requirement, I'd like to be able to support Windows; from what I have read so far, Ceph does not support Windows.
[deleted]
Re: CopyCat: Protocol-agnostic implementation of the Raft consensus algorithm
#7I am writing a program which has to durably store large number of small files. My current implementation stores these on the local disk as files in a particular folder structure. I am considering designing some approach to make this program run on a cluster of machines for high-availability, fault-tolerance & scalability reasons. One approach I can think of is to run a distributed document database and use that for s…
CopyCat looks really good for HA and fault-tolerance. Not so sure about its scalability since all writes go through the single leader node. It's more appropriate for the use of maintaining metadata of a distributed system, rather than maintaining the data themselves. For your requirement, can you use S3 or something similar?
Thanks! The software I am writing needs to be (easily) installed at the customer site. So, S3 won't work for me. Are there any open source S3 clones that are easy to setup?
Re: CopyCat: Protocol-agnostic implementation of the Raft consensus algorithm
#8I am writing a program which has to durably store large number of small files. My current implementation stores these on the local disk as files in a particular folder structure. I am considering designing some approach to make this program run on a cluster of machines for high-availability, fault-tolerance & scalability reasons. One approach I can think of is to run a distributed document database and use that for s…
Sounds like the kind of problem OrientDB is supposed to solve (I haven't tried it though). I would stay away from network filesystems (they're fiddly, they'll add much more operational overhead than a database product, and there is no mature distributed one. OpenAFS is probably your best bet if you do want to go that route).
Re: CopyCat: Protocol-agnostic implementation of the Raft consensus algorithm
#9I am writing a program which has to durably store large number of small files. My current implementation stores these on the local disk as files in a particular folder structure. I am considering designing some approach to make this program run on a cluster of machines for high-availability, fault-tolerance & scalability reasons. One approach I can think of is to run a distributed document database and use that for s…
Sounds like the kind of problem OrientDB is supposed to solve (I haven't tried it though). I would stay away from network filesystems (they're fiddly, they'll add much more operational overhead than a database product, and there is no mature distributed one. OpenAFS is probably your best bet if you do want to go that route).
OrientDBs clustering is, IIRC, built on top of Hazelcast. Storing my files in a clustered Hazelcast is another option I am considering, although I forgot to include it in my first comment.
Re: CopyCat: Protocol-agnostic implementation of the Raft consensus algorithm
#10Earlier quoted context omitted.
Sounds like the kind of problem OrientDB is supposed to solve (I haven't tried it though). I would stay away from network filesystems (they're fiddly, they'll add much more operational overhead than a database product, and there is no mature distributed one. OpenAFS is probably your best bet if you do want to go that route).
OrientDBs clustering is, IIRC, built on top of Hazelcast. Storing my files in a clustered Hazelcast is another option I am considering, although I forgot to include it in my first comment.
Oh yuck. I didn't know that. I've long found Hazelcast a pain to work with.
Having seen in a sibling comment that you're talking about very small files, I'd recommend Zookeeper - it's mature and has pretty low admin overhead, IME.