Hello,
what is the best choice for a private cloud storage between openstack's swift and a riak/luwak solution?
The purpose of the storage is long term archiving of files (1MB - 500MB), what would you choose and why?
Ask HN: riak/luwak vs openstack swift for private S3 clone?
1–6 of 6 posts
Re: Ask HN: riak/luwak vs openstack swift for private S3 clone?
#2I'm going to lean towards swift, but then again, I'm a core dev for swift and I haven't played with riak/luwak.
Swift is ideal for storing static content (backups, web resources, documents). It's designed to be very scalable (both with concurrency and with total storage space). And it's designed to work with commodity hardware (read: cheap), so it handles failures well (ensuring that data is replicated and safe from bit rot). Also, swift has basic S3 compatibility support.
Please let me know if you have other, more specific questions, and I'll try to answer to the best I can.
Re: Ask HN: riak/luwak vs openstack swift for private S3 clone?
#3I'm going to lean towards swift, but then again, I'm a core dev for swift and I haven't played with riak/luwak. Swift is ideal for storing static content (backups, web resources, documents). It's designed to be very scalable (both with concurrency and with total storage space). And it's designed to work with commodity hardware (read: cheap), so it handles failures well (ensuring that data is replicated and safe from…
thank you very much for your answer.
I have several question : can we mix node with different storage capacities (for example when we add new node later)?
Is this kind of storage really cheaper than Nas with RAID on "small" capacity like 40 or 60TB total space? Ok hardware is cheaper, but what about other cost like power consumption or cooling?
Re: Ask HN: riak/luwak vs openstack swift for private S3 clone?
#4I'm going to lean towards swift, but then again, I'm a core dev for swift and I haven't played with riak/luwak. Swift is ideal for storing static content (backups, web resources, documents). It's designed to be very scalable (both with concurrency and with total storage space). And it's designed to work with commodity hardware (read: cheap), so it handles failures well (ensuring that data is replicated and safe from…
thank you very much for your answer. I have several question : can we mix node with different storage capacities (for example when we add new node later)? Is this kind of storage really cheaper than Nas with RAID on "small" capacity like 40 or 60TB total space? Ok hardware is cheaper, but what about other cost like power consumption or cooling?
Power consumption and cooling will, of course, be very dependent on your chosen hardware and hosting (DC).
Yes, swift can handle heterogenous drive sizes. For example, you can start with 2T drives and start adding 3T drives when they become more cost effective. You initial 40-60TB is well within reason for a swift cluster. I know of several running clusters at much larger scale than that (PBs of data, billions of objects).
Cost consideration are also highly dependent on your particular hardware choice. We recommend that you optimize your hardware for price per GB rather than for IOps or RAM or CPU. As you can imagine, there are nuances to all of this, too.
In case you haven't seen it yet, all the auto-generated docs for swift are at http://swift.openstack.org and much help can be found in #openstack on freenode IRC.
I'd recommend that you first look at the swift all-in-one docs (http://swift.openstack.org/development_saio.html) for running an entire cluster in a single VM. It should give you a good feel of the different parts of the system.
Re: Ask HN: riak/luwak vs openstack swift for private S3 clone?
#5I know nothing about Switft so I'll keep my comments strictly related to Luwak.
Luwak preforms really well as a large object store...up to a point. I've personally seen latencies of [1]: http://en.wikipedia.org/wiki/Persistent_data_structure
Re: Ask HN: riak/luwak vs openstack swift for private S3 clone?
#6(I'm going to have some bias here as I've done Swift deployments as well.)
One of the advantages of using something like Swift for long-term storage is it's simplicity of how data is stored on disk. Data isn't chunked up and distributed throughout a cluster. Each replica is whole and on disk. And, as notmyname, mentions there are auditors that continuously run to check for bitrot and to ensure replicas are in place. The data is extremely durable in a swift cluster if you deploy and configure everything right.
I've done most of my price modeling for petabyte-scale deployments, but it actually has good scaling-down properties as well. If you're concerned about power, 2.5" drives may be an option. Although I would avoid 'green' drives as Swift tends to keep the drives quite active (replica and integrity checks) and you won't see much benefit.