Show HN: Minio – S3 Compatible Object Storage
1–10 of 11 posts
Re: Show HN: Minio – S3 Compatible Object Storage
#2Wow this looks interesting. Nice job team minio
Re: Show HN: Minio – S3 Compatible Object Storage
#3[deleted]
Re: Show HN: Minio – S3 Compatible Object Storage
#4Wow this looks interesting. Nice job team minio
Thanks @unknownhad
Re: Show HN: Minio – S3 Compatible Object Storage
#5Hello HN,
Here is a bit more explanation about erasure code and bit-rot protection.
Minio along with being S3 compatible also protects data against hardware failures and silent data corruption using erasure code and checksums. You may lose roughly half the number ((N/2)-1) of drives and still be able to recover the data.
Minio uses Reed-Solomon code to stripe objects into N/2 data and N/2 parity blocks. This means that in a 12 drive setup, an object is striped across as 6 data and 6 parity blocks. You can lose as many as 5 drives (be it parity or data) and still reconstruct the data reliably from the remaining drives.
Minio also supports bit-rot detection transparently which works along with erasure code using high speed BLAKE2 - https://github.com/minio/blake2b-simd hash based checksums.
Re: Show HN: Minio – S3 Compatible Object Storage
#6I tried minio and was really impressed with how easy it is to get started.
But I couldn't figure out how to scale it or make it reliable. Having my entire system depend on a single server not going down is a non-starter for me. Do you consider this use case in-scope?
I've been looking at Skylable instead. While setting it up is a little clunky, it's the only open source package I've found that explicitly supports replication across multiple datacenters - which is even more interesting to me than simply across servers within a data center.
Re: Show HN: Minio – S3 Compatible Object Storage
#7I tried minio and was really impressed with how easy it is to get started. But I couldn't figure out how to scale it or make it reliable. Having my entire system depend on a single server not going down is a non-starter for me. Do you consider this use case in-scope? I've been looking at Skylable instead. While setting it up is a little clunky, it's the only open source package I've found that explicitly supports rep…
For reliability, you could start another minio server process on a different machine and use "mc mirror" subcommand (https://docs.minio.io/docs/minio-client-complete-guide#mirro...) to asynchronously copy out data. In this release, one needs to run "mc mirror" periodically using cron(8) or an equivalent tool in their platform. We are looking at providing "-c" option to "mc mirror" subcommand, which performs continuous replication. i.e, there would be no need run it periodically.
Minio encourages micro-storage architecture and scalability is achieved by deploying many minio server instances.
Hope that helps.
Disclaimer: I work at Minio.
Re: Show HN: Minio – S3 Compatible Object Storage
#8I tried minio and was really impressed with how easy it is to get started. But I couldn't figure out how to scale it or make it reliable. Having my entire system depend on a single server not going down is a non-starter for me. Do you consider this use case in-scope? I've been looking at Skylable instead. While setting it up is a little clunky, it's the only open source package I've found that explicitly supports rep…
For reliability, you could start another minio server process on a different machine and use "mc mirror" subcommand ( https://docs.minio.io/docs/minio-client-complete-guide#mirro... ) to asynchronously copy out data. In this release, one needs to run "mc mirror" periodically using cron(8) or an equivalent tool in their platform. We are looking at providing "-c" option to "mc mirror" subcommand, which performs continu…
To clarify my previous reply, in a cloud environment, scalability and multi-tenancy is achieved by spawning many instances of minio server, one per tenant. We are also working on a distributed minio server at the moment.
Re: Show HN: Minio – S3 Compatible Object Storage
#9Earlier quoted context omitted.
For reliability, you could start another minio server process on a different machine and use "mc mirror" subcommand ( https://docs.minio.io/docs/minio-client-complete-guide#mirro... ) to asynchronously copy out data. In this release, one needs to run "mc mirror" periodically using cron(8) or an equivalent tool in their platform. We are looking at providing "-c" option to "mc mirror" subcommand, which performs continu…
To clarify my previous reply, in a cloud environment, scalability and multi-tenancy is achieved by spawning many instances of minio server, one per tenant. We are also working on a distributed minio server at the moment.
Cool!
The continuous async replication feature you mention would be ideal, especially if it's setup so that I can use consul/etcd/... to automatically elect a "master". If you also added (1) a feature so that writes can be redirected to another instance, and (2) a feature so that I can direct SOME of my reads to a master (reads where I care about consistency), then you'd have a not too difficulty to implement - but also very powerful - distributed version of minio. Maybe something along those lines is what you are planning.
Many instances of minio makes a lot of sense to me, and it seems light weight enough (i.e. resources like memory) that it would work. The only deal-breaker feature which looked like it was missing was the ability to set a disk quota. In multi-tenant environments it's important to have resource limits for reliability reasons. I considered putting each minio instance in it's own ZFS volume to handle disk quotas, but I would have had to write a lot of glue code to make that work from a deployment perspective.
Great work on minio - it's the least painful to use S3 server implementation I've run across so far.
Re: Show HN: Minio – S3 Compatible Object Storage
#10Earlier quoted context omitted.
To clarify my previous reply, in a cloud environment, scalability and multi-tenancy is achieved by spawning many instances of minio server, one per tenant. We are also working on a distributed minio server at the moment.
Cool! The continuous async replication feature you mention would be ideal, especially if it's setup so that I can use consul/etcd/... to automatically elect a "master". If you also added (1) a feature so that writes can be redirected to another instance, and (2) a feature so that I can direct SOME of my reads to a master (reads where I care about consistency), then you'd have a not too difficulty to implement - but a…
Good point @devplusops. Disk quota should be built into Minio. Will add it to the roadmap.