How easy is it to embed this into go tests? Right now I use goamz/s3test for that, but it has a lot of limitations.
Quite easy actually you can look at
https://github.com/restic/restic/blob/master/run_integration...
AWS S3 open source alternative written in Go
51–60 of 139 posts
Re: AWS S3 open source alternative written in Go
#52After evaluating a couple of options mentioned in the other comments here, we recently replaced our in-house built s3 clone with minio for our on-prem version of our app. Very robust and stable.
Keep in mind that there are plenty of object stores that are robust and stable until you put 1 billion keys in them.
Re: AWS S3 open source alternative written in Go
#53Earlier quoted context omitted.
I think these days we should by default think of storing blobs of data (like video files) in storage systems like S3 or the alternatives, and that ordinary filesystems should be thought of as a special case where you want to attach storage to an individual computer. Edit: I'm going to elaborate, because people are calling me naïve. Full disclosure: I work at a cloud provider on a storage team. For most people and app…
That's awfully naive, especially for tasks like video editing that are significantly impacted by disk read/write speeds. Even a NAS on a gigabit network is going to be roughly 6x slower than a standard internal SATA III spinning disk.
I said "by default", the implication being that you'd do something else if your application needs it. But it's much easier from an operational perspective if you start with a reliable system (replicated, networked storage) and cache locally for speed, then to try and make local filesystems reliable and durable.
Re: AWS S3 open source alternative written in Go
#54After evaluating a couple of options mentioned in the other comments here, we recently replaced our in-house built s3 clone with minio for our on-prem version of our app. Very robust and stable.
Keep in mind that there are plenty of object stores that are robust and stable until you put 1 billion keys in them.
That's a very good point - but for what we do (on-premise version control for Excel where each workbook version represents one object) we won't be getting even close to that number. But yes, agreed, it entirely depends on your use case.
Re: AWS S3 open source alternative written in Go
#55Is this just meant to emulate S3 for the sake of dev/test environments? Without clustering/HA I don't really see the point of using this over the plain old file system. Or am I missing something?
Re: AWS S3 open source alternative written in Go
#56Does this have the ability to mirror to an encrypted remote? I'm looking for something like this for a simple home storage server, but emphasis on being able to replicate to something like B2 Storage for cheap backup. Currently Infinit.sh has my attention the most, but it's quite young still. edit : https://news.ycombinator.com/item?id=12125344 this thread seems to be talking about what i want. With that said, i'm no…
Current opinion is that "borg" is the holy grail of backup schemes ... it takes attic, which fixed all of the duplicity shortcomings, and improved on that ... [1]
We[2][3] tend to agree with that.
One reason it might not work for you is that we are an order of magnitude more expensive than B2, so perhaps that's a better bet for you. On the other hand, $7.20 per year for our smallest borg account is almost as close to zero as your B2 minimum order would be, so ... who knows.
One upside of choosing our service is that you can choose your location (US, Zurich, HK, etc.)
[1] https://www.stavros.io/posts/holy-grail-backups/
[2] rsync.net
Re: AWS S3 open source alternative written in Go
#57Re: AWS S3 open source alternative written in Go
#58Is this just meant to emulate S3 for the sake of dev/test environments? Without clustering/HA I don't really see the point of using this over the plain old file system. Or am I missing something?
Absolutely, our focus currently is on multi-server minio which is being actively developed on the "distributed" branch https://github.com/minio/minio/tree/distributed
Our current stable version can export single disk or multiple disks (using erasure coding providing protection against disk failures) As it is very easy to get started with (single binary, thanks to Go) people find it attractive for dev/test environments.
To replicate for HA (even for the single server version), use "mc mirror -watch SOURCE TARGET" command to pair them up. If you have multiple drives (JBOD), you can eliminate RAID or ZFS and use Minio's erasure code to pool them up. Distributed version is also in dev/testing at the moment. It should be out in a month.
Re: AWS S3 open source alternative written in Go
#59Earlier quoted context omitted.
Keep in mind that there are plenty of object stores that are robust and stable until you put 1 billion keys in them.
That's a very good point - but for what we do (on-premise version control for Excel where each workbook version represents one object) we won't be getting even close to that number. But yes, agreed, it entirely depends on your use case.
If you don't mind me asking, why not use Amazon S3? It's cheap and -- importantly -- somebody else is on-call for its uptime.
Re: AWS S3 open source alternative written in Go
#60Earlier quoted context omitted.
Riak CS is another one: https://github.com/basho/riak_cs
Ran this in production and dealt with a lot of issues. I would caution people against it's use in anything critical or customer facing.
We use riak and riak-cs in production and in fact we are one of biggest riak users.
If you use riak in production you probably do want their (Basho) support. Their product when works, works great, but when there is a problem it's a bit hard to troubleshoot it without knowing erlang and being familiar with riak's source code.