Live data from Hacker News

AWS S3 open source alternative written in Go

minio.io

81–90 of 139 posts

Re: AWS S3 open source alternative written in Go

#81
post #46

Theory here is that people will build apps that talk to S3. But sometimes those apps might need to run inside the perimeter and can't talk to the cloud. So rather than rewrite an app to talk to a new internal datastore, you just point it at a locally hosted Minio and you're up and running. Smart.

What kind of situations do you see this becoming a factor in? 5 or 10 years ago this was an issue with early cloud adopters. Now a days cloud providers are ramping up their DCs to be compliant and allow companies/government entities with strict policies to still onboard. Its a good strategy but not one that I see being exercised frequently enough.

The software I work on is targeted towards customers who generally have really spotty internet connections (eg: they all are in the less forgiving parts of the ocean or middle of nowhere if on land). This pretty much mandates using software like this to build out your app as you can't rely on internet connectivity.

There pretty much isn't anything you can do to improve their internet connections as cables to remote places are always getting dug up with week+ times to repair so you need something that can run locally for long periods. Ships have a different problem with very slow speeds that effectively means you can only transmit the absolute minimum off the ship when its out as sea (when they are at port they typically have normal internet connections to bulk dump data off on).

Re: AWS S3 open source alternative written in Go

#82
post #47

Or, run Riak with their S3 compatibility layer. Riak is extremely stable and the work Basho has done to make a truly robust distributed database is significant. http://docs.basho.com/riak/cs/2.1.1/

Other alternatives: ceph - http://docs.ceph.com/docs/master/radosgw/s3/ swift - https://wiki.openstack.org/wiki/Swift/APIFeatureComparison#A...

Don't forget:

manta - https://www.joyent.com/manta

Re: AWS S3 open source alternative written in Go

#83

Minio will always be 100% free software / open source. We have no plans to add any proprietary extensions or hold back on features for paying customers only. -- Minio Team

Then why not make the license AGPLv3-or-later, to avoid other people creating proprietary forks? I get that it's not a common occurence within the Golang world, but nothing will change unless more Golang projects start making their code copylefted.

Re: AWS S3 open source alternative written in Go

#84

Does this have the ability to mirror to an encrypted remote? I'm looking for something like this for a simple home storage server, but emphasis on being able to replicate to something like B2 Storage for cheap backup. Currently Infinit.sh has my attention the most, but it's quite young still. edit : https://news.ycombinator.com/item?id=12125344 this thread seems to be talking about what i want. With that said, i'm no…

I've been wondering about this for that use case myself: https://ipfs.io/

Yea IPFS is awesome, i have often pondered about the idea of using it for an internal family storage.

Re: AWS S3 open source alternative written in Go

#86
post #72
post #61

Earlier quoted context omitted.

I'm going to guess that it is part of the "on premises superstition" where companies feel that the stuff they own is more secure somehow. Obviously, this is not often true in practice, but 5 years of research/consulting has taught me that they feel this way all the same. Occasionally, there are laws that also mandate certain controls that cloud providers in general did not have. That is also becoming rarer as time go…

On-Premises is also "one less third-party service to manage". I work for a small company, and was asked to list the external services we use a couple of days ago, and the list went into dozens. It's not the primary reason for On Premises, but it's one less thing to worry about: "our tech team has full control, instead of yet another company having some control that we can't see"

I would argue that from a management point of view it's actually a lot nicer for small places to offload to third parties. As a point, my original comment pointed out that somebody else is on-call for that services uptime.

Re: AWS S3 open source alternative written in Go

#87
post #83

Minio will always be 100% free software / open source. We have no plans to add any proprietary extensions or hold back on features for paying customers only. -- Minio Team

Then why not make the license AGPLv3-or-later, to avoid other people creating proprietary forks? I get that it's not a common occurence within the Golang world, but nothing will change unless more Golang projects start making their code copylefted.

GNU AGPL is an ideal license for free software projects. We are a strong supporter of the GNU project. We chose Apache License for Minio purely for adoption reasons. Most of our users build proprietary software around Minio and their legal council has a default NO policy towards GNU licenses. Besides, FSF has also approved Apache License v2 as a free software license.

Proprietary forks are OK with us. It will be too expensive to maintain branches of their own and catch up with the upstream.

Re: AWS S3 open source alternative written in Go

#88
post #47

Or, run Riak with their S3 compatibility layer. Riak is extremely stable and the work Basho has done to make a truly robust distributed database is significant. http://docs.basho.com/riak/cs/2.1.1/

There's also Pithos from Exoscale. Runs on top of Cassandra. Code is Clojure and open source. Http://Pithos.io

Re: AWS S3 open source alternative written in Go

#89
post #7
post #4

Sorry for two posts (the other one was unrelated). If anyone has experience with this I have a few questions regarding a particular use case. How does something like this behave with really large files. Video files in 100s of gigabytes, for example. I'm asking because if one could set up a resilient online (online as in available) storage with fat pipes like this it could be used as a platform to build a centralized…

I think these days we should by default think of storing blobs of data (like video files) in storage systems like S3 or the alternatives, and that ordinary filesystems should be thought of as a special case where you want to attach storage to an individual computer. Edit: I'm going to elaborate, because people are calling me naïve. Full disclosure: I work at a cloud provider on a storage team. For most people and app…

You're right, but I would reframe it along the lines of Network filesystems (like NFS or OCFS3) vs. Distributed Object Storage (S3). In that sense, certainly, the current "default" is to use the latter and avoid the former.

Local filesystems and/or volume managers won't go away anytime soon. Internally, a system like S3 needs a unified access to the storage, which is provided by the filesystem.

I think we are going to see the emergence of new filesystems that are much simpler in design compared to ZFS (as reliability is left to an upper layer in the stack) for use in the Cloud. Somewhat similar to the trend toward lightweight OSes built for the cloud (CoreOS, Project Atomic, etc.). Many features that were in the realm of the operating system are now delegated to upper layers in the stack.

Re: AWS S3 open source alternative written in Go

#90

Earlier quoted context omitted.

Other alternatives: ceph - http://docs.ceph.com/docs/master/radosgw/s3/ swift - https://wiki.openstack.org/wiki/Swift/APIFeatureComparison#A...

Also ceph (& swift) are known to scale well in prod. clusters with over 30+ PB of data (at least looking at CERN's cluster) and the latest version of RGW does support geographic redundancy for S3 like apis

+1 for Ceph. We're running several ~3.5 PB clusters in production. We've not taken advantage of the new RGW features in Jewel, but it works well as an object storage solution.
Post reply on HN