AWS S3 open source alternative written in Go • Hacker News Board

Re: AWS S3 open source alternative written in Go

#11

Post by hashin » Tue, Aug 30, 2016, 6:11 PM UTC

ranman wrote: post #10

hansjorg wrote: post #6

Earlier quoted context omitted.

Riak CS is another one: https://github.com/basho/riak_cs

Ran this in production and dealt with a lot of issues. I would caution people against it's use in anything critical or customer facing.

Could you please elaborate it? What were the issues you were facing?

Re: AWS S3 open source alternative written in Go

#12

Post by olalonde » Tue, Aug 30, 2016, 6:11 PM UTC

Previous discussion: https://news.ycombinator.com/item?id=12122998

Re: AWS S3 open source alternative written in Go

#13

Post by llambiel » Tue, Aug 30, 2016, 6:11 PM UTC

frugalmail wrote: post #3

The canonical open source alternative to S3 https://wiki.openstack.org/wiki/Swift

Also http://pithos.io/ backed by Cassandra

Re: AWS S3 open source alternative written in Go

#14

Post by mi100hael » Tue, Aug 30, 2016, 6:11 PM UTC

klodolph wrote: post #7

Keyframe wrote: post #4

Sorry for two posts (the other one was unrelated). If anyone has experience with this I have a few questions regarding a particular use case. How does something like this behave with really large files. Video files in 100s of gigabytes, for example. I'm asking because if one could set up a resilient online (online as in available) storage with fat pipes like this it could be used as a platform to build a centralized…

I think these days we should by default think of storing blobs of data (like video files) in storage systems like S3 or the alternatives, and that ordinary filesystems should be thought of as a special case where you want to attach storage to an individual computer. Edit: I'm going to elaborate, because people are calling me naïve. Full disclosure: I work at a cloud provider on a storage team. For most people and app…

That's awfully naive, especially for tasks like video editing that are significantly impacted by disk read/write speeds. Even a NAS on a gigabit network is going to be roughly 6x slower than a standard internal SATA III spinning disk.

Re: AWS S3 open source alternative written in Go

#15

Post by fizzbatter » Tue, Aug 30, 2016, 6:18 PM UTC

Does this have the ability to mirror to an encrypted remote? I'm looking for something like this for a simple home storage server, but emphasis on being able to replicate to something like B2 Storage for cheap backup.

Currently Infinit.sh has my attention the most, but it's quite young still.

edit: https://news.ycombinator.com/item?id=12125344 this thread seems to be talking about what i want. With that said, i'm not yet sure if `mc mirror` supports Backblaze, as that (per price point) is my prime need

Re: AWS S3 open source alternative written in Go

#16

Post by cdnsteve » Tue, Aug 30, 2016, 6:18 PM UTC

Practical use case:

- Spin up a bunch of droplets on DigitalOcean, because I want reliability, etc.

- What's the best way to share drive space across these to create a single Minio storage volume, so if one DO node goes away I don't lose my stuff?

Re: AWS S3 open source alternative written in Go

#17

Post by spudfkc » Tue, Aug 30, 2016, 6:19 PM UTC

frugalmail wrote: post #3

The canonical open source alternative to S3 https://wiki.openstack.org/wiki/Swift

I use Swift at work, and while it is a great tool, it is a bitch to set up. I would be curious to learn how Minio works more technically on a distributed level: how is object replication handled? are downloads automatically routed to the closest server? can I make downloads temporarily available (think Swift tempURLs)?

Re: AWS S3 open source alternative written in Go

#18

Post by majewsky » Tue, Aug 30, 2016, 6:19 PM UTC

frugalmail wrote: post #3

The canonical open source alternative to S3 https://wiki.openstack.org/wiki/Swift

I run OpenStack Swift at work, currently working on deploying it on Kubernetes. Swift is a very fine piece of software, with a pleasant operations experience, but it will take a lot of time to set up initially. Plan at least half a man-year until you have it all up and running.

Re: AWS S3 open source alternative written in Go

#19

Post by SteveNuts » Tue, Aug 30, 2016, 6:20 PM UTC

cdnsteve wrote: post #16

Practical use case: - Spin up a bunch of droplets on DigitalOcean, because I want reliability, etc. - What's the best way to share drive space across these to create a single Minio storage volume, so if one DO node goes away I don't lose my stuff?

So far the best option I've found has been GlusterFS

Re: AWS S3 open source alternative written in Go

#20

Post by krishnasrinivas » Tue, Aug 30, 2016, 6:21 PM UTC

fizzbatter wrote: post #15

Does this have the ability to mirror to an encrypted remote? I'm looking for something like this for a simple home storage server, but emphasis on being able to replicate to something like B2 Storage for cheap backup. Currently Infinit.sh has my attention the most, but it's quite young still. edit : https://news.ycombinator.com/item?id=12125344 this thread seems to be talking about what i want. With that said, i'm no…

Minio is object-storage server. You can use https://github.com/restic/restic to encrypt and mirror to remote minio server. For more help https://docs.minio.io/docs/restic-with-minio