Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3 open source alternative written in Go

minio.io

Post reply on HN
Print view View original

21–30 of 139 posts

Re: AWS S3 open source alternative written in Go

#21

Post by Keyframe »

mi100hael wrote: post #14
klodolph wrote: post #7

Earlier quoted context omitted.

I think these days we should by default think of storing blobs of data (like video files) in storage systems like S3 or the alternatives, and that ordinary filesystems should be thought of as a special case where you want to attach storage to an individual computer. Edit: I'm going to elaborate, because people are calling me naïve. Full disclosure: I work at a cloud provider on a storage team. For most people and app…

That's awfully naive, especially for tasks like video editing that are significantly impacted by disk read/write speeds. Even a NAS on a gigabit network is going to be roughly 6x slower than a standard internal SATA III spinning disk.

I agree. Network-wise we start at 10GbE. It's a lot more complicated than simple file storage on network though. Many needs and solutions. And I mean MANY.

Re: AWS S3 open source alternative written in Go

#22

Post by krishnasrinivas »

SteveNuts wrote: post #19
cdnsteve wrote: post #16

Practical use case: - Spin up a bunch of droplets on DigitalOcean, because I want reliability, etc. - What's the best way to share drive space across these to create a single Minio storage volume, so if one DO node goes away I don't lose my stuff?

So far the best option I've found has been GlusterFS

Minio is by ex-GlusterFS developers!

Re: AWS S3 open source alternative written in Go

#23

Post by fizzbatter »

krishnasrinivas wrote: post #20
fizzbatter wrote: post #15

Does this have the ability to mirror to an encrypted remote? I'm looking for something like this for a simple home storage server, but emphasis on being able to replicate to something like B2 Storage for cheap backup. Currently Infinit.sh has my attention the most, but it's quite young still. edit : https://news.ycombinator.com/item?id=12125344 this thread seems to be talking about what i want. With that said, i'm no…

Minio is object-storage server. You can use https://github.com/restic/restic to encrypt and mirror to remote minio server. For more help https://docs.minio.io/docs/restic-with-minio

Looks like restic doesn't support backblaze, as of yet: https://github.com/restic/restic/issues/512

Re: AWS S3 open source alternative written in Go

#24

Post by squiguy7 »

cdnsteve wrote: post #16

Practical use case: - Spin up a bunch of droplets on DigitalOcean, because I want reliability, etc. - What's the best way to share drive space across these to create a single Minio storage volume, so if one DO node goes away I don't lose my stuff?

I was going to suggest using their new block storage but I read the docs some more:

> A volume may only be attached to one Droplet at a time. However, up to five volumes can be attached to a single Droplet.

Looks like you would have to roll your own solution.

Re: AWS S3 open source alternative written in Go

#25

Post by krishnasrinivas »

cdnsteve wrote: post #16

Practical use case: - Spin up a bunch of droplets on DigitalOcean, because I want reliability, etc. - What's the best way to share drive space across these to create a single Minio storage volume, so if one DO node goes away I don't lose my stuff?

We are working on distributed minio https://github.com/minio/minio/tree/distributed

The minio available today for production use can export single disk or aggregate multiple disks on the same machine using erasure coding.

For this, if you want backup you can use github.com/minio/mc tool to mirror, more help here https://docs.minio.io/docs/minio-client-complete-guide#mirro...

Re: AWS S3 open source alternative written in Go

#26

Post by krishnasrinivas »

cdnsteve wrote: post #16

Practical use case: - Spin up a bunch of droplets on DigitalOcean, because I want reliability, etc. - What's the best way to share drive space across these to create a single Minio storage volume, so if one DO node goes away I don't lose my stuff?

We are working on distributed minio https://github.com/minio/minio/tree/distributed

The minio available today for production use can export single disk or aggregate multiple disks on the same machine using erasure coding.

For single node version if you want backup you can use https://github.com/minio/mc tool to mirror, more help here https://docs.minio.io/docs/minio-client-complete-guide#mirro...

Re: AWS S3 open source alternative written in Go

#27

Post by howeyc »

fizzbatter wrote: post #23
krishnasrinivas wrote: post #20

Earlier quoted context omitted.

Minio is object-storage server. You can use https://github.com/restic/restic to encrypt and mirror to remote minio server. For more help https://docs.minio.io/docs/restic-with-minio

Looks like restic doesn't support backblaze, as of yet: https://github.com/restic/restic/issues/512

True, but if you have the space to hold the encrypted data, you can "rclone"[0] that to most clouds.

[0] http://rclone.org/

Re: AWS S3 open source alternative written in Go

#28

Post by hansjorg »

ranman wrote: post #10
hansjorg wrote: post #6

Earlier quoted context omitted.

Riak CS is another one: https://github.com/basho/riak_cs

Ran this in production and dealt with a lot of issues. I would caution people against it's use in anything critical or customer facing.

For what it's worth, I've worked with it for a couple of customers with pretty large Riak stores and never ran into or heard of any problems myself.

I've used the official JS aws-sdk and boto3 as clients.

Re: AWS S3 open source alternative written in Go

#29

Post by krishnasrinivas »

Keyframe wrote: post #4

Sorry for two posts (the other one was unrelated). If anyone has experience with this I have a few questions regarding a particular use case. How does something like this behave with really large files. Video files in 100s of gigabytes, for example. I'm asking because if one could set up a resilient online (online as in available) storage with fat pipes like this it could be used as a platform to build a centralized…

Uses cases like these are a really good fit for Minio. i.e videos/photos ... actually any blob/file.

Re: AWS S3 open source alternative written in Go

#30

Post by brightball »

fizzbatter wrote: post #15

Does this have the ability to mirror to an encrypted remote? I'm looking for something like this for a simple home storage server, but emphasis on being able to replicate to something like B2 Storage for cheap backup. Currently Infinit.sh has my attention the most, but it's quite young still. edit : https://news.ycombinator.com/item?id=12125344 this thread seems to be talking about what i want. With that said, i'm no…

I've been wondering about this for that use case myself: https://ipfs.io/
Post reply on HN