Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3 open source alternative written in Go

minio.io

Post reply on HN
Print view View original

31–40 of 139 posts

Re: AWS S3 open source alternative written in Go

#31

Post by y4m4b4 »

spudfkc wrote: post #17
frugalmail wrote: post #3

The canonical open source alternative to S3 https://wiki.openstack.org/wiki/Swift

I use Swift at work, and while it is a great tool, it is a bitch to set up. I would be curious to learn how Minio works more technically on a distributed level: how is object replication handled? are downloads automatically routed to the closest server? can I make downloads temporarily available (think Swift tempURLs)?

We are currently working on the distributed version and will be making a beta release soon.

Currently minio supports

- pure FS backend with single disk - pure Erasure coded backend with multiple disks on single node (like ZFS)

For more information you can read here - https://docs.minio.io/docs/minio-erasure-code-quickstart-gui...

We do not do any sort of replication erasure code handles disk failures and we also implement transparent bit-rot protection as well.

To replicate one setup to many you can use 'mc mirror -w' which would watch on events and do continuous replication.

Relevant docs can be found here

https://docs.minio.io/docs/minio-client-complete-guide#mirro...

Re: AWS S3 open source alternative written in Go

#32

Post by Keyframe »

krishnasrinivas wrote: post #29
Keyframe wrote: post #4

Sorry for two posts (the other one was unrelated). If anyone has experience with this I have a few questions regarding a particular use case. How does something like this behave with really large files. Video files in 100s of gigabytes, for example. I'm asking because if one could set up a resilient online (online as in available) storage with fat pipes like this it could be used as a platform to build a centralized…

Uses cases like these are a really good fit for Minio. i.e videos/photos ... actually any blob/file.

Would one expect any issues with large files? Can one file span machines? For example, you have a 1TB single file, but one machine has 500GB free and the other 200GB and third 400GB or whatever (stupid example).

I really think this could be useful to build something like Avid Interplay on top of.

Re: AWS S3 open source alternative written in Go

#33

Post by fizzbatter »

howeyc wrote: post #27
fizzbatter wrote: post #23

Earlier quoted context omitted.

Looks like restic doesn't support backblaze, as of yet: https://github.com/restic/restic/issues/512

True, but if you have the space to hold the encrypted data, you can "rclone"[0] that to most clouds. [0] http://rclone.org/

That's really cool, appreciated! Surprised it doesn't offer an encrypt feature, seems really useful for the given featureset

Re: AWS S3 open source alternative written in Go

#34

Post by orestes910 »

klodolph wrote: post #7
Keyframe wrote: post #4

Sorry for two posts (the other one was unrelated). If anyone has experience with this I have a few questions regarding a particular use case. How does something like this behave with really large files. Video files in 100s of gigabytes, for example. I'm asking because if one could set up a resilient online (online as in available) storage with fat pipes like this it could be used as a platform to build a centralized…

I think these days we should by default think of storing blobs of data (like video files) in storage systems like S3 or the alternatives, and that ordinary filesystems should be thought of as a special case where you want to attach storage to an individual computer. Edit: I'm going to elaborate, because people are calling me naïve. Full disclosure: I work at a cloud provider on a storage team. For most people and app…

Can you help me understand this statement better? Why should we do that?

I may sound like I'm playing dumb, but I'm really struggling to see whats compelling about this in its current state aside from the fact that its one tool as opposed to a RAID + filesystem + something to make the data available.

Re: AWS S3 open source alternative written in Go

#36

Post by zzzcpan »

Keyframe wrote: post #4

Sorry for two posts (the other one was unrelated). If anyone has experience with this I have a few questions regarding a particular use case. How does something like this behave with really large files. Video files in 100s of gigabytes, for example. I'm asking because if one could set up a resilient online (online as in available) storage with fat pipes like this it could be used as a platform to build a centralized…

It's a bit confusing, but minio is not a resilient storage. It's just a server, kind of like webdav, but with s3 api and a capability to use multiple filesystem folders with erasure coding.

As for distributed object storages, I would expect them to work great for video editing, since they can saturate any link given enough servers. But not out of the box, you would need a client designed for it, splitting files into chunks in parallel, etc.

Re: AWS S3 open source alternative written in Go

#37

Post by alfalfasprout »

klodolph wrote: post #7
Keyframe wrote: post #4

Sorry for two posts (the other one was unrelated). If anyone has experience with this I have a few questions regarding a particular use case. How does something like this behave with really large files. Video files in 100s of gigabytes, for example. I'm asking because if one could set up a resilient online (online as in available) storage with fat pipes like this it could be used as a platform to build a centralized…

I think these days we should by default think of storing blobs of data (like video files) in storage systems like S3 or the alternatives, and that ordinary filesystems should be thought of as a special case where you want to attach storage to an individual computer. Edit: I'm going to elaborate, because people are calling me naïve. Full disclosure: I work at a cloud provider on a storage team. For most people and app…

Uhhh have you seen the internet data transfer costs for S3? That would become absurdly expensive quickly. Even with a dedicated cross connect.

Re: AWS S3 open source alternative written in Go

#38

Post by y4m4b4 »

Keyframe wrote: post #32
krishnasrinivas wrote: post #29

Earlier quoted context omitted.

Uses cases like these are a really good fit for Minio. i.e videos/photos ... actually any blob/file.

Would one expect any issues with large files? Can one file span machines? For example, you have a 1TB single file, but one machine has 500GB free and the other 200GB and third 400GB or whatever (stupid example). I really think this could be useful to build something like Avid Interplay on top of.

Yes definitely you can read our docs here https://docs.minio.io/docs/minio-erasure-code-quickstart-gui... for more understanding and even hardware recommendations.

Re: AWS S3 open source alternative written in Go

#39

Post by nickpsecurity »

klodolph wrote: post #7
Keyframe wrote: post #4

Sorry for two posts (the other one was unrelated). If anyone has experience with this I have a few questions regarding a particular use case. How does something like this behave with really large files. Video files in 100s of gigabytes, for example. I'm asking because if one could set up a resilient online (online as in available) storage with fat pipes like this it could be used as a platform to build a centralized…

I think these days we should by default think of storing blobs of data (like video files) in storage systems like S3 or the alternatives, and that ordinary filesystems should be thought of as a special case where you want to attach storage to an individual computer. Edit: I'm going to elaborate, because people are calling me naïve. Full disclosure: I work at a cloud provider on a storage team. For most people and app…

I disagree. I think we should default on storing blobs of data in local storage to retain full legal and technological control of them. Storing them in 3rd party services under their EULA's, SLA's, and API's should be a special case to improve attributes of data like its availability or cost of distribution. The way most people and companies use them now. :)

Re: AWS S3 open source alternative written in Go

#40

Post by koolhead17 »

fizzbatter wrote: post #33
howeyc wrote: post #27

Earlier quoted context omitted.

True, but if you have the space to hold the encrypted data, you can "rclone"[0] that to most clouds. [0] http://rclone.org/

That's really cool, appreciated! Surprised it doesn't offer an encrypt feature, seems really useful for the given featureset

We have documented how to use Minio with Rclone https://docs.minio.io/docs/rclone-with-minio-server hope it helps.
Post reply on HN