Live data from Hacker News

AWS S3 open source alternative written in Go

minio.io

91–100 of 139 posts

Re: AWS S3 open source alternative written in Go

#93
post #11
post #10

Earlier quoted context omitted.

Ran this in production and dealt with a lot of issues. I would caution people against it's use in anything critical or customer facing.

Could you please elaborate it? What were the issues you were facing?

As another user with nothing but negative experiences with Riak-CS in production, I thought I'd take a stab here. We had a 12-node cluster with ~10TB per node, fwiw. In no particular order:

- The restart times of the Riak process ranged from 10 minutes to 3+ hours, during which time the cluster was basically useless. Not a single suggestion from support sped up this process.

- Every single night from 0800 - 0900 UTC, the cluster would grind to a halt (as measured by canaries measuring upload/download cycle times). This continued even after we migrated all customer data and traffic off of the cluster.

- Riak-CS ships with garbage collection disabled despite it being a critical feature. I inherited a cluster that had been run for some months without gc enabled. Turning it on caused the cluster to catastrophically fail. Basho Support, over a period of close to a year, was unable to find a single solution that would get our cluster back to health. If our cluster were a house on a show like Hoarders, the garbage in it would be considered load bearing.

- We attempted to upgrade our way out of our un-garbage-collect-able mess, but the transfer crashed. Every. Single. Time.

- Even had transfers worked, all of the bloated manifests have to be copied in their entirety, so you can't gc the incoming data on the new cluster.

- Even while babying the cluster, it would become unusable at least once a month, requiring a restart of all nodes. The slowest node took 3+ hours to start, followed by another 3+ hours of transferring data. This was 6+ hours of system downtime every month.

- During these monthly episodes, we attempted to engage with support and try to debug the processes (we were a team of seasoned Erlang developers). We could attach Observer and/or use the REPL to grab stats, but not a single support resource was able or willing to engage.

- For giggles, once we had migrated all users off of the cluster, we attempted to let gc run. It never completed. Not once. We let this go on for a few months before nuking the entire cluster.

Now, I absolutely realize that we got ourselves into that mess by running the cluster without gc for an extended period. But in the grand scheme of things, this cluster wasn't storing a very large amount of data -- tens of TB spread over tens of millions of objects. Having the cluster get into a state where gc can never run and where this causes snowballing instability is unacceptable.

We switched to Ceph. We've never looked back.

Re: AWS S3 open source alternative written in Go

#94
post #50

How easy is it to embed this into go tests? Right now I use goamz/s3test for that, but it has a lot of limitations.

goofys and s3fs both use s3proxy for this, which works fine as long as you are ok with having Java as a test dependency: https://github.com/kahing/goofys/blob/master/test/run-tests....

Re: AWS S3 open source alternative written in Go

#96
post #47

Or, run Riak with their S3 compatibility layer. Riak is extremely stable and the work Basho has done to make a truly robust distributed database is significant. http://docs.basho.com/riak/cs/2.1.1/

Ran Riak CS in production and had constant issues. It's not terrible but it's also not ideal. I would caution against anyone depending on it for mission critical systems. Many of the failure modes are undocumented.

Re: AWS S3 open source alternative written in Go

#97

I love the website. I'm a lone developer who doesn't know any HTML, how would I go about getting such a nice design for my own projects? (Or how much would it cost)

Wappalyzer (https://wappalyzer.com/) tells me they're using Bootstrap (http://getbootstrap.com/) (probably customized a bit). HTML isn't very difficult (just another markup language) and if you're not inclined toward design (I am also not) there are a plethora of CSS frameworks to choose from (like Bootstrap) that will get you up and running with something not completely ugly. Personally I like Bulma (http://bulma.io/) right now which showed up (I think as a Show HN) on here a while back. Currently using it for a project and I'm enjoying it.

Re: AWS S3 open source alternative written in Go

#98
post #11
post #10

Earlier quoted context omitted.

Ran this in production and dealt with a lot of issues. I would caution people against it's use in anything critical or customer facing.

Could you please elaborate it? What were the issues you were facing?

We didn't have any issues with lost data but we had a lot of operational issues that didn't have clear fixes. Primarily around TLS, migrations, and performance. We had to contact support for many issues because the documentation for various failure modes wasn't there.

Re: AWS S3 open source alternative written in Go

#99

I love the website. I'm a lone developer who doesn't know any HTML, how would I go about getting such a nice design for my own projects? (Or how much would it cost)

Wappalyzer ( https://wappalyzer.com/ ) tells me they're using Bootstrap ( http://getbootstrap.com/ ) (probably customized a bit). HTML isn't very difficult (just another markup language) and if you're not inclined toward design (I am also not) there are a plethora of CSS frameworks to choose from (like Bootstrap) that will get you up and running with something not completely ugly. Personally I like Bulma ( http://bul…

Really my design sense isn't great, given time I can hack together something with bootstrap, but I do think I lack the designer training and probably instincts

Re: AWS S3 open source alternative written in Go

#100

Earlier quoted context omitted.

Wappalyzer ( https://wappalyzer.com/ ) tells me they're using Bootstrap ( http://getbootstrap.com/ ) (probably customized a bit). HTML isn't very difficult (just another markup language) and if you're not inclined toward design (I am also not) there are a plethora of CSS frameworks to choose from (like Bootstrap) that will get you up and running with something not completely ugly. Personally I like Bulma ( http://bul…

Really my design sense isn't great, given time I can hack together something with bootstrap, but I do think I lack the designer training and probably instincts

Same here. I recommend making friends with some designers or looking around at pre-customized versions of Bootstrap. I also spent some time looking for this: http://jgthms.com/web-design-in-4-minutes/ One of my favorite sites for this type of conversation.
Post reply on HN