Do we know the largest size of data that these object stores have handled? People seem have been moving away from HDFS, yet companies could host exabytes of data on HDFS and serve TBs of scans per second with a single team of fewer than 15 people. I was wondering how production-ready the other OSS alternatives are for such scale of data.
Garage: An open-source distributed object storage service
31–40 of 81 posts
Re: Garage: An open-source distributed object storage service
#32Just finished installing it on my OpenIndiana NAS to replace Minio. Biggest difference so far is that Minio is just files on disk, Garage chunks all files and has a metadata db. Minios listing operations were horribly slow, still have to see if Garage resolves that.
> Biggest difference so far is that Minio is just files on disk, Garage chunks all files and has a metadata db. I'd kind of expect most blob storage solutions to use abstractions other than just the file system, or at least consider doing so. I recently built a system to handle millions of documents as a proof of concept and when I was testing it with 10 million files, the server ran out of inodes, before I went over…
> I'd kind of expect most blob storage solutions to use abstractions other than just the file system, or at least consider doing so.
Honestly, I'd expect the exact opposite. Filesystems are really good at storing files. Why not leverage all that work?
> I recently built a system to handle millions of documents as a proof of concept and when I was testing it with 10 million files, the server ran out of inodes, before I went over to storing the blobs in some attached storage that had XFS
That's a misconfiguration issue though, not a reason to not store blobs as files on disk. Ext4 can handle 2^32 files. ZFS can handle 2^128(?).
> With abstracted storage (say, files bunches up into X MB large containers or chunked into such when too large, with something else to keep track of what is where) that wouldn't be such an issue, though you might end up with other issues along the way.
A few issues that come to mind for me:
* This requires tuning to actually reduce the number of inodes of used for certain datasets. E.g., if I'm storing large media files, that chunking would _increase_ the number of files on disk, not reduce it. At which point, if inode limits are the issue, we're just making it worse.
* It adds additional complexity. Now you need to account for these chunks, and, if you care about the data, check it periodically.
* You need specific tooling to work with it. Files on a filesystem are.. files on a filesystem. Easy to backup, easy to view. Arbitrary chunking and such requires tooling to perform operations on it. Tooling that may break, or have the wrong versions, or.. etc.
> It's curious that we don't advocate for storing blobs in relational databases anymore, even though I can also understand the reasoning
In my experience, the popular RDBMS out there just aren't good at it. With the way locking semantics and their transaction queueing works, storing and retrieving lots of blobs just isn't performant. You can get away with it for a long time though, and it can be pretty nice when you can.
Re: Garage: An open-source distributed object storage service
#33Kind of topic. But I’m wondering if it’s possible to get comparable pricing when self-hosting a solution similar to this in the cloud vs say AWS S3.
Re: Garage: An open-source distributed object storage service
#34A very good alternative is seaweedfs https://github.com/chrislusf/seaweedfs/ based on facebook haystack paper (efficient small files) & more.
Did you pull that URL out of some blog post or something? The accurate URL is https://github.com/seaweedfs/seaweedfs
Re: Garage: An open-source distributed object storage service
#35Kind of topic. But I’m wondering if it’s possible to get comparable pricing when self-hosting a solution similar to this in the cloud vs say AWS S3.
Yes. Specially S3 which is way overpriced. I have setup already a multi-region minio cluster at the cost of $0.0039/GB.
The only "cloud" solution that could get closer to that is Storj. [0]
Re: Garage: An open-source distributed object storage service
#36Earlier quoted context omitted.
This doesn’t seem accurate. Isn’t AGPL viral across RPC boundaries requiring open sourcing not just the service but all supporting code for that service?
As far as I know, this isn't true. Some AGPL users claim that it is a requirement of the AGPL, but I am also unaware of any litigation that substantiates that reading of the license. Can you cite cases that have established this as fact?
I would suspect no one wants to invest the legal team or time to be the "trailblazer" court case to find out whether your theory or the common interpretation is correct. The "just ban AGPL" stance is by far the safer route since it's not like there are no sane replacements for AGPL stuff
IANAL, and thus far my life is worse for any interaction with the legal system
Re: Garage: An open-source distributed object storage service
#37Earlier quoted context omitted.
This doesn’t seem accurate. Isn’t AGPL viral across RPC boundaries requiring open sourcing not just the service but all supporting code for that service?
No it's not. From a practical standpoint, I'm not even sure how that could work. You would have to require all browsers to be open source AGPL in order to load a web page served by it. By way of analogy it seems the equivalent of requiring the mouse and keyboard firmware to be licensed the same as the operating system. A real life example is Instructure, which makes Canvas (which is agpl) but has other proprietary se…
> require all browsers to be open source AGPL in order to load a web page served by it
Don't be silly: a web server is not distributing a web browser, and thus when you visit news.ycombinator.com, they don't have influence over whether you do that via netcat, curl, or Awesome AGPL Browser 1.0
If, however, they used https://git.deuxfleurs.fr/Deuxfleurs/tricot to serve the http request, then AIUI the AGPL entitles you, as a "13. Remote Network Interaction; Use with the GNU General Public License. (https://opensource.org/licenses/AGPL-3.0)", to ask for the source code of tricot and potentially any systems that it subsequently interacts with
I'm certain I'm going to regret posting this, given how hot-button the AGPL is in every one of these threads
Re: Garage: An open-source distributed object storage service
#38Earlier quoted context omitted.
> Biggest difference so far is that Minio is just files on disk, Garage chunks all files and has a metadata db. I'd kind of expect most blob storage solutions to use abstractions other than just the file system, or at least consider doing so. I recently built a system to handle millions of documents as a proof of concept and when I was testing it with 10 million files, the server ran out of inodes, before I went over…
> I'd kind of expect most blob storage solutions to use abstractions other than just the file system, or at least consider doing so. Honestly, I'd expect the exact opposite. Filesystems are really good at storing files. Why not leverage all that work? > I recently built a system to handle millions of documents as a proof of concept and when I was testing it with 10 million files, the server ran out of inodes, before…
> Filesystems are really good at storing files. Why not leverage all that work?
As an asterisk, the S3 API is key-value pairs, not files; that distinction comes up a lot when interacting with Amazon S3, and I would expect the same with an S3 API clone. For example, ListObjects[1] has a "delimiter" that (AFAIK) defaults to / making it appear to be a filesystem but using "." or "!" would be a perfectly fine delimiter and thus would have no obvious filesystem mapping
1: https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...
Re: Garage: An open-source distributed object storage service
#39Earlier quoted context omitted.
This doesn’t seem accurate. Isn’t AGPL viral across RPC boundaries requiring open sourcing not just the service but all supporting code for that service?
No it's not. From a practical standpoint, I'm not even sure how that could work. You would have to require all browsers to be open source AGPL in order to load a web page served by it. By way of analogy it seems the equivalent of requiring the mouse and keyboard firmware to be licensed the same as the operating system. A real life example is Instructure, which makes Canvas (which is agpl) but has other proprietary se…
Instructure doesn't need to comply with AGPL obligations because it owns the product. It isn't licensing it to itself under the AGPL.
Re: Garage: An open-source distributed object storage service
#40Is there any OSS implementation that is not AGPL?
Why would that matter? Apps using its api don't need to be AGPL
First of all, it isn't clear this is the case.
But this entire comment thread on the AGPL misses the mark. It doesn't matter that the AGPL hasn't been tested in court or what fine grained distinctions you apply to the license or what the AGPL intends. No company in their right mind would risk using software licensed under the AGPL because the result of being wrong would be catastrophic. The legal advice to be skeptical of the AGPL is absolutely right. There is no conceivable reason to ever use AGPL software when you could simply license it under a commerical license or use a non-AGPL alternative.
Generally when someone licenses something under the AGPL they totally understand this and that is their intention.