The comments already mention several alternatives (Minio, Ceph, GarageFS). I think another one, not mentioned yet, is JuiceFS [1]. Found one comparison here [2].
[2] https://dzone.com/articles/seaweedfs-vs-juicefs-in-design-an...
SeaweedFS fast distributed storage system for blobs, objects, files and datalake
91–100 of 128 posts
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#92Tried and rejected SeaweedFS due to Postgres failing to even initialize itself on a POSIX FS volume mounted over SeaweedFS' CSI driver. And that's too bad, because SeaweedFS was otherwise working well! What we need and haven't identified yet is an SDS system that provides both fully-compliant POSIX FS and S3 volumes, is FOSS, a production story where individuals can do all tasks competently/quickly/effectively (manag…
running something like postgres over a networked filesystem sounds very wrong
But it also sounds like a dream if it could actually work. If you have enough local, performant disk that you are sharing with the cluster you should be able to get good performance and rely on the system to provide resilience and extra space.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#93The comments already mention several alternatives (Minio, Ceph, GarageFS). I think another one, not mentioned yet, is JuiceFS [1]. Found one comparison here [2]. [1] https://juicefs.com/en/ [2] https://dzone.com/articles/seaweedfs-vs-juicefs-in-design-an...
JuiceFS isn't standalone, it requires separate backing storage for each of data [0] and metadata. So for example, JuiceFS would target SeaweedFS or GarageFS as its data store. JuiceFS can also target the local file system, but .. SDS use cases typically care about things like redundancy and availability of the data itself, things that JuiceFS happily delegates. JuiceFS itself can be distributed, but that's merely the control place as I understand it.
[0] https://juicefs.com/docs/community/reference/how_to_set_up_o...
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#94Tried and rejected SeaweedFS due to Postgres failing to even initialize itself on a POSIX FS volume mounted over SeaweedFS' CSI driver. And that's too bad, because SeaweedFS was otherwise working well! What we need and haven't identified yet is an SDS system that provides both fully-compliant POSIX FS and S3 volumes, is FOSS, a production story where individuals can do all tasks competently/quickly/effectively (manag…
You do know that you cannont implement a fully-compliant POSIX FS with only the S3 API? None of the scalalbe SDS' support random writes. Atomic rename (for building transactional systems like lakehouse table formats) is not there. Listing of files is often eventually consistent. The closest functional API to a posix-compliant one in scalable SDS' is the HDFS API. Only ADLS supports that. But then again, they are the…
This is where we learned that! Ceph does it, because separate components are responsible for each of underlying storage, S3 API, and FS API. We tripped on the Seaweed FS and the Garage FS indicia, where "FS" in these contexts typically means File System. But, neither SeaweedFS nor GarageFS is a File System at all; with grace and lenience they could be mildly regarded as Filing Systems, but the reality is that they are actually object stores. SeaweedOS? SeaweedS3?
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#95SeaweedFS does the thing: I've used it to store billions of medium-sized XML documents, image thumbnails, PDF files, etc. It fills the gap between "databases" (broadly defined; maybe you can do few-tens-KByte docs but stretching things) and "filesystems" (hard/inefficient in reality to push beyond tens/hundreds of millions of objects; yes I know it is possible with tuning, etc, but SeaweedFS is better-suited). The do…
When you had corruption and failures, what was the general procedure to deal with that? I love SeaweedFS and want to try it (Neocities is a nearly perfect use case), but part of my concern is not having a manual/documentation for the edge cases so I can figure things out on the fringes. I didn't see any documentation around that when I last looked but maybe I missed something.
(If any SeaweedFS devs are seeing this, having a section of the wiki that describes failure situations and how to manage them would be a huge add-on.)
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#96Earlier quoted context omitted.
Yes and no. While for most application, the GC is hardly an issue and is fast enough, the problem is for application where you need to be able to control exactly when and how memory/objects will be freed. These will never do well with any form of GC. But a looot of software can perform perfectly fine with a GC. If anything, it is mostly Go error handling that is the bigger issue...
Why is Go error handling the bigger issue?
For not disciplined devs (…) it can easily eat errors. Linters catch some of that and of course you can also do that in exception based languages but in those you have to really explicitly put catch {} which is a code smell while missing an error check in go is easier to just ‘forget’. I actually like the go way but not that it’s easy to forget handling; that’s why I prefer a Haskell/Idris return error (monad) way like Go but making it impossible to pass the result without explicitly testing for errors more.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#97Earlier quoted context omitted.
what makes it different is a new way of programming for the cloud era. just fallocate some space to write to, and a file_id is returned. Use the file_id similar to a pointer to a memory block. How is that not mmap? Also what is the difference between a file, an object, a blob, a filesystem and an object store? Is all this just files indexed with sql?
> How is that not mmap? The allocated storage is append only. For updates, just allocate another blob. The deleted blobs would be garbage collected later. So it is not really mmap. > Also what is the difference between a file, an object, a blob, a filesystem and an object store? The answer would be too long to fit here. Maybe chatgpt can help. :) > Is all this just files indexed with sql? Sort of yes.
You made the claim:
what makes it different is a new way of programming for the cloud era.
but you aren't even explaining how anything is different from what a normal file system can do, let alone what makes it a "new way of programming for the cloud era".
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#98SeaweedFS does the thing: I've used it to store billions of medium-sized XML documents, image thumbnails, PDF files, etc. It fills the gap between "databases" (broadly defined; maybe you can do few-tens-KByte docs but stretching things) and "filesystems" (hard/inefficient in reality to push beyond tens/hundreds of millions of objects; yes I know it is possible with tuning, etc, but SeaweedFS is better-suited). The do…
When you had corruption and failures, what was the general procedure to deal with that? I love SeaweedFS and want to try it (Neocities is a nearly perfect use case), but part of my concern is not having a manual/documentation for the edge cases so I can figure things out on the fringes. I didn't see any documentation around that when I last looked but maybe I missed something. (If any SeaweedFS devs are seeing this,…
The dev is suprisingly helpful but yeah I agree the wiki is in need of some beefing up w.r.t operations.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#99Earlier quoted context omitted.
"What if I have 3 raspberry pi 4's [...] with no significant performance requirement?" Ceph is nice, but performance is lackluster on anything but a proper cluster (pun intended). It's also somewhat heavyweight. I ran ZFS over iSCSI with four RPis serving the iSCSI targets via SATA-USB. It was network limited mostly. The advantage of that is that you can take the same disks and plug them all into a single host and im…
I would love to learn more about that setup. Do you have it documented anywhere?
Unfortunately I have not.
The core concept is rather simple though: iSCSI allows for an iSCSI server to expose raw block devices (called targets[1]), so do that and use them from a client machine as-if they were locally connected disks.
So I used LIO[2] as the iSCSI server on each of the Pi's, making sure to use the "by-id" to reference the block device so it would work fine across reboots (skipped changing "directories"):
sudo targetcli
/backstores/block> create name=block0 dev=/dev/disk/by-id/usb-SAMSUNG_MZ7PD256HAFV-000_0123456789000000005-0:0
/iscsi> create
/iscsi> cd iqn.2003-01.org.linux-iscsi.rpi4.armv7l:sn.fc1b1c9879a1/
/iscsi/iqn.20....fc1b1c9879a1> cd tpg1/luns
/iscsi/iqn.20...9a1/tpg1/luns> create /backstores/iblock/block0
Then you create the portal which exposes the target (backed by the raw disk) to the network as normal (see Debian guide fex). I did it like this, you have to adjust for the name of your device:
/iscsi/iqn.20...1c9879a1/tpg1> cd portals
/iscsi/iqn.20.../tpg1/portals> delete 0.0.0.0 3260
/iscsi/iqn.20.../tpg1/portals> create 10.1.1.101
/iscsi/iqn.20.../tpg1/portals> cd ..
/iscsi/iqn.20...1c9879a1/tpg1> set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1
Next you install the iSCSI client (called initiator)[4], I did this on a separate machine. There I connected to each of the target on each Pi. Once successful you should get a number of disks under /dev/disk/by-path/ip-*, which you can then specify when importing the pool on the iSCSI client machine, say
zpool -d /dev/disk/by-path/ip-X -d /dev/disk/by-path/ip-Y
Btw, I recommend not messing with authentication at first, though it's not terribly difficult to set up.
edit: As mentioned the nice thing about this approach is that it's effectively "harmless". If it doesn't work out for you, you can always put all the disks in a single machine and import the pool as normal.
Also, iSCSI has a lot of robustness built in. For example the iSCSI client will temporarily store writes and re-issue once the target (server) is back online. I safely rebooted one of the Pis while copying data to the pool, for example.
[1]: https://en.wikipedia.org/wiki/ISCSI#Concepts
[2]: https://wiki.debian.org/SAN/iSCSI/LIO
[3]: https://github.com/ClusterLabs/resource-agents/pull/1373
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#100Earlier quoted context omitted.
"Gaming is now the third largest business at Microsoft." After Office and Azure, before Windows. https://www.theverge.com/2024/1/30/24055445/microsoft-q2-202...
Sure. But windows was/is the loss leader for that. No gaming or office without windows. Even Azure benefits from it as it ties into Active Directory with Azure AD. This makes it a completely integrated story. So that they are even still making money off of windows directly is just a direct benefit.
It is amazing how sticky auth is.