I am aware of some research into operating systems with a database rather than filesystem as a base layer. If SeaweedFS serves a middle-ground between databases and filesystems, could it also suggest a middle-ground in conceiving of research operating systems??
SeaweedFS fast distributed storage system for blobs, objects, files and datalake
31–40 of 128 posts
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#32SeaweedFS does the thing: I've used it to store billions of medium-sized XML documents, image thumbnails, PDF files, etc. It fills the gap between "databases" (broadly defined; maybe you can do few-tens-KByte docs but stretching things) and "filesystems" (hard/inefficient in reality to push beyond tens/hundreds of millions of objects; yes I know it is possible with tuning, etc, but SeaweedFS is better-suited). The do…
Written in Go no less, a GC language!
I was expecting C/C++ or Rust, pleasantly surprised to see Go.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#33Earlier quoted context omitted.
Forgive my ignorance but why is this preferable to a big ZFS pool?
Not the only reason, but we have a distributed workload so HTTP is a better protocol than NFS for our use case.
Internally ZFS is more or less an object store. I seem to recall there was a project to expose this to users directly, similar to how ZPL (filesystem) or ZVOLs are just two different ways to expose the internal object store.
Seems like it could be an interesting approach.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#34We (https://hivegames.io/) use this for storing 50+ TB of multiplayer match recordings ("replays"), heavily using the built-in expiry feature. It's incredibly easy to use and to built on top off; never had an issue updating, migrating or utilizing new features.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#35SeaweedFS does the thing: I've used it to store billions of medium-sized XML documents, image thumbnails, PDF files, etc. It fills the gap between "databases" (broadly defined; maybe you can do few-tens-KByte docs but stretching things) and "filesystems" (hard/inefficient in reality to push beyond tens/hundreds of millions of objects; yes I know it is possible with tuning, etc, but SeaweedFS is better-suited). The do…
> almost-sort-kinda solutions like base64-encoded bytes in HBase/postgresql/etc
why you would base64 encode them, they all store binary formats?
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#36Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#37This sounds like what Microsoft has tried but failed to do in numerous iterations for two decades: OFS (Cairo, unreleased predecessor to Windows 95), Storage+ (SQL Server 7.0), RFS (SQL Server 2000), Exchange Webstore, Outlook LIS, WinFS, and finally Microsoft Semantic Engine. All projects were either cancelled, features cut, or officially left in limbo. It's a pretty remarkable piece of Microsoft history as it has b…
>> and indeed in 2013, he cited the failure of WinFS as his greatest disappointment at Microsoft,
Failing to capture any of the mobile handset market while missing out almost entirely on search and social media businesses would be higher on my list if I were in BG's shoes.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#38We ( https://hivegames.io/ ) use this for storing 50+ TB of multiplayer match recordings ("replays"), heavily using the built-in expiry feature. It's incredibly easy to use and to built on top off; never had an issue updating, migrating or utilizing new features.
What do you use for the metadata store?
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#39I am aware of some research into operating systems with a database rather than filesystem as a base layer. If SeaweedFS serves a middle-ground between databases and filesystems, could it also suggest a middle-ground in conceiving of research operating systems??
SeaweedFS is a non hierarchical distributed key value store. It makes different tradeoffs to a filesystem which provides a hierarchical view of local only data. There’s some evidence to suggest that a hierarchical structuring of the data itself isn’t actually beneficial for modern systems. And you could design a system that used similar techniques to SeaweedFS to do a semi-distributed local store (ie locally stored data for fast access with offloading to cheap remote storage for durability / infinite extensibility). And the plain KV store will likely be faster for most operations although in practice you’ll probably only see it in micro benchmarks.
Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake
#40Thanks for sharing! I work on SeaweedFS.
SeaweedFS is built on top of a blob storage based on Facebook's Haystack paper. The features are not fully developed yet, but what makes it different is a new way of programming for the cloud era.
When needing some storage, just fallocate some space to write to, and a file_id is returned. Use the file_id similar to a pointer to a memory block.
There will be more features built on top of it. File system and Object store are just a couple of them. Need more help on this.