Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SeaweedFS fast distributed storage system for blobs, objects, files and datalake

github.com

Post reply on HN
Print view View original

41–50 of 128 posts

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#41

Post by chrislusf »

mbreese wrote: post #19

Does anyone know how well the Seaweed Filer works for renaming files or missing files? My use case’s involves writing a lot of data to temporary files that are then renamed to their final name. This is always the Achilles heel for distributed file storage, where files are put into buckets based on the file path… when you rename the path, but keep the data, lookups become more complicated. (This is HPC work with large…

Should not be a problem.

One similar use case used Cassandra as SeaweedFS filer store, and created thousands of files per second in a temp folder, and moved the files to a final folder. It caused a lot of tombstones for the updates in Cassandra.

Later, they changed to use Redis for the temp folder, and keep Cassandra for other folders. Everything has been very smooth since then.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#42

Post by CyberDildonics »

chrislusf wrote: post #40

Thanks for sharing! I work on SeaweedFS. SeaweedFS is built on top of a blob storage based on Facebook's Haystack paper. The features are not fully developed yet, but what makes it different is a new way of programming for the cloud era. When needing some storage, just fallocate some space to write to, and a file_id is returned. Use the file_id similar to a pointer to a memory block. There will be more features built…

what makes it different is a new way of programming for the cloud era.

just fallocate some space to write to, and a file_id is returned. Use the file_id similar to a pointer to a memory block.

How is that not mmap?

Also what is the difference between a file, an object, a blob, a filesystem and an object store? Is all this just files indexed with sql?

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#43

Post by chillfox »

bomewish wrote: post #11
4by4by4 wrote: post #4

We tested both SeaweedFS and Min.io for cheaply (HDD) storing > 100TB of audio data. Seaweed had much better performance for our use case.

Forgive my ignorance but why is this preferable to a big ZFS pool?

Because you need an S3 compatible API?

I use ZFS for most of my things, but I have yet to find a good way of just sharing a ZFS dataset over S3.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#45

Post by hardwaresofton »

jakjak123 wrote: post #27

Have used SeaweedFS to store billions of thumbnails. The tooling is a bit clunky, but it mostly works. The performance is very good for small-ish objects (memory usage + latency), and latency remains consistently good into 99.9 percentiles. We had some issues with data loss and downtime, but that was mostly our own fault.

What issues did you run into? Not settling replication?

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#46

Post by mbreese »

osigurdson wrote: post #37
jug wrote: post #21

This sounds like what Microsoft has tried but failed to do in numerous iterations for two decades: OFS (Cairo, unreleased predecessor to Windows 95), Storage+ (SQL Server 7.0), RFS (SQL Server 2000), Exchange Webstore, Outlook LIS, WinFS, and finally Microsoft Semantic Engine. All projects were either cancelled, features cut, or officially left in limbo. It's a pretty remarkable piece of Microsoft history as it has b…

>> and indeed in 2013, he cited the failure of WinFS as his greatest disappointment at Microsoft, Failing to capture any of the mobile handset market while missing out almost entirely on search and social media businesses would be higher on my list if I were in BG's shoes.

Intellectually though, I could see the WinFS failure as being more disappointing. If it had worked, local computing would have been completely different. Much like BeFS, WinFS (as marketed) would have introduced many new ways to interact with your computer and data.

Having a bigger presence in mobile and social would have been more lucrative, but from a CS geek point of view, the failure of WinFS might have been more stinging.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#47

Post by Guthur »

osigurdson wrote: post #37
jug wrote: post #21

This sounds like what Microsoft has tried but failed to do in numerous iterations for two decades: OFS (Cairo, unreleased predecessor to Windows 95), Storage+ (SQL Server 7.0), RFS (SQL Server 2000), Exchange Webstore, Outlook LIS, WinFS, and finally Microsoft Semantic Engine. All projects were either cancelled, features cut, or officially left in limbo. It's a pretty remarkable piece of Microsoft history as it has b…

>> and indeed in 2013, he cited the failure of WinFS as his greatest disappointment at Microsoft, Failing to capture any of the mobile handset market while missing out almost entirely on search and social media businesses would be higher on my list if I were in BG's shoes.

Microsoft has never been good in either consumer electronics or advertising (social media).

MS carved out economic rent on business with Windows and Office, Apple actually failed at that.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#48

Post by foota »

osigurdson wrote: post #37
jug wrote: post #21

This sounds like what Microsoft has tried but failed to do in numerous iterations for two decades: OFS (Cairo, unreleased predecessor to Windows 95), Storage+ (SQL Server 7.0), RFS (SQL Server 2000), Exchange Webstore, Outlook LIS, WinFS, and finally Microsoft Semantic Engine. All projects were either cancelled, features cut, or officially left in limbo. It's a pretty remarkable piece of Microsoft history as it has b…

>> and indeed in 2013, he cited the failure of WinFS as his greatest disappointment at Microsoft, Failing to capture any of the mobile handset market while missing out almost entirely on search and social media businesses would be higher on my list if I were in BG's shoes.

Did those happen under Gates?

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#49

Post by chrislusf »

CyberDildonics wrote: post #42
chrislusf wrote: post #40

Thanks for sharing! I work on SeaweedFS. SeaweedFS is built on top of a blob storage based on Facebook's Haystack paper. The features are not fully developed yet, but what makes it different is a new way of programming for the cloud era. When needing some storage, just fallocate some space to write to, and a file_id is returned. Use the file_id similar to a pointer to a memory block. There will be more features built…

what makes it different is a new way of programming for the cloud era. just fallocate some space to write to, and a file_id is returned. Use the file_id similar to a pointer to a memory block. How is that not mmap? Also what is the difference between a file, an object, a blob, a filesystem and an object store? Is all this just files indexed with sql?

> How is that not mmap?

The allocated storage is append only. For updates, just allocate another blob. The deleted blobs would be garbage collected later. So it is not really mmap.

> Also what is the difference between a file, an object, a blob, a filesystem and an object store?

The answer would be too long to fit here. Maybe chatgpt can help. :)

> Is all this just files indexed with sql?

Sort of yes.

Re: SeaweedFS fast distributed storage system for blobs, objects, files and datalake

#50

Post by redundantly »

magicalhippo wrote: post #30
chaxor wrote: post #26

Earlier quoted context omitted.

I could be wrong here, but I believe this (ceph, et al) is the answer to the question: > """But what if I don't have a JBOD of 6x18TB hard drives with good amount of ECC RAM for ZFS? What if I have 3 raspberry pi 4's, at different houses with 3x 12TB externals on them, and 2 other computers with 2x 4TB externals on them, and I want to use that all together with some redundancy/error checking?" That would give (3x3x12…

"What if I have 3 raspberry pi 4's [...] with no significant performance requirement?" Ceph is nice, but performance is lackluster on anything but a proper cluster (pun intended). It's also somewhat heavyweight. I ran ZFS over iSCSI with four RPis serving the iSCSI targets via SATA-USB. It was network limited mostly. The advantage of that is that you can take the same disks and plug them all into a single host and im…

I would love to learn more about that setup. Do you have it documented anywhere?
Post reply on HN