101–106 of 106 posts
A distributed file system is honed as one of the trickiest software to write, and we are usually advised not to write a file system from scratch (even on top of FUSE), let alone a highly optimized one. When a silicon value company is having the 100th meeting to align god-knows-what, a team of fewer than 60 already came up with a production-grade highly efficient parallel file system. Have we in the valley companies l…
Earlier quoted context omitted.
Seems easy to find: https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/
3FS: 180 nodes, 2x200Gbps InfiniBand and 16x 14TiB NVMe SSDs per node, ~500 clients, 6.6 TiB/s of read throughput with training jobs workload Ceph: 68 nodes, 2x100Gbps Mellanox and 10x 14TiB NVMe SSDs per node, 504 clients, 1TiB/s of FIO random read workload
I also assume that the batch size (block size) is different enough that this alone would make a big difference.
Can someone convince me this isn't NIH syndrome? Why would you use this instead of SeaweedFS, Ceph, or MinIO?
Seems like Ceph is considerably lower in throughput: https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/ A serious concern when saving hundreds of terabytes of weights and optimizer states every now and again, or loading large precomputed prefix KV-caches. Minio seems to be slower still. IDK about SeaweedFS - they don't mention performance in their selling points at all.
The hardware in the Ceph test is only capable of max 1.7TiB/s traffic (optimally without any overhead whatsoever).
I also assume that the batch size (block size) is different enough that this alone would make a big difference.
Earlier quoted context omitted.
3FS: 180 nodes, 2x200Gbps InfiniBand and 16x 14TiB NVMe SSDs per node, ~500 clients, 6.6 TiB/s of read throughput with training jobs workload Ceph: 68 nodes, 2x100Gbps Mellanox and 10x 14TiB NVMe SSDs per node, 504 clients, 1TiB/s of FIO random read workload
The comparison is a little pears to apple. Similar nutritions but different enough to not draw conclusions. The hardware in the Ceph test is only capable of max 1.7TiB/s traffic (optimally without any overhead whatsoever). I also assume that the batch size (block size) is different enough that this alone would make a big difference.
Ceph cluster achieves 1 TiB/s / 1.7 TiB/s = 0.58% of theoretical throughput.
3FS cluster achieves 6.6 TiB/s / 9 TiB/s = 0.73% of theoretical throughput.
Earlier quoted context omitted.
The comparison is a little pears to apple. Similar nutritions but different enough to not draw conclusions. The hardware in the Ceph test is only capable of max 1.7TiB/s traffic (optimally without any overhead whatsoever). I also assume that the batch size (block size) is different enough that this alone would make a big difference.
Even if we take different hardware into account we can readjust for measured vs theoretical throughput. Ceph cluster achieves 1 TiB/s / 1.7 TiB/s = 0.58% of theoretical throughput. 3FS cluster achieves 6.6 TiB/s / 9 TiB/s = 0.73% of theoretical throughput.