41–50 of 106 posts
Earlier quoted context omitted.
enlighten us
my theory is that their own DeepSeek writes the code for them, so they are highly productive.
Earlier quoted context omitted.
> The final aggregate read throughput reached approximately 6.6 TiB/s with background traffic from training jobs. The Ceph team has been working on Crimson for years to get past performance bottlenecks inherent to the HDD-based design. I’m having troubles finding any ceph benchmark results that show any close to 100 GB/s.
Seems easy to find: https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/
Ceph: 68 nodes, 2x100Gbps Mellanox and 10x 14TiB NVMe SSDs per node, 504 clients, 1TiB/s of FIO random read workload
Earlier quoted context omitted.
enlighten us
my theory is that their own DeepSeek writes the code for them, so they are highly productive.
For those who are interested, the design was originally published here: (Chinese) https://www.high-flyer.cn/blog/3fs/ This file system has been developed and utilized by them for several years . Compared to the traditional file systems, it is more focused on model training that contains a lot of random reads. Read cache and prefetching are useless in this case. Therefore, they designed the file system without those f…
Was curious how they get such performance with a FUSE based design. It seems that they sort of cheat, FUSE is used to manage metadata but to get high performance you have to link in the C++ client library and do all your reads and writes through that. So it isn't general purpose, you have to modify your application to take advantage of it. Still, that's a clever trick, and makes me wonder if there's a LD_PRELOAD stra…
Why is FUSE that much slower than providing your own read/write functions? I get that it has to go through the kernel, but the operations are on entire blocks and network should be the bottleneck by far (and disk/main memory should be a bottleneck if the data is local).
I believe there’s work to minimize this using io_uring so that you can talk to the fuse driver without the kernel being in the middle, but that work isn’t ready last time I checked.
For what it’s worth at Palm we had a similar problem because our applications were stored compressed but exposed through fuse uncompressed, instead of O_DIRECT I just did an fadvise to dump the cache after a read. Not as high throughput but the least risky change to get the same effect.
Can someone convince me this isn't NIH syndrome? Why would you use this instead of SeaweedFS, Ceph, or MinIO?
The thing is those stuff are so prevalent those in house tech have reach the point they are competitive. This doubles for quant firm like DeepSeek.
It’s not clear to me where and how the current popular systems fall short. Do they talk about I anywhere? Also, what specifically is the data access patterns for training and inference that are different from traditional use cases?
Well current popular systems are pretty much limited to Lustre and the new kid Weka, mostly Lustre though tbh. You can try to use "standard" options like MinIO/Ceph(RADOS)/SeaweedFS but you will very quickly learn those systems aren't remotely fast enough for these usecases. AI training is what this is used for, not inference (which has absolutely no need for any filesystem at all). What makes the workload somewhat s…
They sure are productive. What are we going to see tomorrow? DeepSeek OS or something?