Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire-Flyer File System (3FS)

github.com

Post reply on HN
Print view View original

31–40 of 106 posts

Re: Fire-Flyer File System (3FS)

#31

Post by jpgvm »

yalogin wrote: post #30

It’s not clear to me where and how the current popular systems fall short. Do they talk about I anywhere? Also, what specifically is the data access patterns for training and inference that are different from traditional use cases?

Well current popular systems are pretty much limited to Lustre and the new kid Weka, mostly Lustre though tbh.

You can try to use "standard" options like MinIO/Ceph(RADOS)/SeaweedFS but you will very quickly learn those systems aren't remotely fast enough for these usecases.

AI training is what this is used for, not inference (which has absolutely no need for any filesystem at all). What makes the workload somewhat special is that it's entirely random read and not cacheable at all as most reads are one and done.

Would Lustre be perfectly fine at 6TiB/s? Yes. Is it a huge pain in the ass to operate and make remotely highly available? Also yes. If this thing is capable of the throughput but easier to operate and generally more modern and less baroque it's probably an improvement. TLDR is Lustre is fast but that is literally it's only redeeming quality. I have lost far too many hours of my life to the Lustre gods.

Re: Fire-Flyer File System (3FS)

#33

Post by rvba »

ammo1662 wrote: post #13

For those who are interested, the design was originally published here: (Chinese) https://www.high-flyer.cn/blog/3fs/ This file system has been developed and utilized by them for several years . Compared to the traditional file systems, it is more focused on model training that contains a lot of random reads. Read cache and prefetching are useless in this case. Therefore, they designed the file system without those f…

Why is that a random read? Also is it truely random, or from seed? But if prng then they could cache right?

Re: Fire-Flyer File System (3FS)

#34

Post by pella »

related research paper (english - html ) - https://arxiv.org/html/2408.14158v2

arXiv:2408.14158v2 [cs.DC] 31 Aug 2024

"Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning"

Abstract:

"The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic hardware-software co-design framework and its best practices. For DL training, we deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved performance approximating the DGX-A100 while reducing costs by half and energy consumption by 40%. We specifically engineered HFReduce to accelerate allreduce communication and implemented numerous measures to keep our Computation-Storage Integrated Network congestion-free. Through our software stack, including HaiScale, 3FS, and HAI-Platform, we achieved substantial scalability by overlapping computation and communication. Our system-oriented experience from DL training provides valuable insights to drive future advancements in AI-HPC."

Re: Fire-Flyer File System (3FS)

#36

Post by kilburn »

rvba wrote: post #33
ammo1662 wrote: post #13

For those who are interested, the design was originally published here: (Chinese) https://www.high-flyer.cn/blog/3fs/ This file system has been developed and utilized by them for several years . Compared to the traditional file systems, it is more focused on model training that contains a lot of random reads. Read cache and prefetching are useless in this case. Therefore, they designed the file system without those f…

Why is that a random read? Also is it truely random, or from seed? But if prng then they could cache right?

Random is prng. They still cannot cache though because they do many reading "passes" through the same data.

If build a cache that gets hits on the first pass, then it won't work for the second and later passes.

Re: Fire-Flyer File System (3FS)

#37

Post by tucnak »

rfoo wrote: post #19
startupsfail wrote: post #3

Earlier quoted context omitted.

It’s not. When you are a high frequency trader and you’ve mastered RDMA, everything around you looks slow. You are thinking in terms of 20 nanoseconds intervals, while everyone around still thinks that serving a query under a millisecond is fast.

Huh? What kind of RDMA has a completion latency of 20 nanoseconds? It's more like 5 microseconds. I agree that a lot of "modern" storage stack is way too slow though, tried to find a replication-first object storage for crazy-fast random read in small number of objects last year and found none.

Completion latency is one thing, bandwidth would be another. There's apparently a whole world of Alveo SmartNIC's and related FPGA platforms, and it can totally get in nanosecond range for whatever nails that may fit the compute-in-network hammer, even if bound by latency of the consuming system / RDMA interface. Also: https://github.com/corundum/corundum is really popular with the Chinese!

Re: Fire-Flyer File System (3FS)

#39

Post by amelius »

tetron wrote: post #12

Was curious how they get such performance with a FUSE based design. It seems that they sort of cheat, FUSE is used to manage metadata but to get high performance you have to link in the C++ client library and do all your reads and writes through that. So it isn't general purpose, you have to modify your application to take advantage of it. Still, that's a clever trick, and makes me wonder if there's a LD_PRELOAD stra…

Why is FUSE that much slower than providing your own read/write functions? I get that it has to go through the kernel, but the operations are on entire blocks and network should be the bottleneck by far (and disk/main memory should be a bottleneck if the data is local).

Re: Fire-Flyer File System (3FS)

#40

Post by wenyuanyu »

WithinReason wrote: post #29

Why is this even necessary? Can you just shard your training set to the training nodes ahead of time instead?

No, besides accessing training data, there is also logging and checkpointing... When you run k8s over it, and there are multiple training jobs... isolated local storage is a nightmare...
Post reply on HN