Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire-Flyer File System (3FS)

github.com

Post reply on HN
Print view View original

81–90 of 106 posts

Re: Fire-Flyer File System (3FS)

#81

Post by hintymad »

htrp wrote: post #75
hintymad wrote: post #69

A distributed file system is honed as one of the trickiest software to write, and we are usually advised not to write a file system from scratch (even on top of FUSE), let alone a highly optimized one. When a silicon value company is having the 100th meeting to align god-knows-what, a team of fewer than 60 already came up with a production-grade highly efficient parallel file system. Have we in the valley companies l…

> team of fewer than 10 the highflyer team are pretty well resourced.... think they have more than 10 people

Thanks! Updated to 60 per their author list in their paper.

Re: Fire-Flyer File System (3FS)

#82

Post by bee_rider »

htrp wrote: post #77
bee_rider wrote: post #60

Earlier quoted context omitted.

As someone who did some simulation focused engineering grad school stuff; there is a tendency for some of the best to go become quants. Does the field need it? I don’t know. But for whatever reason the draw of the “print money using math tricks” seems to attract some hardcore folks, haha. It is really frustrating to see good engineers go to play trading games. We should study how exactly it is China managed to unlock…

Government effectively banned unproductive tech (adtech, fintech etc) and told ppl to go do stuff like robotics and AI

I’m beginning to suspect this invisible hand isn’t as clever as it was made out to be.

Re: Fire-Flyer File System (3FS)

#83

Post by c4wrd »

dekhn wrote: post #56
ammo1662 wrote: post #13

For those who are interested, the design was originally published here: (Chinese) https://www.high-flyer.cn/blog/3fs/ This file system has been developed and utilized by them for several years . Compared to the traditional file systems, it is more focused on model training that contains a lot of random reads. Read cache and prefetching are useless in this case. Therefore, they designed the file system without those f…

How critical is random reading of training data when assembling batches? Put another way: in my experience, supporting fast random reads is a challenging problem, while supporting high sequential reads is fairly straightforward. When is random access to a training set absolutely necessary for training a model?

Imagine you're studying for a test where you are given an image and need to answer the correct class. To prepare, you're given a deck of flashcards with an image on the front and the class on the back.

(Random) You shuffle the deck every time you go through it. You're forced to learn the images and their classifications without relying on any specific sequence, as the data has no signal from sequence order.

(Fixed order) Every time you go through the deck, the images appear in the exact same order. Over time you may start to unconsciously memorize the sequence of flashcards, rather than the actual classification of each image.

When it comes to actually training a model, if the batches are sampled sequentially from a dataset, it risks learning from correlations caused by the sequencing of the data, resulting in poor generalization. In contrast, when you sample the batches randomly, the model is biased and encouraged to learn features from the data itself rather than from any signals that arise from artifacts of the ordering.

Re: Fire-Flyer File System (3FS)

#85

Post by arjvik »

dekhn wrote: post #56
ammo1662 wrote: post #13

For those who are interested, the design was originally published here: (Chinese) https://www.high-flyer.cn/blog/3fs/ This file system has been developed and utilized by them for several years . Compared to the traditional file systems, it is more focused on model training that contains a lot of random reads. Read cache and prefetching are useless in this case. Therefore, they designed the file system without those f…

How critical is random reading of training data when assembling batches? Put another way: in my experience, supporting fast random reads is a challenging problem, while supporting high sequential reads is fairly straightforward. When is random access to a training set absolutely necessary for training a model?

On an SSD, random and sequential reads have nearly the exact same performance. Even on large arrays of spinning rust this is essentially true.

Re: Fire-Flyer File System (3FS)

#86

Post by ssivark »

hintymad wrote: post #81
htrp wrote: post #75

Earlier quoted context omitted.

> team of fewer than 10 the highflyer team are pretty well resourced.... think they have more than 10 people

Thanks! Updated to 60 per their author list in their paper.

Why do you assume everyone in the company (including folks working on infra) are authors on the paper? That’s possible, of course, but isn’t it unlikely?

Re: Fire-Flyer File System (3FS)

#87

Post by hintymad »

ssivark wrote: post #86
hintymad wrote: post #81

Earlier quoted context omitted.

Thanks! Updated to 60 per their author list in their paper.

Why do you assume everyone in the company (including folks working on infra) are authors on the paper? That’s possible, of course, but isn’t it unlikely?

That's the only information I have. That said, High-Flyer had about 160 people, total, in 2021. Given that F3 was in production in 2019, 60 people is a generous estimation.

Re: Fire-Flyer File System (3FS)

#88

Post by bezmiran »

arjvik wrote: post #85
dekhn wrote: post #56

Earlier quoted context omitted.

How critical is random reading of training data when assembling batches? Put another way: in my experience, supporting fast random reads is a challenging problem, while supporting high sequential reads is fairly straightforward. When is random access to a training set absolutely necessary for training a model?

On an SSD, random and sequential reads have nearly the exact same performance. Even on large arrays of spinning rust this is essentially true.

By what metric? I think this is close to true for identical blocksizes, but most benchmarks test sequential transfers with large 1M blocks and random ones with small 4K blocks. In this case, the speed of the fastest NVME drives is more than double for sequential transfers than it is for random ones.

I don't like comparing the two, they're completely different workloads and it's better IMO to look at the IOPS for random transfers, which is where newer, faster SSDs truly excel, and where most people "notice" the performance.

Re: Fire-Flyer File System (3FS)

#89

Post by dekhn »

c4wrd wrote: post #83
dekhn wrote: post #56

Earlier quoted context omitted.

How critical is random reading of training data when assembling batches? Put another way: in my experience, supporting fast random reads is a challenging problem, while supporting high sequential reads is fairly straightforward. When is random access to a training set absolutely necessary for training a model?

Imagine you're studying for a test where you are given an image and need to answer the correct class. To prepare, you're given a deck of flashcards with an image on the front and the class on the back. (Random) You shuffle the deck every time you go through it. You're forced to learn the images and their classifications without relying on any specific sequence, as the data has no signal from sequence order. (Fixed or…

Why, then are so many successful models trained on multiple passes through sequential data? Note, I'm not naive in this field, but as an infra person, random reads make my life very difficult, and if they're not necessary, I'd rather not deal with them by having to switch to an approach that can't use readahead and other strategies for high throughput io.

Re: Fire-Flyer File System (3FS)

#90

Post by dekhn »

arjvik wrote: post #85
dekhn wrote: post #56

Earlier quoted context omitted.

How critical is random reading of training data when assembling batches? Put another way: in my experience, supporting fast random reads is a challenging problem, while supporting high sequential reads is fairly straightforward. When is random access to a training set absolutely necessary for training a model?

On an SSD, random and sequential reads have nearly the exact same performance. Even on large arrays of spinning rust this is essentially true.

This hasn't been my experience; I see much higher sequential read results compared to random reads on a wide range of storage from low-end home PC SSDs to high end NVME flash storage in large servers.

It's certainly not true on actual hard drives, and never has been. A seek is around 10ms.

Post reply on HN