Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire-Flyer File System (3FS)

github.com

Post reply on HN
Print view View original

91–100 of 106 posts

Re: Fire-Flyer File System (3FS)

#91

Post by InkCanon »

bugglebeetle wrote: post #73
mustpax wrote: post #55

Earlier quoted context omitted.

Someone should write a blog post about the prestige/effectiveness negative feedback loop. This is also the Achilles heel of top tier SV VCs including YC.

The problem isn’t the prestige it’s that prestigious institutions in America don’t produce high-quality talent. They’re instead mostly corrupt credentialing mills for the rich and well-connected. From what I understand, DeepSeek also only hires from the best universities in China, but “best” actually means something relative to how difficult entrance to those organizations is to achieve and their coursework.

I read this too but there was no source on this. The founder Liang Wenfeng himself comes from Zhejiang university. Its admissions rate is 20%, which is much higher than traditional US "elite" schools. Wenfeng has said this about hiring though:

"If you are pursuing short-term goals, it is right to find people with ready experience. But if you look at the long-term, experience is not that important. Basic skills, creativity, and passion are much more important.”

Re: Fire-Flyer File System (3FS)

#92

Post by genewitch »

bee_rider wrote: post #82
htrp wrote: post #77

Earlier quoted context omitted.

Government effectively banned unproductive tech (adtech, fintech etc) and told ppl to go do stuff like robotics and AI

I’m beginning to suspect this invisible hand isn’t as clever as it was made out to be.

|O|I|L|!|o o|W|A|R|!|

Re: Fire-Flyer File System (3FS)

#93

Post by Terretta »

dekhn wrote: post #89
c4wrd wrote: post #83

Earlier quoted context omitted.

Imagine you're studying for a test where you are given an image and need to answer the correct class. To prepare, you're given a deck of flashcards with an image on the front and the class on the back. (Random) You shuffle the deck every time you go through it. You're forced to learn the images and their classifications without relying on any specific sequence, as the data has no signal from sequence order. (Fixed or…

Why, then are so many successful models trained on multiple passes through sequential data? Note, I'm not naive in this field, but as an infra person, random reads make my life very difficult, and if they're not necessary, I'd rather not deal with them by having to switch to an approach that can't use readahead and other strategies for high throughput io.

Why did they take time to invent an entire FS?

Re: Fire-Flyer File System (3FS)

#94

Post by sureglymop »

codingwagie wrote: post #51

I think the difference between deepseek and OpenAI/Anthropic is one of the difference between practitioners and academics. Ofcourse there is world class talent at OpenAI. But there are also alot of "I went to Harvard and want to work in AI", and those types of people just simply dont have the technical exposure to even think of building something like this.

I think it would be a bit irrational to claim that so broadly. I've met some incredibly talented people in academia and I've also met people that made me question how they even pass.

My hypothesis is that there is not such a big difference at all. All three of the companies you mentioned are world class competitors in this. DeepSeek were the last to have a "hit" but that isn't an indication that they'll be the next of the three (or other yet unknown entities) to have the next hit. We try to predict what happens next now but perhaps we should rather focus on who or what we want to succeed. For me it's quite clear: it should be open source or I'm long term not that interested.

Re: Fire-Flyer File System (3FS)

#95

Post by rfoo »

dekhn wrote: post #89
c4wrd wrote: post #83

Earlier quoted context omitted.

Imagine you're studying for a test where you are given an image and need to answer the correct class. To prepare, you're given a deck of flashcards with an image on the front and the class on the back. (Random) You shuffle the deck every time you go through it. You're forced to learn the images and their classifications without relying on any specific sequence, as the data has no signal from sequence order. (Fixed or…

Why, then are so many successful models trained on multiple passes through sequential data? Note, I'm not naive in this field, but as an infra person, random reads make my life very difficult, and if they're not necessary, I'd rather not deal with them by having to switch to an approach that can't use readahead and other strategies for high throughput io.

They did this back in their trading firm days, and...

Imagine that you have a sequence of numbers. You want to randomly select a window of, say, 1024 consecutive numbers, a sequence, as input to your model. Now, say, you have n items in this sequence, you want to sample n/c (c is a constant and The key is, we have overlap in data we want to read. If we brute force fixed shuffle and expand, we need to save 1024/c times more than original data.

This isn't useful for LLMs, but hey, wonder how it started?

Re: Fire-Flyer File System (3FS)

#96

Post by tway223 »

smallmancontrov wrote: post #72
tway223 wrote: post #57

Earlier quoted context omitted.

I would say most if not every large company in China has their own AI infra stack, partially because tech talent is relatively more abundant and partially some of the tech leads have been exposed to western tech via open source and work experience so they have a good success rate (which makes it a more common practice). Anecdotally, specifically Google, FB ex-employees from oversea offices, MSFT and Intel ex-employee…

As opposed to the US, where every large company has its own AI infra stack, often extending down to the silicon and up to large open source projects? What's going on here, why are people forgetting what's around them? Does familiarity breed contempt? Are attention spans so shot that failure to participate in this week's news cycle is enough for "out of sight, out of mind"? Or is HN full of Chinese bots now?

That was to answer the previous question. Also the point is why Chinese companies can produce infra work in a cheap and fast way. With regard to US companies, I don’t see that is possible with MSFT, AMZN, AAPL, and likely GOOG as well. (Don’t get me wrong, they all have solid infra, probably except Apple)

Re: Fire-Flyer File System (3FS)

#97

Post by dekhn »

rfoo wrote: post #95
dekhn wrote: post #89

Earlier quoted context omitted.

Why, then are so many successful models trained on multiple passes through sequential data? Note, I'm not naive in this field, but as an infra person, random reads make my life very difficult, and if they're not necessary, I'd rather not deal with them by having to switch to an approach that can't use readahead and other strategies for high throughput io.

They did this back in their trading firm days, and... Imagine that you have a sequence of numbers. You want to randomly select a window of, say, 1024 consecutive numbers, a sequence, as input to your model. Now, say, you have n items in this sequence, you want to sample n/c (c is a constant and The key is, we have overlap in data we want to read. If we brute force fixed shuffle and expand, we need to save 1024/c time…

I guess I'm much more of the "materialize the shuffle asychronously from the training loop" kind of person. I agree, the materialization storage cost is very high, but that's normally been a cost I've been willing to accept.

As an ML infra guy I have had to debug a lot of failing jobs over the years, and randomizing datapipes are one of the hardest to debug. Sometimes there will be a "record-of-death" that randomly gets shuffled into a batch, but only causes problems when it is (extremely rarely) coupled with a few other records.

I guess I'll just have to update my priors and accept that inline synchronous randomization with random reads is a useful-enough access pattern in HPC that it should be optimized for. Certainly a lot more work and complexity, hence my question of just how necessary it is.

Re: Fire-Flyer File System (3FS)

#98

Post by dekhn »

Terretta wrote: post #93
dekhn wrote: post #89

Earlier quoted context omitted.

Why, then are so many successful models trained on multiple passes through sequential data? Note, I'm not naive in this field, but as an infra person, random reads make my life very difficult, and if they're not necessary, I'd rather not deal with them by having to switch to an approach that can't use readahead and other strategies for high throughput io.

Why did they take time to invent an entire FS?

The engineers at DeepSeek seem extremely smart and well-funded, so my guess is they looked at the alternatives and concluded none of them would allow them to make their models work well enough.

Re: Fire-Flyer File System (3FS)

#99

Post by rfoo »

dekhn wrote: post #97
rfoo wrote: post #95

Earlier quoted context omitted.

They did this back in their trading firm days, and... Imagine that you have a sequence of numbers. You want to randomly select a window of, say, 1024 consecutive numbers, a sequence, as input to your model. Now, say, you have n items in this sequence, you want to sample n/c (c is a constant and The key is, we have overlap in data we want to read. If we brute force fixed shuffle and expand, we need to save 1024/c time…

I guess I'm much more of the "materialize the shuffle asychronously from the training loop" kind of person. I agree, the materialization storage cost is very high, but that's normally been a cost I've been willing to accept. As an ML infra guy I have had to debug a lot of failing jobs over the years, and randomizing datapipes are one of the hardest to debug. Sometimes there will be a "record-of-death" that randomly g…

Yeah, I don't want to do this either. This is a super special case, after exploring alternatives with our researchers it's unfortunately needed. As for record-of-death, we made sure that we do serialize all rng state and have our data pipeline perfectly reproducible even when starting from checkpoint.

Building a system for serving read-only data at NVMe SSD speed (as in IOPS) took surprisingly few effort, and is mostly enough for training data. Kudos to DeepSeek who decided to spend extra effort to build a full PFS and share it.

Re: Fire-Flyer File System (3FS)

#100

Post by ycui1986 »

InkCanon wrote: post #91
bugglebeetle wrote: post #73

Earlier quoted context omitted.

The problem isn’t the prestige it’s that prestigious institutions in America don’t produce high-quality talent. They’re instead mostly corrupt credentialing mills for the rich and well-connected. From what I understand, DeepSeek also only hires from the best universities in China, but “best” actually means something relative to how difficult entrance to those organizations is to achieve and their coursework.

I read this too but there was no source on this. The founder Liang Wenfeng himself comes from Zhejiang university. Its admissions rate is 20%, which is much higher than traditional US "elite" schools. Wenfeng has said this about hiring though: "If you are pursuing short-term goals, it is right to find people with ready experience. But if you look at the long-term, experience is not that important. Basic skills, creat…

The Chinese college application works way differently from American ones. The admission rate is meaningless. Zhejinag University is state assigned 985 university (there are in total 9 of them). Believe me any students in elite high schools in China will be very happy if they can be accepted into Zhejiang University. Most of them unforunately don't have the score to even think of applying. It technically is not applying. Students take the once a year exam, if they don't score higher than top 500 in their province, don't even think about trying to apply Zhejing Univ.
Post reply on HN