I think the difference between deepseek and OpenAI/Anthropic is one of the difference between practitioners and academics. Ofcourse there is world class talent at OpenAI. But there are also alot of "I went to Harvard and want to work in AI", and those types of people just simply dont have the technical exposure to even think of building something like this.
Fire-Flyer File System (3FS)
51–60 of 106 posts
Re: Fire-Flyer File System (3FS)
#52Re: Fire-Flyer File System (3FS)
#53Earlier quoted context omitted.
Why is FUSE that much slower than providing your own read/write functions? I get that it has to go through the kernel, but the operations are on entire blocks and network should be the bottleneck by far (and disk/main memory should be a bottleneck if the data is local).
You have to bounce through the kernel back out to use space. The number of syscalls is quite high. In many cases this is mitigated somewhat by the page cache making reads cheaper, but that’s explicitly an anti design here. I believe there’s work to minimize this using io_uring so that you can talk to the fuse driver without the kernel being in the middle, but that work isn’t ready last time I checked. For what it’s w…
Fuse over io_uring has just been merged: https://www.phoronix.com/news/Linux-6.14-FUSE
So has uncached buffered IO: https://www.phoronix.com/news/Uncached-Buffered-IO-Linux-6.1...
6.14 is an exciting kernel!
Re: Fire-Flyer File System (3FS)
#54Earlier quoted context omitted.
my theory is that their own DeepSeek writes the code for them, so they are highly productive.
That would be terrifying in itself if true because for this type of work you really need the best of the best. But I doubt this is the case here. LLMs as we know them today are not quite yet there for this type of work.
Do you, though? Need the best of the best?
Re: Fire-Flyer File System (3FS)
#55I think the difference between deepseek and OpenAI/Anthropic is one of the difference between practitioners and academics. Ofcourse there is world class talent at OpenAI. But there are also alot of "I went to Harvard and want to work in AI", and those types of people just simply dont have the technical exposure to even think of building something like this.
Someone should write a blog post about the prestige/effectiveness negative feedback loop. This is also the Achilles heel of top tier SV VCs including YC.
Re: Fire-Flyer File System (3FS)
#56For those who are interested, the design was originally published here: (Chinese) https://www.high-flyer.cn/blog/3fs/ This file system has been developed and utilized by them for several years . Compared to the traditional file systems, it is more focused on model training that contains a lot of random reads. Read cache and prefetching are useless in this case. Therefore, they designed the file system without those f…
How critical is random reading of training data when assembling batches?
Put another way: in my experience, supporting fast random reads is a challenging problem, while supporting high sequential reads is fairly straightforward. When is random access to a training set absolutely necessary for training a model?
Re: Fire-Flyer File System (3FS)
#57I think the difference between deepseek and OpenAI/Anthropic is one of the difference between practitioners and academics. Ofcourse there is world class talent at OpenAI. But there are also alot of "I went to Harvard and want to work in AI", and those types of people just simply dont have the technical exposure to even think of building something like this.
I would say most if not every large company in China has their own AI infra stack, partially because tech talent is relatively more abundant and partially some of the tech leads have been exposed to western tech via open source and work experience so they have a good success rate (which makes it a more common practice). Anecdotally, specifically Google, FB ex-employees from oversea offices, MSFT and Intel ex-employees from their China offices could be the key elements for this trend in the past two decades (Google left China around 2010).
The infra work is usually technically tedious so I think it may become some lost art in the west just like those manufacturing jobs.
Re: Fire-Flyer File System (3FS)
#58I think the difference between deepseek and OpenAI/Anthropic is one of the difference between practitioners and academics. Ofcourse there is world class talent at OpenAI. But there are also alot of "I went to Harvard and want to work in AI", and those types of people just simply dont have the technical exposure to even think of building something like this.
Someone should write a blog post about the prestige/effectiveness negative feedback loop. This is also the Achilles heel of top tier SV VCs including YC.
[deleted]
Re: Fire-Flyer File System (3FS)
#59Re: Fire-Flyer File System (3FS)
#60Earlier quoted context omitted.
That would be terrifying in itself if true because for this type of work you really need the best of the best. But I doubt this is the case here. LLMs as we know them today are not quite yet there for this type of work.
Do you, though? Need the best of the best?
As someone who did some simulation focused engineering grad school stuff; there is a tendency for some of the best to go become quants. Does the field need it? I don’t know. But for whatever reason the draw of the “print money using math tricks” seems to attract some hardcore folks, haha.
It is really frustrating to see good engineers go to play trading games. We should study how exactly it is China managed to unlock this capacity.