Live data from Hacker News

Fire-Flyer File System (3FS)

github.com

61–70 of 106 posts

Re: Fire-Flyer File System (3FS)

#61
post #50

They sure are productive. What are we going to see tomorrow? DeepSeek OS or something?

To be fair they’ve been working on this since 2019 for HFT. So it’s not like they just whipped this up.

The amount of brainpower wasted on HFT games that will never see the light of day is kind of a bummer. Congrats to China.

Re: Fire-Flyer File System (3FS)

#62
post #55

I think the difference between deepseek and OpenAI/Anthropic is one of the difference between practitioners and academics. Ofcourse there is world class talent at OpenAI. But there are also alot of "I went to Harvard and want to work in AI", and those types of people just simply dont have the technical exposure to even think of building something like this.

Someone should write a blog post about the prestige/effectiveness negative feedback loop. This is also the Achilles heel of top tier SV VCs including YC.

Can you expand on this?

Re: Fire-Flyer File System (3FS)

#63

Earlier quoted context omitted.

enlighten us

my theory is that their own DeepSeek writes the code for them, so they are highly productive.

They wrote this in 2019, well before any useful codegen LLMs existed.

https://www.high-flyer.cn/blog/3fs/

Re: Fire-Flyer File System (3FS)

#64
post #19

Earlier quoted context omitted.

It’s not. When you are a high frequency trader and you’ve mastered RDMA, everything around you looks slow. You are thinking in terms of 20 nanoseconds intervals, while everyone around still thinks that serving a query under a millisecond is fast.

Huh? What kind of RDMA has a completion latency of 20 nanoseconds? It's more like 5 microseconds. I agree that a lot of "modern" storage stack is way too slow though, tried to find a replication-first object storage for crazy-fast random read in small number of objects last year and found none.

I was talking about, thinking in terms of 20 nanoseconds intervals, rather than completing a request in 20 nanoseconds. To get 1 microsecond wire-to-wire latency you do need to count your nanoseconds.

Why this number - this is because it’s roughly the time it takes to read 64 bytes from L3 cache. And NICs tend to be able to push data into L3 (or equivalents).

Current state of the art - look up nanoPU, from Stanford. Wire-to-wire under 100ns is not impossible, but this would normally assume pre-cooked packet, selected from a number of packets (which is not an unusual scenario in HFT).

Re: Fire-Flyer File System (3FS)

#65
post #55

I think the difference between deepseek and OpenAI/Anthropic is one of the difference between practitioners and academics. Ofcourse there is world class talent at OpenAI. But there are also alot of "I went to Harvard and want to work in AI", and those types of people just simply dont have the technical exposure to even think of building something like this.

Someone should write a blog post about the prestige/effectiveness negative feedback loop. This is also the Achilles heel of top tier SV VCs including YC.

You mean this one? https://news.ycombinator.com/item?id=9125816

Re: Fire-Flyer File System (3FS)

#68

I think the difference between deepseek and OpenAI/Anthropic is one of the difference between practitioners and academics. Ofcourse there is world class talent at OpenAI. But there are also alot of "I went to Harvard and want to work in AI", and those types of people just simply dont have the technical exposure to even think of building something like this.

Makes me wonder where is the best place to learn how to put together and operate something like this then? Certainly there should be resources out there somewhere to teach yourself?

Re: Fire-Flyer File System (3FS)

#69
A distributed file system is honed as one of the trickiest software to write, and we are usually advised not to write a file system from scratch (even on top of FUSE), let alone a highly optimized one. When a silicon value company is having the 100th meeting to align god-knows-what, a team of fewer than 60 already came up with a production-grade highly efficient parallel file system.

Have we in the valley companies lost touch?

Re: Fire-Flyer File System (3FS)

#70

Earlier quoted context omitted.

That would be terrifying in itself if true because for this type of work you really need the best of the best. But I doubt this is the case here. LLMs as we know them today are not quite yet there for this type of work.

Do you, though? Need the best of the best?

I think so yes. There's very few engineers that can pull out this type of work IME. In a pool of ~30M SEs around the globe I'd say there's no more than ~30K of such engineers. This is 0.001% and it's a very optimistic number I'd say.

Why do you think this would be controversial? This isn't an every day work.

Post reply on HN