Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TernFS – An exabyte scale, multi-region distributed filesystem

xtxmarkets.com

Post reply on HN
Print view View original

71–80 of 111 posts

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#71

Post by charleshn »

A few questions if the authors are around!

> Is hardware agnostic and uses TCP/IP to communicate.

So no RDMA? It's very hard to make effective use of modern NVMe drives bandwidth over TCP/IP.

> A logical shard is further split into five physical instances, one leader and four followers, in a typical distributed consensus setup. The distributed consensus engine is provided by a purpose-built Raft-like implementation, which we call LogsDB

Raft-like, so not Raft, a custom algorithm? Implementing distributed consensus correctly from scratch is very hard - why not use some battle-tested implementations?

> Read/write access to the block service is provided using a simple TCP API currently implemented by a Go process. This process is hardware agnostic and uses the Go standard library to read and write blocks to a conventional local file system. We originally planned to rewrite the Go process in C++, and possibly write to block devices directly, but the idiomatic Go implementation has proven performant enough for our needs so far.

The document mentions it's designed to reach TB/s though. Which means that for an IO intensive workload, one would end up wasting a lot of drive bandwidth, and require a huge number of nodes.

Modern parallel filesystems can reach 80-90GB/s per node, using RDMA, DPDK etc.

> This is in contrast to protocols like NFS, whereby each connection is very stateful, holding resources such as open files, locks, and so on.

This is not true for NFSv3 and older, it tends to be stateless (no notion of open file).

No mention of the way this was developed and tested - does it use some formal methods, simulator, chaos engineering etc?

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#72

Post by chatmasta »

> There's a reason why every major tech company has developed its own distributed filesystem

I haven't worked at FAANG, but is this a well-known fact? I've never heard of it. Unless they're referring to things like S3? Are these large corps running literal custom filesystem implementations?

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#73

Post by foota »

charleshn wrote: post #71

A few questions if the authors are around! > Is hardware agnostic and uses TCP/IP to communicate. So no RDMA? It's very hard to make effective use of modern NVMe drives bandwidth over TCP/IP. > A logical shard is further split into five physical instances, one leader and four followers, in a typical distributed consensus setup. The distributed consensus engine is provided by a purpose-built Raft-like implementation,…

Out of curiosity, you seem knowledgeable here, is it possible to do NVME over RDMA in public cloud (e.g., on AWS)? I was recently looking into this and my conclusion was no, but I'd love to be wrong :)

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#74

Post by foota »

chatmasta wrote: post #72

> There's a reason why every major tech company has developed its own distributed filesystem I haven't worked at FAANG, but is this a well-known fact? I've never heard of it. Unless they're referring to things like S3? Are these large corps running literal custom filesystem implementations?

Not sure about everyone, but probably yes. Imo the killer feature is durability and replication. You can use single disks if you don't care about data loss, but once you need to start replicating data you need a distributed filesystem.

Tectonic is Facebooks, Google's is Colossus. I'm not sure about the others.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#75

Post by stonogo »

foota wrote: post #73
charleshn wrote: post #71

A few questions if the authors are around! > Is hardware agnostic and uses TCP/IP to communicate. So no RDMA? It's very hard to make effective use of modern NVMe drives bandwidth over TCP/IP. > A logical shard is further split into five physical instances, one leader and four followers, in a typical distributed consensus setup. The distributed consensus engine is provided by a purpose-built Raft-like implementation,…

Out of curiosity, you seem knowledgeable here, is it possible to do NVME over RDMA in public cloud (e.g., on AWS)? I was recently looking into this and my conclusion was no, but I'd love to be wrong :)

Amazon FSx for Lustre is the product. They do have information on DIY with the underlying tech: https://aws.amazon.com/blogs/hpc/scaling-a-read-intensive-lo...

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#76

Post by allset_ »

chatmasta wrote: post #72

> There's a reason why every major tech company has developed its own distributed filesystem I haven't worked at FAANG, but is this a well-known fact? I've never heard of it. Unless they're referring to things like S3? Are these large corps running literal custom filesystem implementations?

I can only comment about the one I work for, but yes. It's also discussed publicly to some degree.

https://cloud.google.com/blog/products/storage-data-transfer...

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#77

Post by StrangeDoctor »

foota wrote: post #74
chatmasta wrote: post #72

> There's a reason why every major tech company has developed its own distributed filesystem I haven't worked at FAANG, but is this a well-known fact? I've never heard of it. Unless they're referring to things like S3? Are these large corps running literal custom filesystem implementations?

Not sure about everyone, but probably yes. Imo the killer feature is durability and replication. You can use single disks if you don't care about data loss, but once you need to start replicating data you need a distributed filesystem. Tectonic is Facebooks, Google's is Colossus. I'm not sure about the others.

Deepseek has their own and they’re relatively small https://github.com/deepseek-ai/3FS

It’s specialized knowledge, hard to do “correctly” (read posix here) but obtainable and implementable by a small team if you pick your battles right. Also supporting very specific use cases helps a lot.

It’s also pretty easy to justify as the hardware and software from vanguard tech companies is outrageously expensive. I used to develop software for a blue colored distributed filesystem.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#78

Post by foota »

stonogo wrote: post #75
foota wrote: post #73

Earlier quoted context omitted.

Out of curiosity, you seem knowledgeable here, is it possible to do NVME over RDMA in public cloud (e.g., on AWS)? I was recently looking into this and my conclusion was no, but I'd love to be wrong :)

Amazon FSx for Lustre is the product. They do have information on DIY with the underlying tech: https://aws.amazon.com/blogs/hpc/scaling-a-read-intensive-lo...

Thanks for the link! I had seen this, but it wasn't clear to me either how to configure the host as an nvme-of target, nor whether it would actually bypass the host CPU. The article (admittedly now 4 years old) cites single digit GB/second, while I was really hoping for something closer to the full NVME bandwidth. Maybe that's just a reflection of the time though, drives have gotten a lot faster since then.

Edit: this is more like what I was hoping for: https://aws.amazon.com/blogs/aws/amazon-fsx-for-lustre-unloc... although I wasn't looking for a file system product. Ideally a tutorial like... "Create a couple VMs, store a file on one, do XYZ, and then read it from another with this API" was what I was hoping for, or at least some first party documentation of how to use these things together.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#79

Post by AtlasBarfed »

charleshn wrote: post #71

A few questions if the authors are around! > Is hardware agnostic and uses TCP/IP to communicate. So no RDMA? It's very hard to make effective use of modern NVMe drives bandwidth over TCP/IP. > A logical shard is further split into five physical instances, one leader and four followers, in a typical distributed consensus setup. The distributed consensus engine is provided by a purpose-built Raft-like implementation,…

Not to mention you simply want a large distributed system implemented in multiple clouds / on prems / use cases, with battle tested procedures on node failure, replacement, expansion, contraction, backup/restore, repair/verification, install guides, an error "zoo".

Not to mention a Jepsen test suite, detailed CAP tradeoff explanation, etc.

There's a reason those big DFS at the FAANGs aren't really implemented anywhere else: they NEED the original authors with a big, deeply experienced infrastructure/team in house.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#80

Post by mindslight »

Is anyone else bored of seeing the endless line of anti-human-scale distributed filesystems?

It's like the engineers building them keep trying to scratch their own itch for a better filesystem that could enable seamless cross-device usage, collaboration, etc. But the engineers only get paid if they express themselves in terms of corporate desires, and corpos aren't looking to pay them to solve those hard problems. So they solve the horizontal scaling problem for the thousandth time, but only end up creating things that requires a full time engineer (or perhaps even a whole team) to use. Hooray, another centralizing "distributed" filesystem.

Post reply on HN