Live data from Hacker News

TernFS – An exabyte scale, multi-region distributed filesystem

xtxmarkets.com

61–70 of 111 posts

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#61

Earlier quoted context omitted.

I have worked on exabyte-scale storage engines. There is a good engineering reason for this type of limitation. If you had 1 KiB average file size then you have quadrillions of metadata objects to quickly search and manage with fine-granularity. The kinds of operations and coordination you need to do with metadata is difficult to achieve reliably when the metadata structure itself is many PB in size. There are intere…

This sounds like a fascinating niche piece of technical expertise I would love to hear more about. What are the biggest challenges in scaling metadata from a trillion to a quadrillion objects?

you really notice metadata performance (try a git checkout on EFS on AWS. loads of small files takes fucking ages) However EFS is actually pretty fast. you can get decent throughput if you're writing to just one file. but if you're trying to open 1000 1meg files to read from vs 1 1G file, it'll be much slower (unless they'd dramatically improved performance recently)

Trying to have a fast globally consistent database for quadrillion items in the _same_ name space is super hard. You need to chose a tradeoff between speed, partition resistance and consistency.

You're much better off sharding into discreet logical units. Its very rare that you need a global namespace for a filesystem. For VFX where we used lustre a lot, the large namespace was a nice to have, it was more about getting a raid-0 across file servers (well object stores) to get performance.

For filesystems specifically, if you're using folders, then you don't actually need to guarantee much outside of a folder. So long as filenames are unique to that folder, you can get away with a lot of shit you can't do in a normal database. you also don't need directories to be on the same filesystem (well in linux at least) so you can also shard by using directories as a key.

The directory-key-filesystem approach is actually hilariously simple, fast scalable and reliable. If a single server/Fs goes down it only takes out that area. On the downside it does mean that you can overwhelm/get hot spots.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#62

Hudson River Trading's distributed file system for comparison: https://www.hudsonrivertrading.com/hrtbeat/distributed-files...

Cool. Also https://github.com/deepseek-ai/3FS by DeepSeek which came out of High-Flyer a Chinese HFT firm.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#63
post #42

Over 500PB of data, wow. Would love to know how and why "statistical models that produce price forecasts for over 50,000 financial instruments worldwide" require that much storage.

If you keep all order book changes for a large number of financial instruments volume adds up quickly.

Would that kind of data not compress like crazy? Or would they need to keep all that data hot and fast.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#64

Cool project and kudos for open sourcing it. Noteworthy limitation: > TernFS should not be used for tiny files — our median file size is 2MB.

I have worked on exabyte-scale storage engines. There is a good engineering reason for this type of limitation. If you had 1 KiB average file size then you have quadrillions of metadata objects to quickly search and manage with fine-granularity. The kinds of operations and coordination you need to do with metadata is difficult to achieve reliably when the metadata structure itself is many PB in size. There are intere…

Small files suck on normal filesystems too. There's reasons to have them, but if the stars align and you can go from M directories of N directories of O files to M directories of N files with O sub-files, it can make a lot of operations way faster, but probably not updates to individual sub-files (but if you're updating all the files and can update all of the M/N.db at once, then that probably is faster)

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#65
post #34

> The firm started out with a couple of desktops and an NFS server, and 10 years later ended up with tens of thousands of high-end GPUs, hundreds of thousands of CPUs, and hundreds of petabytes of storage. So much resources for producing nothing of real value. What a waste. Great project though, appreciate open sourcing it.

higher competition increases market efficiency - this is the real value

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#66

Earlier quoted context omitted.

This sounds like a fascinating niche piece of technical expertise I would love to hear more about. What are the biggest challenges in scaling metadata from a trillion to a quadrillion objects?

It is dependent on the intended workload but there are a few common design problems. Keep in mind that you can't just deal in the average case, you have to design for the worst possible cases of extremely skewed or pathologically biased distributions. A lot of the design work is proving worst case resource bounds under various scenarios and then proving the worst case behavior of designs intended to mitigate that. An…

This is quite fascinating, thank you!

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#67

Earlier quoted context omitted.

This sounds like a fascinating niche piece of technical expertise I would love to hear more about. What are the biggest challenges in scaling metadata from a trillion to a quadrillion objects?

you really notice metadata performance (try a git checkout on EFS on AWS. loads of small files takes fucking ages) However EFS is actually pretty fast. you can get decent throughput if you're writing to just one file. but if you're trying to open 1000 1meg files to read from vs 1 1G file, it'll be much slower (unless they'd dramatically improved performance recently) Trying to have a fast globally consistent database…

We are truly spoiled by all the improvements that went into local filesystems that are lacking in network filesystems. So much of our perception of "computer is fast" is really just write-caching, read-caching, read-ahead.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#68
Sounds more like an object system (immutable) with the veneer of a file system for their use cases. I sort of read the doc - sounds like data is replicated and not erasure encoded (so perhaps more expensive?).

I think many people have said this, but "file systems" get a lot easier if you don't have to worry about overwrites, appends, truncates, etc. Anyway, always interesting to see what people come up with for their use cases.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#69
post #68

Sounds more like an object system (immutable) with the veneer of a file system for their use cases. I sort of read the doc - sounds like data is replicated and not erasure encoded (so perhaps more expensive?). I think many people have said this, but "file systems" get a lot easier if you don't have to worry about overwrites, appends, truncates, etc. Anyway, always interesting to see what people come up with for their…

We do use Reed-Solomon codes, as the blog post explains.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#70
post #21

Cool project and kudos for open sourcing it. Noteworthy limitation: > TernFS should not be used for tiny files — our median file size is 2MB.

Shameless plug: https://github.com/Barre/ZeroFS I initially developed it for a usecase where I needed to store billions of tiny files, and it just requires a single s3 bucket as infrastructure.

Interesting, can I use SeaweedFS as bucket provider?
Post reply on HN