Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TernFS – An exabyte scale, multi-region distributed filesystem

xtxmarkets.com

Post reply on HN
Print view View original

11–20 of 111 posts

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#11

Post by mgrandl »

rostayob wrote: post #7
ttfvjktesd wrote: post #4

How does TernFS compare to CephFS and why not CephFS, since it is also tested for the multiple Petabyte range?

(Disclaimer: I'm one of the authors of TernFS and while we evaluated Ceph I am not intimately familiar with it) Main factors: * Ceph stores both metadata and file contents using the same object store (RADOS). TernFS uses a specialized database for metadata which takes advantage of various properties of our datasets (immutable files, few moves between directories, etc.). * While Ceph is capable of storing PBs, we curr…

There are definitely insanely large Ceph deployments. I have seen hundreds of PBs in production myself. Also your usecase sounds like something that should be quite manageable for Ceph to handle due to limited metadata activity, which tends to be the main painpoint with CephFS.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#12

Post by jandrewrogers »

mrbluecoat wrote: post #2

Cool project and kudos for open sourcing it. Noteworthy limitation: > TernFS should not be used for tiny files — our median file size is 2MB.

I have worked on exabyte-scale storage engines. There is a good engineering reason for this type of limitation.

If you had 1 KiB average file size then you have quadrillions of metadata objects to quickly search and manage with fine-granularity. The kinds of operations and coordination you need to do with metadata is difficult to achieve reliably when the metadata structure itself is many PB in size. There are interesting edge cases that show up when you have to do deep paging of this metadata off of storage. Making this not slow requires unorthodox and unusual design choices that introduce a lot of complexity. Almost none of the metadata fits in memory, including many parts of conventional architectures we assume will always fit in memory.

A mere trillion objects is right around the limit of where the allocators, metadata, etc can be made to scale with heroic efforts before conventional architectures break down and things start to become deeply weird on the software design side. Storage engines need to be reliable, so avoiding that design frontier makes a lot of sense if you can avoid it.

It is possible to break this barrier but it introduces myriad interesting design and computer science problems for which there is little literature.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#14

Post by kachapopopow »

mgrandl wrote: post #11
rostayob wrote: post #7

Earlier quoted context omitted.

(Disclaimer: I'm one of the authors of TernFS and while we evaluated Ceph I am not intimately familiar with it) Main factors: * Ceph stores both metadata and file contents using the same object store (RADOS). TernFS uses a specialized database for metadata which takes advantage of various properties of our datasets (immutable files, few moves between directories, etc.). * While Ceph is capable of storing PBs, we curr…

There are definitely insanely large Ceph deployments. I have seen hundreds of PBs in production myself. Also your usecase sounds like something that should be quite manageable for Ceph to handle due to limited metadata activity, which tends to be the main painpoint with CephFS.

Ceph is more of: here's a raw block of data, do whatever the hell you want with it, not really good for immutable data.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#15

Post by rostayob »

mgrandl wrote: post #11
rostayob wrote: post #7

Earlier quoted context omitted.

(Disclaimer: I'm one of the authors of TernFS and while we evaluated Ceph I am not intimately familiar with it) Main factors: * Ceph stores both metadata and file contents using the same object store (RADOS). TernFS uses a specialized database for metadata which takes advantage of various properties of our datasets (immutable files, few moves between directories, etc.). * While Ceph is capable of storing PBs, we curr…

There are definitely insanely large Ceph deployments. I have seen hundreds of PBs in production myself. Also your usecase sounds like something that should be quite manageable for Ceph to handle due to limited metadata activity, which tends to be the main painpoint with CephFS.

I'm not fully up to date since we looked into this a few years ago, at the time the CERN deployments of Ceph were cited as particularly large examples and they topped out at ~30PB.

Also note that when I say "single deployment" I mean that the full storage capacity is not subdivided in any way (i.e. there are no "zones" or "realms" or similar concepts). We wanted this to be the case after experiencing situations where we had significant overhead due to having to rebalance different storage buckets (albeit with a different piece of software, not Ceph).

If there are EB-scale Ceph deployments I'd love to hear more about them.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#18

Post by mgrandl »

kachapopopow wrote: post #14
mgrandl wrote: post #11

Earlier quoted context omitted.

There are definitely insanely large Ceph deployments. I have seen hundreds of PBs in production myself. Also your usecase sounds like something that should be quite manageable for Ceph to handle due to limited metadata activity, which tends to be the main painpoint with CephFS.

Ceph is more of: here's a raw block of data, do whatever the hell you want with it, not really good for immutable data.

Well sure you would have to enforce immutability at the client side.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#19

Post by mgrandl »

rostayob wrote: post #15
mgrandl wrote: post #11

Earlier quoted context omitted.

There are definitely insanely large Ceph deployments. I have seen hundreds of PBs in production myself. Also your usecase sounds like something that should be quite manageable for Ceph to handle due to limited metadata activity, which tends to be the main painpoint with CephFS.

I'm not fully up to date since we looked into this a few years ago, at the time the CERN deployments of Ceph were cited as particularly large examples and they topped out at ~30PB. Also note that when I say "single deployment" I mean that the full storage capacity is not subdivided in any way (i.e. there are no "zones" or "realms" or similar concepts). We wanted this to be the case after experiencing situations where…

There are much larger Ceph clusters, but they are enterprise owned and not really publicly talked about. Sadly I can’t share what I personally worked on.
Post reply on HN