Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TernFS – An exabyte scale, multi-region distributed filesystem

xtxmarkets.com

Post reply on HN
Print view View original

51–60 of 111 posts

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#51

Post by formerly_proven »

lsecondario wrote: post #41
EugeneG wrote: post #35

Earlier quoted context omitted.

In theory what they are doing of value, is that at any time you can go to an exchange and say "I want to buy x" or "I want to sell y" and someone will buy it from you our sell it from you... at a price that's likely to be the accurate price. At the extreme if nobody was providing this service, investors (e.g. pension funds), wouldn't be confident that they can buy/sell their assets as needed in size and at the right…

> In theory > At the extreme > The theoretical model These qualifiers would seem to belie the whole argument. Surely the volume of HFT arbitrage is some large multiple of what would be necessary to provide commercial liquidity with an acceptable spread?

Does the HFT volume actually matter? Is it a real problem that the HFT volume exceeds the theoretical minimum amount of volume needed to maintain liquid markets?

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#52

Post by hintymad »

> Most of the metadata activity is contained within a single shard: > > - File creation, same-directory renames, and deletion. > - Listing directory contents. > - Getting attributes of files or directories.

I guess this is a trade-off between a file system and an object store? As in S3, ListObjects() is a heavy hitter and there can be potentially billions of objects under any prefix. Scanning only on a single instance won't be sufficient.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#54

Post by jcul »

cmdrk wrote: post #38
ttfvjktesd wrote: post #4

How does TernFS compare to CephFS and why not CephFS, since it is also tested for the multiple Petabyte range?

CephFS implements a (fully?) POSIX filesystem while it seems that TernFS makes tradeoffs by losing permissions and mutability for further scale. Their docs mention they have a custom kernel module, which I suppose is (today) shipped out of tree. Ceph is in-tree and also has a FUSE implementation. The docs mention that TernFS also has its own S3 gateway, while RADOSGW is fully separate from CephFS.

My (limited) understanding is that cephfs, RGW (S3), RBD (block device) are all different things using the same underlying RADOS storage.

You can't mount and access RGW S3 objects as cephfs or anything, they are completely separate (not counting things like goofys, s3fs etc.), even if both are on the same rados cluster.

Not sure if TernFS differs there, would be kind of nice to have the option of both kinds of access to the same data.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#56

Post by KaiserPro »

pandemic_region wrote: post #5
mrbluecoat wrote: post #2

Cool project and kudos for open sourcing it. Noteworthy limitation: > TernFS should not be used for tiny files — our median file size is 2MB.

What happens if you put a tiny file on it then? Bad perf, possible file corruption, ... ?

Bad space efficiency and possibly exhausting your inode system (ie there is space left on the device, but you can't put any files on it)

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#57

Post by KaiserPro »

ttfvjktesd wrote: post #4

How does TernFS compare to CephFS and why not CephFS, since it is also tested for the multiple Petabyte range?

Ceph isn't that well suited for high performance. its also young and more complex than you'd want it to be (ie you get a block storage system, which you then have to put a FS layer on after.)

if you want performance, then you'll probably want lustre, or GPFS, or if you're rich a massive isilon system.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#58

Post by KaiserPro »

eigenvalue wrote: post #17

This sounds like it would be a good underpinning for a decentralized blockchain file storage system with its focus on immutability and redundancy.

if you're storing the blocks in one place, its not decentralised.

The metadata would be crucial for performance, and given that I assume you'll want a full chain of history for every file, your metadata table will get progressively bigger every time you do any kind of metadata operation.

Plus you can only have one person write metadata at one time, so you're gonna get huge top of line blocking.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#60

Post by adastra22 »

Beijinger wrote: post #33
rickette wrote: post #26

Over 500PB of data, wow. Would love to know how and why "statistical models that produce price forecasts for over 50,000 financial instruments worldwide" require that much storage.

Me too. Is is really hard for me to understand, what XTX is actually doing. Trading? VC? AI/ML? Have you seen their portfolio? PS: Company seems legit. Impressive growth. But I still don't understand what they are doing. Provide "electronic liquidity". Well....

High frequency trading.
Post reply on HN