Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TernFS – An exabyte scale, multi-region distributed filesystem

xtxmarkets.com

Post reply on HN
Print view View original

31–40 of 111 posts

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#31

Post by bitonico »

eps wrote: post #29

That was a good read. Compliments to the chefs. It'd be helpful to have a couple of usage examples that illustrate common operations, like creating a file or finding and reading one, right after the high-level overview section. Just to get an idea what happens at the service level in these cases.

Yes, that would be very useful, we just didn't get to it and we didn't want perfect to be the enemy of good, since otherwise we would have never open sourced :).

But if we have the time it would definitely be a good addition to the docs.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#33

Post by Beijinger »

rickette wrote: post #26

Over 500PB of data, wow. Would love to know how and why "statistical models that produce price forecasts for over 50,000 financial instruments worldwide" require that much storage.

Me too. Is is really hard for me to understand, what XTX is actually doing. Trading? VC? AI/ML?

Have you seen their portfolio?

PS: Company seems legit. Impressive growth. But I still don't understand what they are doing. Provide "electronic liquidity". Well....

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#34

Post by d12bb »

> The firm started out with a couple of desktops and an NFS server, and 10 years later ended up with tens of thousands of high-end GPUs, hundreds of thousands of CPUs, and hundreds of petabytes of storage.

So much resources for producing nothing of real value. What a waste.

Great project though, appreciate open sourcing it.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#35

Post by EugeneG »

d12bb wrote: post #34

> The firm started out with a couple of desktops and an NFS server, and 10 years later ended up with tens of thousands of high-end GPUs, hundreds of thousands of CPUs, and hundreds of petabytes of storage. So much resources for producing nothing of real value. What a waste. Great project though, appreciate open sourcing it.

In theory what they are doing of value, is that at any time you can go to an exchange and say "I want to buy x" or "I want to sell y" and someone will buy it from you our sell it from you... at a price that's likely to be the accurate price.

At the extreme if nobody was providing this service, investors (e.g. pension funds), wouldn't be confident that they can buy/sell their assets as needed in size and at the right price... and because of that, in aggregate stocks would be worth less, and companies wouldn't be able to raise as much capital.

The theoretical model is: - You want to have efficient primary markets that allow companies to raise a lot of assets at the best possible prices - To enable efficient primary markets, investors want efficient secondary markets (so they don't need to buy and hold forever, but feel they can sell) - To enable efficient secondary markets, you need many folks that are in the business of XTX ... it just so happens that XTX is quite good at it, and so they do a lot of this work.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#36

Post by doctorpangloss »

Ha ha, I forecast, SPY goes up, and I’ve already made more money than XTX or any of its clients…

Look I like technology as much as anyone. Improbable spent $500 million on product development, and its most popular product is its grpc-web client. It didn't release any of its exotic technology. You could also go and spend that money on making $500m of games without any exotic technology, and also make it open source.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#37

Post by _jsmh »

rostayob wrote: post #7
ttfvjktesd wrote: post #4

How does TernFS compare to CephFS and why not CephFS, since it is also tested for the multiple Petabyte range?

(Disclaimer: I'm one of the authors of TernFS and while we evaluated Ceph I am not intimately familiar with it) Main factors: * Ceph stores both metadata and file contents using the same object store (RADOS). TernFS uses a specialized database for metadata which takes advantage of various properties of our datasets (immutable files, few moves between directories, etc.). * While Ceph is capable of storing PBs, we curr…

Any compression at the filesystem level?

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#38

Post by cmdrk »

ttfvjktesd wrote: post #4

How does TernFS compare to CephFS and why not CephFS, since it is also tested for the multiple Petabyte range?

CephFS implements a (fully?) POSIX filesystem while it seems that TernFS makes tradeoffs by losing permissions and mutability for further scale.

Their docs mention they have a custom kernel module, which I suppose is (today) shipped out of tree. Ceph is in-tree and also has a FUSE implementation.

The docs mention that TernFS also has its own S3 gateway, while RADOSGW is fully separate from CephFS.

Re: TernFS – An exabyte scale, multi-region distributed filesystem

#40

Post by rostayob »

_jsmh wrote: post #37
rostayob wrote: post #7

Earlier quoted context omitted.

(Disclaimer: I'm one of the authors of TernFS and while we evaluated Ceph I am not intimately familiar with it) Main factors: * Ceph stores both metadata and file contents using the same object store (RADOS). TernFS uses a specialized database for metadata which takes advantage of various properties of our datasets (immutable files, few moves between directories, etc.). * While Ceph is capable of storing PBs, we curr…

Any compression at the filesystem level?

No, we have our custom compressor as well but it's outside the filesystem.
Post reply on HN