The Nimble File Format by Meta
github.com
The Nimble File Format by Meta
1–10 of 26 posts
Re: The Nimble File Format by Meta
#2[deleted]
Re: The Nimble File Format by Meta
#3Although "wide" data is touted as a optimization guideline for Nimble, how well does it fare against "normal"(?) data, i.e. with just a few to tens of columns?
Also, are there any preliminary benchmarks?
Re: The Nimble File Format by Meta
#4Although "wide" data is touted as a optimization guideline for Nimble, how well does it fare against "normal"(?) data, i.e. with just a few to tens of columns? Also, are there any preliminary benchmarks?
There are some numbers in the youtube presentation https://www.youtube.com/watch?v=bISBNVtXZ6M
It seems to be optimized towards ML where sequential scan is the access pattern. so it wouldn't be suitable for analytical workloads yet, though they are planning on working on that.
Re: The Nimble File Format by Meta
#5Where is the file format specification?
I would prefer to write a parser with zero dependencies.
Re: The Nimble File Format by Meta
#6I wonder how well it functions as a blobstore.
Re: The Nimble File Format by Meta
#7I thought the future would have been unstructured.
Re: The Nimble File Format by Meta
#8Where is the file format specification? I would prefer to write a parser with zero dependencies.
They strongly discourage you.
Unified: More than a specification, Nimble is a product. We strongly discourage developers to (re-)implement Nimble’s spec to prevent environmental fragmentation issues observed with similar projects in the past. We encourage developers to leverage the single unified Nimble library, and create high-quality bindings to other languages as needed.
Re: The Nimble File Format by Meta
#9Video link and more comments from two weeks ago: https://news.ycombinator.com/item?id=39995112
Re: The Nimble File Format by Meta
#10https://xkcd.com/927/ ?
We still use HDF (https://en.wikipedia.org/wiki/Hierarchical_Data_Format).
But I wonder, if I would choose a new file format today, what to choose? Nimble is maybe too new and there is too less experience with it (outside Meta).
Is there anywhere a good overview of all available options, and some fair comparison? Some that I found, but older:
https://www.hopsworks.ai/post/guide-to-file-formats-for-mach...
https://iopscience.iop.org/article/10.1088/1742-6596/1085/3/...