Interesting if there are benefits over MergeTree (ClickHouse's data format).
The Nimble File Format by Meta
11–20 of 26 posts
Re: The Nimble File Format by Meta
#12[deleted]
Re: The Nimble File Format by Meta
#13Where is the file format specification? I would prefer to write a parser with zero dependencies.
They strongly discourage you. Unified: More than a specification, Nimble is a product. We strongly discourage developers to (re-)implement Nimble’s spec to prevent environmental fragmentation issues observed with similar projects in the past. We encourage developers to leverage the single unified Nimble library, and create high-quality bindings to other languages as needed.
Sigh. I get more than enough of that in the blockchain world.
Call me a greybeard; I want multiple implementations and a spec.
Re: The Nimble File Format by Meta
#14https://xkcd.com/927/ ? We still use HDF ( https://en.wikipedia.org/wiki/Hierarchical_Data_Format ). But I wonder, if I would choose a new file format today, what to choose? Nimble is maybe too new and there is too less experience with it (outside Meta). Is there anywhere a good overview of all available options, and some fair comparison? Some that I found, but older: https://www.hopsworks.ai/post/guide-to-file-forma…
[deleted]
Re: The Nimble File Format by Meta
#15What are the differences versus Arrow/Parquet?
Re: The Nimble File Format by Meta
#16What are the differences versus Arrow/Parquet?
Yes, I'd be curious to know how much better it is than them - from my limited understanding, they also share many of the advantages that Nimble boasts of, thus I can appreciate they'd both be better than legacy formats but it's not clear how close these two are.
Re: The Nimble File Format by Meta
#17Earlier quoted context omitted.
They strongly discourage you. Unified: More than a specification, Nimble is a product. We strongly discourage developers to (re-)implement Nimble’s spec to prevent environmental fragmentation issues observed with similar projects in the past. We encourage developers to leverage the single unified Nimble library, and create high-quality bindings to other languages as needed.
Sigh. I get more than enough of that in the blockchain world. Call me a greybeard; I want multiple implementations and a spec.
A spec. There is no substitute for it. Code isn't a spec. A spec is documentation that describes a format and/or operation. The absence of it is laziness and/or arrogance.
Re: The Nimble File Format by Meta
#18Earlier quoted context omitted.
Sigh. I get more than enough of that in the blockchain world. Call me a greybeard; I want multiple implementations and a spec.
A spec. There is no substitute for it. Code isn't a spec. A spec is documentation that describes a format and/or operation. The absence of it is laziness and/or arrogance.
Some dudes worked hard to create this thing and open source it, and here you are calling them lazy and/or arrogant.
On the other hand - it's an open source project, I hope someone can contribute to it and create a PR with the spec.
Re: The Nimble File Format by Meta
#19Where is the file format specification? I would prefer to write a parser with zero dependencies.
They strongly discourage you. Unified: More than a specification, Nimble is a product. We strongly discourage developers to (re-)implement Nimble’s spec to prevent environmental fragmentation issues observed with similar projects in the past. We encourage developers to leverage the single unified Nimble library, and create high-quality bindings to other languages as needed.
Why not instead make a test suite of inputs and expected pass/fail for validation of implementations.
It's not more than a specification if a single implementation is to be used--then it's a 'spec' defined by the implementation because any idiosyncrasies of that implementation become defacto specification.
Re: The Nimble File Format by Meta
#20Earlier quoted context omitted.
A spec. There is no substitute for it. Code isn't a spec. A spec is documentation that describes a format and/or operation. The absence of it is laziness and/or arrogance.
Some dudes worked hard to create this thing and open source it, and here you are calling them lazy and/or arrogant. On the other hand - it's an open source project, I hope someone can contribute to it and create a PR with the spec.
The outrage! The vague feelings of people I don't know taking things out of context may survive. If not, I'll bring them soup.
There are some unfortunately popular projects like Puppet over the past 20 years who tried to sell the lack of documentation as a positive, or just had incomplete and outdated documentation like Chef. ;] On the flip side, IETF RFCs, Python PEPs, and Rust RFCs are examples where specifications made things clearer and open to everyone. It's not enough to open source a thing without context or comments, the learning curve must be accounted for to make it as self-service and understandable as possible to people who have domain knowledge but have never used this particular thing before. OOTBE UX. Code maybe a communication of intended system behavior, but it is often too low-level and spread around to be relied upon as a singular, compact reference.