Bijou64: A variable-length integer encoding
inkandswitch.com
Bijou64: A variable-length integer encoding
1–10 of 93 posts
Re: Bijou64: A variable-length integer encoding
#2This seems quite convoluted just to avoid the "0 can be represented in more than one way" problem.
Re: Bijou64: A variable-length integer encoding
#3This seems quite convoluted just to avoid the "0 can be represented in more than one way" problem.
It allows finding out the length (and allocating memory) after reading the first byte.
Re: Bijou64: A variable-length integer encoding
#4This seems quite convoluted just to avoid the "0 can be represented in more than one way" problem.
I think it's neat.
Re: Bijou64: A variable-length integer encoding
#5I've used LEB128 (with canonicalisation) extensively and... this looks so much nicer for most use-cases (length prefixed, supports the full uint64 range without that extra 10th byte).
The downside is the encoding size. LEB128 quickly grows to 2 bytes, but stays at 2 bytes all the way to 2^14. This is important if you're using these numbers as tags/identifiers as we were in the multicodec [1] project, or for network message lengths. bijou64 only gives you 500 [1]: https://github.com/multiformats/multicodec
Re: Bijou64: A variable-length integer encoding
#6Maybe someone can explain why an encoder would ever create the padding bytes allowed in LEB128. I contributed the parser for LEB128 in apple/swift-binary-parsing and I’m still none the wiser. I’m genuinely mystified.
Re: Bijou64: A variable-length integer encoding
#7In short: instead of a truly indefinite-length solution with a signal bit on the current byte saying whether to check the next byte, this uses a counter. Values 0x0 to 0xF7 are one-byte integers, 0xF8 to 0xFF use the upper 5 bits as a counter for the number of subsequent bytes. This limits the maximum magnitude to slightly less than 2 ^ 264 (almost all 33-byte values), which seems to be okay for practical computations. The proposed standard limits the supported size to u64 though.
The upsides: the size of the integer is apparent upon reading the first byte, and every number has exactly one canonical representation. I wish C strings had been standardized around something similar, instead on null termination.
> ...adversarial input, which is rarely in the test suite.
This made my scratch my head. My tests for quite pedestrian APIs often contain adversarial input of obvious shapes. I though that for anything security-related (like the author's project) testing against adversarial input would be be a prominent part.
Re: Bijou64: A variable-length integer encoding
#8This seems quite convoluted just to avoid the "0 can be represented in more than one way" problem.
Comparing a number to zero is something that's done a lot
Re: Bijou64: A variable-length integer encoding
#9Maybe someone can explain why an encoder would ever create the padding bytes allowed in LEB128. I contributed the parser for LEB128 in apple/swift-binary-parsing and I’m still none the wiser. I’m genuinely mystified.
You wouldn't. It's a strange argument that can be countered with, "maybe don't do that?"
Re: Bijou64: A variable-length integer encoding
#10I'm working on a C++ library at work that binds wire data and application data through token and model layers, which includes among other things a fair amount of tokenizers/composers for various formats (JSON, CBOR, BSON, CSV...).
This looks neat, but if encoding/decoding performance is important, payload size isn't and the integer is bounded, I would just put a fixed-size integer into the payload as-is.
LEB128 (and JSON for that matter) can encode integer values of arbitrary length. This doesn't, which may or may not be important but it's different.
I'll admit that I do not do any cryptographic work with my library and therefore canonical representations aren't a huge concern in my use-cases. I merely provide various configurable limits (max value length, max depth, max items per collection) in an effort to prevent infinitely long documents from hogging my tokenizers indefinitely.