Libbbf: Bound Book Format, A high-performance container for comics and manga • Hacker News Board

Re: Libbbf: Bound Book Format, A high-performance container for comics and manga

#61

Post by wyldfire » Wed, Jan 21, 2026, 4:56 PM UTC

jmillikin wrote: post #47

zigzag312 wrote: post #45

Earlier quoted context omitted.

From SMHasher test results quality of xxhash seems higher. It has less bias / higher uniformity that CRC. What bothers me with probability calculations, is that they always assume perfect uniformity. I've never seen any estimates how bias affects collision probability and how to modify the probability formula to account for non-perfect uniformity of a hash function.

It doesn't matter, though. xxhash is better than crc32 for hashing keys in a hash table, but both of them are inappropriate for file checksums -- especially as part of a data archival/durability strategy. It's not obvious to me that per-page checksums in an archive format for comic books are useful at all, but if you really wanted them for some reason then crc32 (fast, common, should detect bad RAM or a decoder bug)…

> both of them are inappropriate for file checksums

CRCs like CRC32 were born for this kind of work. CRCs detect corruption when transmitting/storing data. What do you mean when you say that it's inappropriate for file checksums? It's ideal for file checksums.

Re: Libbbf: Bound Book Format, A high-performance container for comics and manga

#62

Post by zigzag312 » Wed, Jan 21, 2026, 5:17 PM UTC

fc417fc802 wrote: post #60

zigzag312 wrote: post #54

Earlier quoted context omitted.

It depends on how you calculate statistics. If you are designing a file format that over the lifetime of the format hundreds of millions of user will use (storing billions of files), what are the chances that 32 bits checksum won't be able to catch at least one corruption? During transfer over unstable wireless internet connection, storage on cheap flash drive, poor HDD with a higher error rate, unstable RAM etc. We…

No, it doesn't "depend on how you calculate statistics". Or rather you are not asking the right question. We do not care if a different person suffers a false negative. The question is if you, personally, are likely to suffer a false negative. In other words, will any given real world deployment of the solution be expected to suffer from an unacceptably high rate of false negatives? Answering that requires figuring o…

I can't say I agree with your logic here. We are not talking about any specific backup or anything like that. We are talking about the design of a file format that is going to be used globally.

Business running a lottery has to calculate the odds of anyone winning, not just the odds of a single person winning. Same, a designer of a file format has to consider chances for all users. What percent of users will be affected by any design decision.

For example, what if you would offer a guarantee that 32 bit hash will protect you from corruption, and compensate generously anyone who would get this type of corruption; how would you calculate probability then?

Re: Libbbf: Bound Book Format, A high-performance container for comics and manga

#63

Post by zigzag312 » Wed, Jan 21, 2026, 5:26 PM UTC

minitech wrote: post #59

zigzag312 wrote: post #57

Earlier quoted context omitted.

> Uniformity isn’t directly important for error detection. Is there any proof of this? I'm interested in reading more about it. > detect all burst errors up to 32 bits in size What if errors are not consecutive bits?

There’s a whole field’s worth of really cool stuff about error correction that I wish I knew a fraction of enough to give reading recommendations about, but my comment wasn’t that deep – it’s just that in hashes, you obviously care about distribution because that’s almost the entire point of non-cryptographic hashes, and in error correction you only care that x ≠ y implies f(x) ≠ f(y) with high probability, which is…

This seems to make sense, but I need to read more about error correction to fully understand it. I was considering possibility that data could also contain patterns where error detection performs poorly due to bias, and I haven't seen how to include these estimates in probability calculations.

Re: Libbbf: Bound Book Format, A high-performance container for comics and manga

#64

Post by fc417fc802 » Wed, Jan 21, 2026, 5:32 PM UTC

zigzag312 wrote: post #62

fc417fc802 wrote: post #60

Earlier quoted context omitted.

No, it doesn't "depend on how you calculate statistics". Or rather you are not asking the right question. We do not care if a different person suffers a false negative. The question is if you, personally, are likely to suffer a false negative. In other words, will any given real world deployment of the solution be expected to suffer from an unacceptably high rate of false negatives? Answering that requires figuring o…

I can't say I agree with your logic here. We are not talking about any specific backup or anything like that. We are talking about the design of a file format that is going to be used globally. Business running a lottery has to calculate the odds of anyone winning, not just the odds of a single person winning. Same, a designer of a file format has to consider chances for all users. What percent of users will be affec…

If you offer compensation then of course you need to consider your risk exposure, ie total users. That's similar to a lottery where the central authority is concerned with all payouts while an individual is only concerned with their own payout.

Outside of brand reputation issues that is not how real world products are designed. You design a tool for the specific task it will be used for. You don't run your statistics in aggregate based on the expected number of customers.

Users are independent from one another. If the population doubles my filesystem doesn't suddenly become less reliable. If more people purchase the same laptop that I have the chance of mine failing doesn't suddenly go up. If more people deep fry things in their kitchen my own personal risk of a kitchen fire isn't increased regardless of how busy the fire department might become.

Re: Libbbf: Bound Book Format, A high-performance container for comics and manga

#65

Post by gwern » Wed, Jan 21, 2026, 7:57 PM UTC

Imustaskforhelp wrote: post #39

toyg wrote: post #36

Earlier quoted context omitted.

It was benchmarked with AI. Benchmarks being the main reason for this thing existing...

I mean, its open source so people can create benchmark and independently verify if the AI was wrong and then have the claims be passed to the author. I haven't read the reddit thread or anything but If the author coded it by hand or is passionate about this project, he will probably understand what we are talking about. But I don't believe its such a big deal to have a benchmark be written by AI though? no?

> I mean, its open source so people can create benchmark and independently verify if the AI was wrong and then have the claims be passed to the author.

Thank you for volunteering. I look forward to your results.

Re: Libbbf: Bound Book Format, A high-performance container for comics and manga

#66

Post by Imustaskforhelp » Wed, Jan 21, 2026, 8:17 PM UTC

gwern wrote: post #65

Imustaskforhelp wrote: post #39

Earlier quoted context omitted.

I mean, its open source so people can create benchmark and independently verify if the AI was wrong and then have the claims be passed to the author. I haven't read the reddit thread or anything but If the author coded it by hand or is passionate about this project, he will probably understand what we are talking about. But I don't believe its such a big deal to have a benchmark be written by AI though? no?

> I mean, its open source so people can create benchmark and independently verify if the AI was wrong and then have the claims be passed to the author. Thank you for volunteering. I look forward to your results.

> Thank you for volunteering. I look forward to your results.

Sure can you wait a few weeks tho? I know nothing about benchmarking so gonna learn it first and I have a few tests to prepare for irl.

I do feel like someone else more passionate about the project should try to pick the benchmarking though.

I don't mind benchmarking it but I only know tools like hyper for benchmarks & I have played with my fair share of zip archives and their random access retrieval but I feel like even that would depend from source to source.

There are some experienced people in here who are really cool at what they do, I just wanted to say that if someone's interested and already has the Domain Specific knowledge to benchmark & they enjoy it in the first place, this having AI benchmark shouldn't be much of a problem in comparison.