31–40 of 66 posts
> Footer indexed So, like ZIP? > Uses XXH3 for integrity checks I don’t think XXH3 is suitable for that purpose. It’s not cryptographically secure and designed mostly for stuff like hash tables (e.g. relatively small data).
I use CBZ to archive both physical and digital comic books so I was interested in the idea of an improved container format, but the claimed improvements here don't make sense. --- For example they make a big deal about each archive entry being aligned to a 4 KiB boundary "allowing for DirectStorage transfers directly from disk to GPU memory", but the pages within a CBZ are going to be encoded (JPEG/PNG/etc) rather th…
It seems that JPEG can be decoded on the GPU [1] [2]
> CRC32 is limited by memory bandwidth if you're using a normal (i.e. SIMD) implementation.
According to smhasher tests [3] CRC32 is not limited by memory bandwidth. Even if we multiply CRC32 scores x4 (to estimate 512 bit wide SIMD from 128 bit wide results), we still don't get close to memory bandwidth.
The 32 bit hash of CRC32 is too low for file checksums. xxhash is definitely an improvement over CRC32.
> to actually check the integrity of archived files you want to use something like sha256, not CRC32 or xxhash
Why would you need to use a cryptographic hash function to check integrity of archived files? Quality a non-cryptographic hash function will detect corruptions due to things like bit-rot, bad RAM, etc. just the same.
And why is 256 bits needed here? Kopia developers, for example, think 128 bit hashes are big enough for backup archives [4].
[1] https://docs.nvidia.com/cuda/nvjpeg/index.html
[2] https://github.com/CESNET/GPUJPEG
Honest question, something I don't understand, if you use DirectStorage to move images directly to the GPU (I assume into the VRAM) where the decoding take place? directly on the GPU? Can GPU decode PNG? it is very unfriendly format for GPU as far as I know
From the readme: > Note: DirectStorage isn't avaliable for images yet (as far as I know), but I've made sure to accomodate such a thing in the future with this format. So the whole DirectStorage thing is just a nothingburger. The author glosses over the fact that decoding images on GPU is not possible (or at least very impractical).
This feels like the wrong end to optimize. Zip is plenty of fast, especially when it comes to a few hundred pages of a comic. Meanwhile the image decoding can take a while when you want to have a quick thumbnail overview showing all those hundred pages at once. No comic/ebook software I have ever touched as managed to match the responsiveness of an actual book where you can flip through those hundreds of pages in a s…
There are already quite a few cbz archives in the wild that contain jxl encoded images. That's a multi-resolution format at least to the extent that it supports progressive decoding at fixed levels that range from 1:8 to as high as 1:4096. I think it might also support other arbitrary ratios subject to certain encoding constraints but I'm less clear on that.
Readers might need to be updated to make use of the feature in an intelligent manner though. The jxl cbzs I've encountered either didn't make use of progressive encoding or else the software I used failed to take advantage of it - I'm not sure which.
Earlier quoted context omitted.
Bullshit asymmetry by way of impulsive LLM slop strikes again. Every new readme, announcement post, and codebase is tailored to achieve maximum bloviation. No substance, no credibility———just vibes.
If you read the reddit thread, it was coded by hand then only bug checked with ai.
I use CBZ to archive both physical and digital comic books so I was interested in the idea of an improved container format, but the claimed improvements here don't make sense. --- For example they make a big deal about each archive entry being aligned to a 4 KiB boundary "allowing for DirectStorage transfers directly from disk to GPU memory", but the pages within a CBZ are going to be encoded (JPEG/PNG/etc) rather th…
> the pages within a CBZ are going to be encoded (JPEG/PNG/etc) rather than just being bitmaps. They need to be decoded first, the GPU isn't going to let you create a texture directly from JPEG data. It seems that JPEG can be decoded on the GPU [1] [2] > CRC32 is limited by memory bandwidth if you're using a normal (i.e. SIMD) implementation. According to smhasher tests [3] CRC32 is not limited by memory bandwidth. E…
What makes you say this? I agree that there are better algorithms than CRC32 for this usecase, but if I was implementing something I'd most likely still truncate the hash to somewhere in the same ballpark (likely either 32, 48, or 64 bits).
Note that the purpose of the hash is important. These aren't being used for deduplication where you need a guaranteed unique value between all independently queried pieces of data globally but rather just to detect file corruption. At 32 bits you have only a 1 out of 2^(32-1) chance of a false negative. That should be more than enough. By the time you make it to 64 bits, if you encounter a corrupted file once _every nanosecond_ for the next 500 years or so you would expect to miss only a single event. That is a rather absurd level of reliability in my view.
Honest question, something I don't understand, if you use DirectStorage to move images directly to the GPU (I assume into the VRAM) where the decoding take place? directly on the GPU? Can GPU decode PNG? it is very unfriendly format for GPU as far as I know
From the readme: > Note: DirectStorage isn't avaliable for images yet (as far as I know), but I've made sure to accomodate such a thing in the future with this format. So the whole DirectStorage thing is just a nothingburger. The author glosses over the fact that decoding images on GPU is not possible (or at least very impractical).
It would be one thing if you were designing a format to optimize feeding data to an ML model during training but that's not even remotely what this is supposed to be.
Earlier quoted context omitted.
If you read the reddit thread, it was coded by hand then only bug checked with ai.
It was benchmarked with AI. Benchmarks being the main reason for this thing existing...
I haven't read the reddit thread or anything but If the author coded it by hand or is passionate about this project, he will probably understand what we are talking about.
But I don't believe its such a big deal to have a benchmark be written by AI though? no?
> Footer indexed So, like ZIP? > Uses XXH3 for integrity checks I don’t think XXH3 is suitable for that purpose. It’s not cryptographically secure and designed mostly for stuff like hash tables (e.g. relatively small data).
> It’s not cryptographically secure Neither is CRC32. I'm pretty sure xxhash is a straight upgrade compared to CRC32.
Unclear; performance should be pretty similar to CRC32 (depending on implementation), and since integrity checking can basically be done at RAM read speeds this should not matter either way.