Computing Adler32 Checksums at 41 GB/s
1–10 of 28 posts
Re: Computing Adler32 Checksums at 41 GB/s
#2While micro-optimizations are interesting, there are two questions left unanswered:
- Does this change noticeably affect the total runtime? The checksum seems simple enough that the slight difference here wouldn't show up in PNG benchmarks.
- The proposed solution uses AVX2, which is not currently used in the original codebase. Would any other part of the processing benefit from using newer instructions?
Re: Computing Adler32 Checksums at 41 GB/s
#3Ooh now that is very interesting. I would really love to see how this speeds up the run-time of fpng as a whole, if you have any numbers. It looks like fjxl [0] and fpnge [1] (which also uses AVX2) are at the Pareto front for lossless image compression right now [2], but if this speeds things significantly then it's possible there'll be a huge shakeup!
[0] https://github.com/libjxl/libjxl/tree/main/experimental/fast...
[1] https://github.com/veluca93/fpnge
[2] https://twitter.com/richgel999/status/1485976101692358656
Re: Computing Adler32 Checksums at 41 GB/s
#4Ooh now that is very interesting. I would really love to see how this speeds up the run-time of fpng as a whole, if you have any numbers. It looks like fjxl [0] and fpnge [1] (which also uses AVX2) are at the Pareto front for lossless image compression right now [2], but if this speeds things significantly then it's possible there'll be a huge shakeup! [0] https://github.com/libjxl/libjxl/tree/main/experimental/fast.…
If image encode/decode speed is the only concern, libjpegturbo is going to be orders of magnitude faster than any of these lossless schemes. With jpeg, you could encode 1080p bitmaps in The frequency domain is a really powerful place to operate in when you are dealing with this amount of data.
Re: Computing Adler32 Checksums at 41 GB/s
#5Ooh now that is very interesting. I would really love to see how this speeds up the run-time of fpng as a whole, if you have any numbers. It looks like fjxl [0] and fpnge [1] (which also uses AVX2) are at the Pareto front for lossless image compression right now [2], but if this speeds things significantly then it's possible there'll be a huge shakeup! [0] https://github.com/libjxl/libjxl/tree/main/experimental/fast.…
If image encode/decode speed is the only concern, libjpegturbo is going to be orders of magnitude faster than any of these lossless schemes. With jpeg, you could encode 1080p bitmaps in The frequency domain is a really powerful place to operate in when you are dealing with this amount of data.
That's not true. libjpeg-turbo is ~50 MB/s last I tried - plus it's not lossless. fjxl and fpnge are basically an order of magnitude faster than that. libjpeg-turbo isn't even the fastest jpeg codec - you should check out the (relatively obscure) libmango - roughly 1 gbps decode on a 2020 macbook pro - or nvJPEG for GPU-based JPEG decoding. Supposedly there's even faster GPU-based decoders than nvJPEG, too.
Re: Computing Adler32 Checksums at 41 GB/s
#6Earlier quoted context omitted.
If image encode/decode speed is the only concern, libjpegturbo is going to be orders of magnitude faster than any of these lossless schemes. With jpeg, you could encode 1080p bitmaps in The frequency domain is a really powerful place to operate in when you are dealing with this amount of data.
That's not true. libjpeg-turbo is ~50 MB/s last I tried - plus it's not lossless. fjxl and fpnge are basically an order of magnitude faster than that. libjpeg-turbo isn't even the fastest jpeg codec - you should check out the (relatively obscure) libmango - roughly 1 gbps decode on a 2020 macbook pro - or nvJPEG for GPU-based JPEG decoding. Supposedly there's even faster GPU-based decoders than nvJPEG, too.
> GPU-based
How does this impact the overall latency of encoding a single image?
Re: Computing Adler32 Checksums at 41 GB/s
#7While micro-optimizations are interesting, there are two questions left unanswered: - Does this change noticeably affect the total runtime? The checksum seems simple enough that the slight difference here wouldn't show up in PNG benchmarks. - The proposed solution uses AVX2, which is not currently used in the original codebase. Would any other part of the processing benefit from using newer instructions?
If checksum calculation was any substantial portion of image decoding, I think that would be a strong case for simply not checking the checksum.
If you put corrupted data into a PNG decoder, I don't think it's awfully important to most users whether they get a decode error or a garbled image out.
Re: Computing Adler32 Checksums at 41 GB/s
#8Earlier quoted context omitted.
That's not true. libjpeg-turbo is ~50 MB/s last I tried - plus it's not lossless. fjxl and fpnge are basically an order of magnitude faster than that. libjpeg-turbo isn't even the fastest jpeg codec - you should check out the (relatively obscure) libmango - roughly 1 gbps decode on a 2020 macbook pro - or nvJPEG for GPU-based JPEG decoding. Supposedly there's even faster GPU-based decoders than nvJPEG, too.
> GPU-based How does this impact the overall latency of encoding a single image?
Probably quite a bit, I don't know. The typical use case is to load up thousands of JPEGs at once to get good throughput despite copy overhead. You can see here the benchmark against jpeg-turbo: https://developer.nvidia.com/blog/leveraging-hardware-jpeg-d...
Re: Computing Adler32 Checksums at 41 GB/s
#9Ooh now that is very interesting. I would really love to see how this speeds up the run-time of fpng as a whole, if you have any numbers. It looks like fjxl [0] and fpnge [1] (which also uses AVX2) are at the Pareto front for lossless image compression right now [2], but if this speeds things significantly then it's possible there'll be a huge shakeup! [0] https://github.com/libjxl/libjxl/tree/main/experimental/fast.…
Unfortunately I haven’t had the time to do a proper benchmark, and the fpng test executable only decodes/encodes a single image which produces very noisy/inconclusive results. However, I’m under the impression that it doesn’t make a large difference in terms of overall time.
fpnge (which I wasn’t aware of until now) appears to already be using a very similar (identical?) algorithm, so I suspect the relative performance of fpng and fpnge would not be significantly impacted by this change.
Re: Computing Adler32 Checksums at 41 GB/s
#10While micro-optimizations are interesting, there are two questions left unanswered: - Does this change noticeably affect the total runtime? The checksum seems simple enough that the slight difference here wouldn't show up in PNG benchmarks. - The proposed solution uses AVX2, which is not currently used in the original codebase. Would any other part of the processing benefit from using newer instructions?
If checksum calculation was any substantial portion of image decoding, I think that would be a strong case for simply not checking the checksum. If you put corrupted data into a PNG decoder, I don't think it's awfully important to most users whether they get a decode error or a garbled image out.
This was actually considered, and other libraries do ignore checksums, or at least have options to: