Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computing Adler32 Checksums at 41 GB/s

wooo.sh

Post reply on HN
Print view View original

11–20 of 28 posts

Re: Computing Adler32 Checksums at 41 GB/s

#11

Post by averne_ »

bob1029 wrote: post #6
pizza wrote: post #5

Earlier quoted context omitted.

That's not true. libjpeg-turbo is ~50 MB/s last I tried - plus it's not lossless. fjxl and fpnge are basically an order of magnitude faster than that. libjpeg-turbo isn't even the fastest jpeg codec - you should check out the (relatively obscure) libmango - roughly 1 gbps decode on a 2020 macbook pro - or nvJPEG for GPU-based JPEG decoding. Supposedly there's even faster GPU-based decoders than nvJPEG, too.

> GPU-based How does this impact the overall latency of encoding a single image?

I've written an open-source driver for the decoding side of the nvjpg module found in the Tegra X1 (ie. earlier hardware revision than the one in the A100).

I did some quick benchmarks against libjpeg-turbo, if that can give you an idea. I expect encoding performance would be similar.

https://github.com/averne/oss-nvjpg#performance

Re: Computing Adler32 Checksums at 41 GB/s

#13

Post by jiggawatts »

I hope this brilliant work has been merged into the relevant open source libraries.

Something that’s unfair about the world is that work like this could reach billions of people and save a million dollars worth of time and electricity annually but is being done gratis.

It would be amazing if there were charities that rewarded high-impact open source contributions like this proportionally to the benefits to humanity…

Re: Computing Adler32 Checksums at 41 GB/s

#14

Post by Nyan »

wooosh wrote: post #9
pizza wrote: post #3

Ooh now that is very interesting. I would really love to see how this speeds up the run-time of fpng as a whole, if you have any numbers. It looks like fjxl [0] and fpnge [1] (which also uses AVX2) are at the Pareto front for lossless image compression right now [2], but if this speeds things significantly then it's possible there'll be a huge shakeup! [0] https://github.com/libjxl/libjxl/tree/main/experimental/fast.…

Unfortunately I haven’t had the time to do a proper benchmark, and the fpng test executable only decodes/encodes a single image which produces very noisy/inconclusive results. However, I’m under the impression that it doesn’t make a large difference in terms of overall time. fpnge (which I wasn’t aware of until now) appears to already be using a very similar (identical?) algorithm, so I suspect the relative performan…

As someone who has been recently optimising fpnge, Adler32 computation is pretty much negligible regarding overall runtime. The Huffman coding and filter search take up most of the time. (IIRC fpng doesn't do any filter search, but Huffman encoding isn't vectorized, so I'd expect that to dominate fpng's runtime)

Re: Computing Adler32 Checksums at 41 GB/s

#16

Post by nigeltao »

Here's another SIMD implementation, with commentary: https://github.com/google/wuffs/blob/main/std/adler32/common...

Like the fpng implementation, it's SSE (128-bit registers), but the inner loop eats 32 bytes at a time, not 16.

"Wuffs’ Adler-32 implementation is around 6.4x faster (11.3GB/s vs 1.76GB/s) than the one from zlib-the-library", which IIUC is roughly comparable to the article's defer32. https://nigeltao.github.io/blog/2021/fastest-safest-png-deco...

Re: Computing Adler32 Checksums at 41 GB/s

#17

Post by ebiggers »

Note that libdeflate has used essentially the same method since 2016 (https://github.com/ebiggers/libdeflate/blob/v0.4/lib/adler32...), though I recently switched it to use a slightly different method (https://github.com/ebiggers/libdeflate/blob/v1.12/lib/x86/ad...) that performs more consistently across different families of x86 CPUs.

Re: Computing Adler32 Checksums at 41 GB/s

#18

Post by benreesman »

jiggawatts wrote: post #13

I hope this brilliant work has been merged into the relevant open source libraries. Something that’s unfair about the world is that work like this could reach billions of people and save a million dollars worth of time and electricity annually but is being done gratis. It would be amazing if there were charities that rewarded high-impact open source contributions like this proportionally to the benefits to humanity…

Effective hiring managers are always paying attention in the hopes of noticing the people doing this excellent stuff and asking whether “now” or “soon” is the right time to offer such people high paying jobs.

I doubt my current garage band could afford the OP just this moment, but I sure wish we could!

Re: Computing Adler32 Checksums at 41 GB/s

#19

Post by wooosh »

benreesman wrote: post #18
jiggawatts wrote: post #13

I hope this brilliant work has been merged into the relevant open source libraries. Something that’s unfair about the world is that work like this could reach billions of people and save a million dollars worth of time and electricity annually but is being done gratis. It would be amazing if there were charities that rewarded high-impact open source contributions like this proportionally to the benefits to humanity…

Effective hiring managers are always paying attention in the hopes of noticing the people doing this excellent stuff and asking whether “now” or “soon” is the right time to offer such people high paying jobs. I doubt my current garage band could afford the OP just this moment, but I sure wish we could!

> I doubt my current garage band could afford the OP just this moment, but I sure wish we could!

Well, I intend to finish high school at a minimum before pursuing employment :)

Post reply on HN