Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computing Adler32 Checksums at 41 GB/s

wooo.sh

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Computing Adler32 Checksums at 41 GB/s

#21

Post by benreesman »

wooosh wrote: post #19
benreesman wrote: post #18

Earlier quoted context omitted.

Effective hiring managers are always paying attention in the hopes of noticing the people doing this excellent stuff and asking whether “now” or “soon” is the right time to offer such people high paying jobs. I doubt my current garage band could afford the OP just this moment, but I sure wish we could!

> I doubt my current garage band could afford the OP just this moment, but I sure wish we could! Well, I intend to finish high school at a minimum before pursuing employment :)

Someone as far ahead of the curve as you clearly are might enjoy the second chapter of Coders at Work, which is an interview with Brad Fitzpatrick. bfitz wrote everything from memcached to big chunks of TailScale and much in between.

He went to university in CS but would have been bored to sleep if he didn’t have something else going on, so he founded and ran LiveJournal simultaneously.

I bet someone on this thread knows him and I bet he’d take the time to offer some pointers to an up-and-comer like yourself. I’ve never met him, but I did some business with Six Apart in a previous life and people say he’s a really nice guy.

Either way, keep it up!

Re: Computing Adler32 Checksums at 41 GB/s

#22

Post by NavinF »

>diminishing returns especially due to it working faster than the speed of my RAM (2667MT/s * 8 = ~21 GB/s).

That sounds kinda slow; Is there only 1 DIMM in the slots? I remember benchmarking 40GiB/s read speed on an older system that had 2 dual-rank DIMMs (4 ranks in total).

I'd expect 3200mbit/s*(64 data lines)*(2 memory channels) = ~48 GiB/s on a typical DDR4 desktop and a lot more with overclocked ram.

Great writeup either way.

Re: Computing Adler32 Checksums at 41 GB/s

#23

Post by wooosh »

NavinF wrote: post #22

>diminishing returns especially due to it working faster than the speed of my RAM (2667MT/s * 8 = ~21 GB/s). That sounds kinda slow; Is there only 1 DIMM in the slots? I remember benchmarking 40GiB/s read speed on an older system that had 2 dual-rank DIMMs (4 ranks in total). I'd expect 3200mbit/s*(64 data lines)*(2 memory channels) = ~48 GiB/s on a typical DDR4 desktop and a lot more with overclocked ram. Great writ…

Yes, this is on a single 8GB 2667MHz DIMM in a laptop.

edit: For dual channel RAM, I would suspect the throughput depends on how the kernel decides to map physical memory to virtual addresses.

Re: Computing Adler32 Checksums at 41 GB/s

#24

Post by josephg »

Does anyone have any recommendations for checksumming algorithms in greenfield systems? It seems like there’s lots of innovation in crypto secure hashing functions. But I have a greenfield project where I need checksums but don’t care about crypto properties. Is CRC32c still a good choice or has the industry moved on?

Re: Computing Adler32 Checksums at 41 GB/s

#25

Post by james-redwood »

jiggawatts wrote: post #13

I hope this brilliant work has been merged into the relevant open source libraries. Something that’s unfair about the world is that work like this could reach billions of people and save a million dollars worth of time and electricity annually but is being done gratis. It would be amazing if there were charities that rewarded high-impact open source contributions like this proportionally to the benefits to humanity…

It works well in science (think the Nobel Prizes), and there's certainly more than enough money floating around in the tech community for it to work.

Re: Computing Adler32 Checksums at 41 GB/s

#26

Post by adrian_b »

wooosh wrote: post #23
NavinF wrote: post #22

>diminishing returns especially due to it working faster than the speed of my RAM (2667MT/s * 8 = ~21 GB/s). That sounds kinda slow; Is there only 1 DIMM in the slots? I remember benchmarking 40GiB/s read speed on an older system that had 2 dual-rank DIMMs (4 ranks in total). I'd expect 3200mbit/s*(64 data lines)*(2 memory channels) = ~48 GiB/s on a typical DDR4 desktop and a lot more with overclocked ram. Great writ…

Yes, this is on a single 8GB 2667MHz DIMM in a laptop. edit: For dual channel RAM, I would suspect the throughput depends on how the kernel decides to map physical memory to virtual addresses.

The memory is already mapped by the BIOS/EFI firmware, before the kernel takes control.

By default, whenever the memory modules used in all different channels have the same size, e.g. two 8 GB modules, the firmware maps the modules with interleaved addresses, to ensure a double throughput for 2 channels, or triple/quadruple/etc. for workstation/server motherboards with more memory channels.

Re: Computing Adler32 Checksums at 41 GB/s

#27

Post by diroussel »

josephg wrote: post #24

Does anyone have any recommendations for checksumming algorithms in greenfield systems? It seems like there’s lots of innovation in crypto secure hashing functions. But I have a greenfield project where I need checksums but don’t care about crypto properties. Is CRC32c still a good choice or has the industry moved on?

What are your requirements? Tamper resistance? Error detection? Error correction? Speed vs time vs space trade off?

Re: Computing Adler32 Checksums at 41 GB/s

#28

Post by josephg »

diroussel wrote: post #27
josephg wrote: post #24

Does anyone have any recommendations for checksumming algorithms in greenfield systems? It seems like there’s lots of innovation in crypto secure hashing functions. But I have a greenfield project where I need checksums but don’t care about crypto properties. Is CRC32c still a good choice or has the industry moved on?

What are your requirements? Tamper resistance? Error detection? Error correction? Speed vs time vs space trade off?

Just error detection for corrupt or partial disk writes.
Post reply on HN