21 GB/s CSV Parsing Using SIMD on AMD 9950X

#1

Post by zigzag312 » Fri, May 09, 2025, 1:38 PM UTC

21 GB/s CSV Parsing Using SIMD on AMD 9950X
nietras.com

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#2

Post by winterbloom » Fri, May 09, 2025, 2:06 PM UTC

This is a staggering ~3x improvement in just under 2 years since Sep was introduced June, 2023.

You can't claim this when you also do a huge hardware jump

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#3

Post by criddell » Fri, May 09, 2025, 2:07 PM UTC

I was expecting to see assembly language and was pleasantly surprised to see C#. Very impressive.

Nice work!

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#4

Post by WD-42 » Fri, May 09, 2025, 2:10 PM UTC

winterbloom wrote: post #2

This is a staggering ~3x improvement in just under 2 years since Sep was introduced June, 2023. You can't claim this when you also do a huge hardware jump

Yea wtf is that chart, it literally skips 4 cpu generations where it shows “massive performance gain”.

Straight to the trash with this post.

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#5

Post by freeone3000 » Fri, May 09, 2025, 2:10 PM UTC

winterbloom wrote: post #2

This is a staggering ~3x improvement in just under 2 years since Sep was introduced June, 2023. You can't claim this when you also do a huge hardware jump

They claim a 3GB/s improvement versus previous version of sep on equal hardware — and unlike “marketing” benchmarks, include the actual speed achieved and the hardware used.

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#6

Post by voidUpdate » Fri, May 09, 2025, 2:11 PM UTC

I shudder to think who needs to process a million lines of csv that fast...

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#7

Post by vessenes » Fri, May 09, 2025, 2:14 PM UTC

If we are lucky we will see Arthur Whitney get triggered and post either a one liner beating this or a shakti engine update and a one liner beating this. Progress!

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#8

Post by stabbles » Fri, May 09, 2025, 2:14 PM UTC

Instead of doing 4 comparisons against each character `\n`, `\r`, `;` and `"` followed by 3 or operations, a common trick is to do 1 shuffle, 1 comparison and 0 or operations. I blogged about this trick: https://stoppels.ch/2022/11/30/io-is-no-longer-the-bottlenec... (Trick 2)

Edit: they do make use of ternary logic to avoid one or operation, which is nice. Basically (a | b | c) | d is computed using `vpternlogd` and `vpor` resp.

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#9

Post by jbverschoor » Fri, May 09, 2025, 2:16 PM UTC

winterbloom wrote: post #2

This is a staggering ~3x improvement in just under 2 years since Sep was introduced June, 2023. You can't claim this when you also do a huge hardware jump

They also included 0.9.0 vs 0.10.0. on the new hardware. (21385 vs 18203), so the jump because of software is 17%.

Then if we take 0.9.0 on previous hardware (13088) and add the 17%, it's 15375. Version 0.1.0 was 7335.

So... 15375/7335 -> a staggering 2.1x improvement in just under 2 years

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#10

Post by g-mork » Fri, May 09, 2025, 2:17 PM UTC

WD-42 wrote: post #4

winterbloom wrote: post #2

This is a staggering ~3x improvement in just under 2 years since Sep was introduced June, 2023. You can't claim this when you also do a huge hardware jump

Yea wtf is that chart, it literally skips 4 cpu generations where it shows “massive performance gain”. Straight to the trash with this post.

It also appears to be reporting whole-CPU vs. single thread, 1.3 GB/sec is not impressive for single thread perf