Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

21 GB/s CSV Parsing Using SIMD on AMD 9950X

nietras.com

Post reply on HN
Print view View original

171–176 of 176 posts

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#171

Post by nly »

cyral wrote: post #97
zzbn00 wrote: post #15

Earlier quoted context omitted.

Humans generate decisions / text information at rates of ~bytes per second at most. There is barely enough humans around to generate 21GB/s of information even if all they did was make financial decisions! So 21 GB/s would be solely algos talking to algos... Given all the investment in the algos, surely they don't need to be exchanging CSV around?

The only real example I can think of is the US options market feed. It is up to something like 50 GiB/s now, and is open 6.5 hours per day. Even a small subset of the feed that someone may be working on for data analysis could be huge. I agree CSV shouldn't even be used here but I am sure it is.

OPRA is a half dozen terabytes of data per day compressed.

CSV wouldn't even be considered.

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#172

Post by gpderetta »

justsid wrote: post #159
pezezin wrote: post #129

Earlier quoted context omitted.

Yeah, VLIW is still used for stuff like DSP and GPUs, but it doesn't make sense for general computing.

GPUs have long since moved away from VLIW as well

They are still mostly statically scheduled which I think it is the point the parent is making.

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#173

Post by wtallis »

chao- wrote: post #38

It feels crazy to me that Intel spent years dedicating die space on consumer SKUs to "make fetch happen" with AVX-512, and as more and more libraries are finally using it, as Intel's goal is achieved, they have removed AVX-512 from their consumer SKUs. It isn't that AMD has better AVX-512 support, which would be an impressive upset on it's own. Instead, it is only that AMD has AVX-512 on consumer CPUs, because Intel…

The craziest thing about Intel's AVX-512 story is that from 2015 through 2020 they were shipping consumer CPUs with die space reserved for the AVX-512 register file, but no actual AVX-512 capability. Then they shipped one short-lived generation of desktop processors and two generations of laptop processors that actually had AVX-512 functionality before disabling it.

AMD has already been shipping AVX-512 in their consumer processors for longer than Intel did.

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#174

Post by jsploit »

constantcrying wrote: post #75

There are very good alternatives to csv for storing and exchanging floating point/other data. The HDF5 format is very good and allows far more structure in your files, as well as metadata and different types of lossless and lossy compression.

The lack of concurrent access support in the official HDF5 library (the only implementation with full format support) can be a major drawback. There is ongoing work on that front [1] though it's unclear when it will land.

[1] https://github.com/LifeboatLLC/MT-HDF5

Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X

#175

Post by nietras1 »

stabbles wrote: post #8

Instead of doing 4 comparisons against each character `\n`, `\r`, `;` and `"` followed by 3 or operations, a common trick is to do 1 shuffle, 1 comparison and 0 or operations. I blogged about this trick: https://stoppels.ch/2022/11/30/io-is-no-longer-the-bottlenec... (Trick 2) Edit: they do make use of ternary logic to avoid one or operation, which is nice. Basically (a | b | c) | d is computed using `vpternlogd` and…

This is true if you have fixed set of characters within niblet, but is is not necessarily faster, as the shuffle requires the extra register. I tried it and it wasn't faster here.
Post reply on HN