21 GB/s CSV Parsing Using SIMD on AMD 9950X

Post by elteto » Sat, May 10, 2025, 1:42 AM UTC

raron wrote: post #59

bombela wrote: post #48

Earlier quoted context omitted.

protobuf is more friction, and actually slow to write and read. For better or worse, CSV is easy to produce via printf. Easy to read by breaking lines and splitting by the delimiter. Escaping delimiters part of the content is not hard, though often added as an afterthought. Protobuf requires to install a library. Understand how it works. Write a schema file. Share the shema to others. The API is cumbersome. Finally t…

> For better or worse, CSV is easy to produce via printf. Easy to read by breaking lines and splitting by the delimiter. Escaping delimiters part of the content is not hard, though often added as an afterthought. Based on the amount of software I seen producing broken CSV or can't parse (more-or-less) valid CSV, I don't think that is true. It seems to be easy, because just printf("%s,%d,%d\n", ...) but it is full of…

Post by bobmcnamara » Sat, May 10, 2025, 2:08 AM UTC

stingraycharles wrote: post #123

sitkack wrote: post #49

Earlier quoted context omitted.

That is what Intel does, they build up a market (Optane) and then do a rug pull (Depth Cameras). They continue to do this thing where they do a huge push into a new technology, then don't see the uptake and let it die. Instead of building slowly and then at the right time, doing a big push. Optane support was just getting mature in the Linux kernel when they pulled it. And they focused on some weird cost cutting move…

Until this day, I miss Optane — I work for a timeseries database company focused on finance, the amount of use cases I have that screams “faster than NVMe, slower than RAM” is insane. And these companies have money to throw at these problems. Which begs the question, why isn’t anyone else stepping into this gap? Is the technology heavily patented?

Post by titzer » Sat, May 10, 2025, 2:17 AM UTC

rcxdude wrote: post #118

kyboren wrote: post #82

Earlier quoted context omitted.

> What the VLIW of Itanium needed and never really got was proper compiler support. This is kinda under-selling it. The fundamental problem with statically-scheduled VLIW machines like Itanium is it puts all of the complexity in the compiler. Unfortunately it turns out it's just really hard to make a good static scheduler! In contrast, dynamically-scheduled out-of-order superscalar machines work great but put all the…

>What happened was that static scheduling stayed really hard while the transistor overhead for dynamic scheduling became irrelevantly cheap Is the latter part true? AFAIK most of modern CPU die area and power consumption goes towards overhead as opposed to the actual ALU operations.

Post by bawolff » Sat, May 10, 2025, 2:30 AM UTC

winterbloom wrote: post #2

This is a staggering ~3x improvement in just under 2 years since Sep was introduced June, 2023. You can't claim this when you also do a huge hardware jump

Post by trollbridge » Sat, May 10, 2025, 3:08 AM UTC

Neywiny wrote: post #138

elteto wrote: post #131

Earlier quoted context omitted.

Not an issue when you control both ends of the pipe. CSV is a great interchange format for tabular data, especially so if it's only/mostly numeric. If you need to pass tabular data from internal service X to internal service Y it's great. And it's really fast.

Hmmm if they're just internal tools, why not just an array of structs? No parsing needed. Can have optionals. Can't go faster than nothing.