I shudder to think who needs to process a million lines of csv that fast...
lots of folks in Finance, you can share csv with any Finance company and they can process it. It's text.
21 GB/s CSV Parsing Using SIMD on AMD 9950X
11–20 of 176 posts
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#12I shudder to think who needs to process a million lines of csv that fast...
I shudder to think of what it means to be storing the _results_ of processing 21 GB/s of CSV. Hopefully some useful kind of aggregation, but if this was powering some kind of search over structured data then it has to be stored somewhere...
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#13Take that, Intel and your "let's remove AVX-512 from every consumer CPU because we want to put slow cores on every single one of them and also not consider multi-pumping it"
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#14Earlier quoted context omitted.
Yea wtf is that chart, it literally skips 4 cpu generations where it shows “massive performance gain”. Straight to the trash with this post.
It also appears to be reporting whole-CPU vs. single thread, 1.3 GB/sec is not impressive for single thread perf
Agreed. How hard is it to keep hardware fixed, load the data into memory, and use a single core for your benchmarks? When I see a chart like that I think, "What else are they hiding?"
Folks should check out https://github.com/dathere/qsv if they need an actually fast CSV parser.
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#15I shudder to think who needs to process a million lines of csv that fast...
lots of folks in Finance, you can share csv with any Finance company and they can process it. It's text.
Humans generate decisions / text information at rates of ~bytes per second at most. There is barely enough humans around to generate 21GB/s of information even if all they did was make financial decisions!
So 21 GB/s would be solely algos talking to algos... Given all the investment in the algos, surely they don't need to be exchanging CSV around?
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#16Considering the non-standard nature of CSV, quoting throughput numbers in bytes is meaningless. It makes sense for JSON, since you know what the output is going to be (e.g. floats, integers, strings, hashmaps, etc).
With CSV you only get strings for each column, so 21 GB/s of comma splitting would be the pinnacle of meaninglessness. Like, okay, but I still have to parse the stringy data, so what gives? Yeah, the blog post does reference float parsing, but a single float per line would count as "CSV".
Now someone might counter and say that I should just read the README.MD, but then that suspicion simply turns out to be true: They don't actually do any escaping or quoting by default, making the quoted numbers an example of heavily misleading advertising.
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#17I shudder to think who needs to process a million lines of csv that fast...
For all its many weaknesses, I believe CSV is still the most common data interchange format.
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#18Earlier quoted context omitted.
lots of folks in Finance, you can share csv with any Finance company and they can process it. It's text.
Humans generate decisions / text information at rates of ~bytes per second at most. There is barely enough humans around to generate 21GB/s of information even if all they did was make financial decisions! So 21 GB/s would be solely algos talking to algos... Given all the investment in the algos, surely they don't need to be exchanging CSV around?
> Humans generate decisions / text information at rates of ~bytes per second at most
Yes, but the consequences of these decisions are worth much more. You attach an ID to the user, and an ID to the transaction. You store the location and time where it was made. Ect.
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#19I shudder to think who needs to process a million lines of csv that fast...
It's become a very common interchange format, even internally; it's also easy to deflate. I have had to work on codebases where CSV was being pumped out at basically the speed of a NIC card (its origin was Netflow, and then aggregated and otherwise processed, and the results sent via CSV to a master for further aggregation and analysis).
I really don't get, though, why people can't just use protocol buffers instead. Is protobuf really that hard?
Re: 21 GB/s CSV Parsing Using SIMD on AMD 9950X
#20Earlier quoted context omitted.
lots of folks in Finance, you can share csv with any Finance company and they can process it. It's text.
Humans generate decisions / text information at rates of ~bytes per second at most. There is barely enough humans around to generate 21GB/s of information even if all they did was make financial decisions! So 21 GB/s would be solely algos talking to algos... Given all the investment in the algos, surely they don't need to be exchanging CSV around?
You might have accumulated some decades of data in that format and now want to ingest it into a database.