Show HN: Parsing CSV files with GPU
github.com
Show HN: Parsing CSV files with GPU
1–10 of 63 posts
Re: Show HN: Parsing CSV files with GPU
#2I like your creativity in applying the GPU to a less typical task.
Is it the case that the CPU version could be sped up dramatically by using multiple cores and a variation on the line splitting technique?
Re: Show HN: Parsing CSV files with GPU
#3I like your creativity in applying the GPU to a less typical task. Is it the case that the CPU version could be sped up dramatically by using multiple cores and a variation on the line splitting technique?
Yes, CPU version can be accelerated as well, depending on availability of SSE extensions.
Re: Show HN: Parsing CSV files with GPU
#4I like your creativity in applying the GPU to a less typical task. Is it the case that the CPU version could be sped up dramatically by using multiple cores and a variation on the line splitting technique?
You could use a lockfree queue, fill it, and spawn a suitable number of consumers that will put the parsed data back into a new queue. If the data order matters, then some extra boilerplate is needed, and I do not know how it's done on the GPU, or if it's needed at all.
Seeing that it takes 14.5s with the handrolled code to parse 750GB, I however doubt the optimization is needed - it's more than 50GB every second and you need some really really exotic hardware to generate that much data. You still need to do something with the data and it's likely significantly more computational intensitive - that should be on a different thread.
That said, using the GPU would free up the CPU to do meaningful work with the data. But to be fair, assuming 1GB/s datarate, it's still 12,5m to read it all and you only gain 14.5s or less than 2% speedup if you're CPU bound.
Re: Show HN: Parsing CSV files with GPU
#5I like your creativity in applying the GPU to a less typical task. Is it the case that the CPU version could be sped up dramatically by using multiple cores and a variation on the line splitting technique?
Yes, CPU version can be accelerated as well, depending on availability of SSE extensions.
Shouldn't that be the default assumption? It's pretty rare to not have SSE.
Re: Show HN: Parsing CSV files with GPU
#6I like your creativity in applying the GPU to a less typical task. Is it the case that the CPU version could be sped up dramatically by using multiple cores and a variation on the line splitting technique?
You could use a lockfree queue, fill it, and spawn a suitable number of consumers that will put the parsed data back into a new queue. If the data order matters, then some extra boilerplate is needed, and I do not know how it's done on the GPU, or if it's needed at all. Seeing that it takes 14.5s with the handrolled code to parse 750GB, I however doubt the optimization is needed - it's more than 50GB every second and…
It should be 750MB, not 750GB. There is no way a single hard drive can read 750GB in 14 seconds.
Re: Show HN: Parsing CSV files with GPU
#7I don't understand the benchmark. How can a 750GB file be read from disk in 1.5 seconds, let alone be parsed? He mentions it's a 2TB drive, so it's not even SSD presumably?
Re: Show HN: Parsing CSV files with GPU
#8I don't understand the benchmark. How can a 750GB file be read from disk in 1.5 seconds, let alone be parsed? He mentions it's a 2TB drive, so it's not even SSD presumably?
The README has just been updated to change 750GB to MB.
https://github.com/antonmks/nvParse/commit/fab4c4728096003bc...
Re: Show HN: Parsing CSV files with GPU
#9Earlier quoted context omitted.
You could use a lockfree queue, fill it, and spawn a suitable number of consumers that will put the parsed data back into a new queue. If the data order matters, then some extra boilerplate is needed, and I do not know how it's done on the GPU, or if it's needed at all. Seeing that it takes 14.5s with the handrolled code to parse 750GB, I however doubt the optimization is needed - it's more than 50GB every second and…
It should be 750MB, not 750GB. There is no way a single hard drive can read 750GB in 14 seconds.
Heh, you cant even DMA 750GB from memory to the graphics card (assuming 16GB/s, using 32 lanes) in 14s.
Re: Show HN: Parsing CSV files with GPU
#10I don't think CPU and GPU is going to make much difference here. You speed up in GPU run could be because of hot cache of file system. Try running GPU run first and non-GPU after that and plz post the results.