Am I crazy? With a hot disk cache, the cut command he gave takes 1.2 seconds on my machine.
Show HN: Parsing CSV files with GPU
61–63 of 63 posts
Re: Show HN: Parsing CSV files with GPU
#62Though there is no standard definition of CSV, de facto processing it properly requires recognizing quotes, and also escapes of literal quotes using double quoting: this, "is, like, CSV", "with three so-called ""fields""" Note that unquoted leading and trailing whitespace, and whitespace around the commas, is deleted, too. (See CSV page in the Wikipedia) A GPU-accelerated string split could be useful but it's not qui…
This is a perfect example of how text parsing is really inherently non-parallelizable. It's very rare that you can do anything useful with a buffer of text without knowing the precise state of the parse at the beginning of that buffer. The kinds of patterns that would make parsing more parallelizable, like marking the beginning of a delimited region with its length, are human unfriendly so would never be part of an a…
http://www.cse.chalmers.se/edu/year/2010/course/TDA341/Paper...
Also look at CYK parsing algorithm, it is highly parallelizable (it is based on matrix multiplication).
Re: Show HN: Parsing CSV files with GPU
#63Earlier quoted context omitted.
You could use a lockfree queue, fill it, and spawn a suitable number of consumers that will put the parsed data back into a new queue. If the data order matters, then some extra boilerplate is needed, and I do not know how it's done on the GPU, or if it's needed at all. Seeing that it takes 14.5s with the handrolled code to parse 750GB, I however doubt the optimization is needed - it's more than 50GB every second and…
It should be 750MB, not 750GB. There is no way a single hard drive can read 750GB in 14 seconds.
Well, that makes a lot more sense.