I think it would be really cool to integrate a GPU into a database to speed up certain operations if an optimizer can decide that certain parts of a query will benefit from it. The thrust docs [0] indicate that the C++ gpu library can be used effectively for sorting, I wonder if sorts on non-indexed fields can be sped up by attaching a GPU to my database!
Show HN: Parsing CSV files with GPU
31–40 of 63 posts
Re: Show HN: Parsing CSV files with GPU
#32Though there is no standard definition of CSV, de facto processing it properly requires recognizing quotes, and also escapes of literal quotes using double quoting: this, "is, like, CSV", "with three so-called ""fields""" Note that unquoted leading and trailing whitespace, and whitespace around the commas, is deleted, too. (See CSV page in the Wikipedia) A GPU-accelerated string split could be useful but it's not qui…
> unquoted leading and trailing whitespace, and whitespace around the commas, is deleted, too. (See CSV page in the Wikipedia)
I'm not sure why you referenced the Wikipedia page -- it indicates that according to RFC 4180, (a) "spaces outside quotes in a field are not allowed", and (b) "Spaces are considered part of a field and should not be ignored."
Maybe you're referring to the comment that CSV parsers should "be liberal in what you accept from others"...?
http://en.wikipedia.org/wiki/Comma-separated_values#Basic_ru...
Re: Show HN: Parsing CSV files with GPU
#33Though there is no standard definition of CSV, de facto processing it properly requires recognizing quotes, and also escapes of literal quotes using double quoting: this, "is, like, CSV", "with three so-called ""fields""" Note that unquoted leading and trailing whitespace, and whitespace around the commas, is deleted, too. (See CSV page in the Wikipedia) A GPU-accelerated string split could be useful but it's not qui…
You forgot with fields with line-breaks, with non printrable characters and so on.
I don't need to reproduce an entire detailed CSV spec here to make the point.
There is a danger in being too detailed because there is no universal spec anyway. For many users, CSV is whatever the most recent version or two of Microsoft Excel accept, as confirmed by trial-and-error reverse engineering.
Re: Show HN: Parsing CSV files with GPU
#34Though there is no standard definition of CSV, de facto processing it properly requires recognizing quotes, and also escapes of literal quotes using double quoting: this, "is, like, CSV", "with three so-called ""fields""" Note that unquoted leading and trailing whitespace, and whitespace around the commas, is deleted, too. (See CSV page in the Wikipedia) A GPU-accelerated string split could be useful but it's not qui…
> unquoted leading and trailing whitespace, and whitespace around the commas, is deleted, too. (See CSV page in the Wikipedia) I'm not sure why you referenced the Wikipedia page -- it indicates that according to RFC 4180, (a) "spaces outside quotes in a field are not allowed", and (b) "Spaces are considered part of a field and should not be ignored." Maybe you're referring to the comment that CSV parsers should "be l…
> (a) "spaces outside quotes in a field are not allowed", and (b) "Spaces are considered part of a field and should not be ignored."
Note that it's clear that (b) refers only to spaces inside quotes, because spaces outside quotes are not allowed.
> Maybe you're referring to the comment that CSV parsers should "be liberal in what you accept from others"...?
I actually don't believe in that principle. If there is a spec that everyone agrees upon, violations should be accurately and loudly diagnosed and rejected.
Both preparation of the data format and processing of that data format should be conservative. Being liberal in what is accepted has unintended negative consequences.
Be that as it may, there is no universal CSV spec, though. RFC 4180 is just someone's opinion on what CSV should be. CSV is something that has been widely implemented in numerous programs over numerous decades, in different ways.
My main point holds that if you split the string on commas and do nothing else, then one of the aspects you're neglecting to handle is the treatment of unquoted whitespace outside of a field.
Re: Show HN: Parsing CSV files with GPU
#35In the real world... read the CSV into a database (doesn't really matter how fast or slow it is). Access the data from the database.
In the real world you may have to ingest hundreds of megabytes or more of CSV data, daily, from a provider or partner, and CPU time spent ingesting that data is not spent processing transactions. So any speedup you can get is well worth it.
Re: Show HN: Parsing CSV files with GPU
#36Though there is no standard definition of CSV, de facto processing it properly requires recognizing quotes, and also escapes of literal quotes using double quoting: this, "is, like, CSV", "with three so-called ""fields""" Note that unquoted leading and trailing whitespace, and whitespace around the commas, is deleted, too. (See CSV page in the Wikipedia) A GPU-accelerated string split could be useful but it's not qui…
This is a perfect example of how text parsing is really inherently non-parallelizable. It's very rare that you can do anything useful with a buffer of text without knowing the precise state of the parse at the beginning of that buffer.
The kinds of patterns that would make parsing more parallelizable, like marking the beginning of a delimited region with its length, are human unfriendly so would never be part of an actual text format. Who would ever want to write this?
# Update "19" whenever string length changes.
x = {19}"String of length 19"Re: Show HN: Parsing CSV files with GPU
#37In the real world... read the CSV into a database (doesn't really matter how fast or slow it is). Access the data from the database.
In the real world, sometimes you have an external data source, in csv (which you can't control), which is changing, which you want to reread as quickly as possible. :)
You then use PostgreSQL Foreign Data Wrapper (aka SQL/MED)
Re: Show HN: Parsing CSV files with GPU
#38Earlier quoted context omitted.
> unquoted leading and trailing whitespace, and whitespace around the commas, is deleted, too. (See CSV page in the Wikipedia) I'm not sure why you referenced the Wikipedia page -- it indicates that according to RFC 4180, (a) "spaces outside quotes in a field are not allowed", and (b) "Spaces are considered part of a field and should not be ignored." Maybe you're referring to the comment that CSV parsers should "be l…
> (a) "spaces outside quotes in a field are not allowed", and (b) "Spaces are considered part of a field and should not be ignored." Note that it's clear that (b) refers only to spaces inside quotes, because spaces outside quotes are not allowed. > Maybe you're referring to the comment that CSV parsers should "be liberal in what you accept from others"...? I actually don't believe in that principle. If there is a spe…
> I actually don't believe in that principle. If there is a spec that everyone agrees upon, violations should be accurately and loudly diagnosed and rejected.
Both preparation of the data format and processing of that data format should be conservative. Being liberal in what is accepted has unintended negative consequences.
I agree on principle. But when Marie in accounting opens the CSV that Bob from customer X sent her, if software A opens it and software B screams "error!!!", she's going to use the one that "works". And that means B's vendor will make their tool liberal of what they accept too.
Similarly for websites, that's why browsers fixes what they see rather than not showing what you are asking it to show.
Does it lead to a worse state of things for clear format, with more errors in the wild and no actual reference to base yourself on ? Yes. But it is still what end users want.
Re: Show HN: Parsing CSV files with GPU
#39Though there is no standard definition of CSV, de facto processing it properly requires recognizing quotes, and also escapes of literal quotes using double quoting: this, "is, like, CSV", "with three so-called ""fields""" Note that unquoted leading and trailing whitespace, and whitespace around the commas, is deleted, too. (See CSV page in the Wikipedia) A GPU-accelerated string split could be useful but it's not qui…
This is a perfect example of how text parsing is really inherently non-parallelizable. It's very rare that you can do anything useful with a buffer of text without knowing the precise state of the parse at the beginning of that buffer. The kinds of patterns that would make parsing more parallelizable, like marking the beginning of a delimited region with its length, are human unfriendly so would never be part of an a…
And that's why you let the computer do it for you.
Re: Show HN: Parsing CSV files with GPU
#40I gotta admit, having written a high-speed, multi-threaded streaming CSV parser for the browser[1], I'm quite impressed by your project. I've done CUDA programming before and it's not easy (albeit libraries do help). Good work! [1] http://papaparse.com
I just used Papa at the office today. Remembered it from a while ago, it really turned out to be effortless. Excellent work.
(While I'm at it, may I suggest to provide CDN URLs on the website? IMO it would make the otherwise awesome page perfect.)
Thank you again.