Effectively one is comparing library performance here and not language performance. Granted, that line can get very blurry indeed, but in this case this says very little about golang the language and far more about a current implementation of one of the golang libraries.
Golang – encoding/csv: Reading is slow
31–40 of 102 posts
Re: Golang – encoding/csv: Reading is slow
#32Earlier quoted context omitted.
It also seems to support UTF8, which is perhaps common now. (CSV doesnt tell you what encoding it is, in the old days it was 95% ASCII).
All 3 support UTF8. The difference with the Go implementation is the mostly useless capability to have a utf-8 multibyte item as the delimiter.
1. there's nothing useless about it
2. the Python 3 CSV library supports arbitrary codepoints as delimiter, quote character and escape character (if applicable)
Re: Golang – encoding/csv: Reading is slow
#33I wrote my own in Go that is blazing fast using bytes. I know the data is ascii so I was able to use that to my advantage.
I did the same [0]. It runs almost as fast as the Java implementation.
I'd be interested to see how yours works if you are willing to share it.
Edit: Plus one that is ~2x faster than Java by avoiding allocations [1].
[0] https://gist.github.com/jmikkola/6ac96ad6d6f66e772c33ec41ed2...
[1] https://gist.github.com/jmikkola/7ded8392226b7659c881f5540be...
Re: Golang – encoding/csv: Reading is slow
#34It seems pretty common for languages to start out with a relatively unoptimized CSV parser (if they have one at all) and then get a faster one contributed by the community once there's enough interest. Ruby had that happen with FasterCSV. The Java comparison here seems inapt, because it doesn't do as much as the other two. It's just a naive "split on commas" implementation that wouldn't handle quoted cells. Really, i…
ignore this, I somehow missed that the poster specifically mentioned Python 3, which does have an encoding-aware CSV module. ~~It's also unclear which version of Python is being used, the Python 2 csv module is byte-based and encoding-unaware which can lead to unexpected behaviours.~~ Go's CSV package apparently only does UTF-8, and suggestions for speeding it up in the tracker is to just remove that and work on raw…
> suggestions for speeding it up in the tracker is to just remove that and work on raw bytes (FFS)
This is valid, because UTF-8 was designed to make this valid. The UTF-8 encoding of a comma, 0x2C (also the ASCII encoding of a comma), does not appear as a part of any other UTF-8 encodings. Same with the UTF-8 encoding of the double quote, 0x22. So scanning for 0x22 and 0x2C bytes, without stopping to decode other UTF-8 sequences along the way, will produce the correct result for a valid UTF-8 input string. Then you fully decode UTF-8 for the individual fields when needed (and if you're doing a string-compare for some target value that's already UTF-8, you never need to decode UTF-8 for that field at all).
Re: Golang – encoding/csv: Reading is slow
#35Earlier quoted context omitted.
It also seems to support UTF8, which is perhaps common now. (CSV doesnt tell you what encoding it is, in the old days it was 95% ASCII).
How would a CSV parser even break UTF8 encoding (by accident)? All CSV control characters (comma, doubleqote and newline) map to the same codepoints in ASCII and UTF8, and no non-ASCII UTF8 character uses any ASCII codepoint in it's encoding.
I've seen one break because of the byte-order marker that sometimes gets added to UTF-8. I don't remember the details of why that broke it, just remember that it worked fine on everything except that.
Re: Golang – encoding/csv: Reading is slow
#36The Java code is not a CSV parser.
I added the results of using Apache Commons CSV to the GitHub thread.
After using that, Python was actually by far the fastest. Hooray for performance sensitive code in C :)
Re: Golang – encoding/csv: Reading is slow
#37Any CSV reader should be limited only be disk access right? I wrapped together a C++ program solving this problem and got 0.124 seconds. But that does not do quotations etc.
Re: Golang – encoding/csv: Reading is slow
#38Earlier quoted context omitted.
ignore this, I somehow missed that the poster specifically mentioned Python 3, which does have an encoding-aware CSV module. ~~It's also unclear which version of Python is being used, the Python 2 csv module is byte-based and encoding-unaware which can lead to unexpected behaviours.~~ Go's CSV package apparently only does UTF-8, and suggestions for speeding it up in the tracker is to just remove that and work on raw…
> suggestions for speeding it up in the tracker is to just remove that and work on raw bytes (FFS) This is valid, because UTF-8 was designed to make this valid. The UTF-8 encoding of a comma, 0x2C (also the ASCII encoding of a comma), does not appear as a part of any other UTF-8 encodings. Same with the UTF-8 encoding of the double quote, 0x22. So scanning for 0x22 and 0x2C bytes, without stopping to decode other UTF…
That's cool about utf8 - what downsides are there to not treating utf-8 as raw bytes?
Re: Golang – encoding/csv: Reading is slow
#39Earlier quoted context omitted.
ignore this, I somehow missed that the poster specifically mentioned Python 3, which does have an encoding-aware CSV module. ~~It's also unclear which version of Python is being used, the Python 2 csv module is byte-based and encoding-unaware which can lead to unexpected behaviours.~~ Go's CSV package apparently only does UTF-8, and suggestions for speeding it up in the tracker is to just remove that and work on raw…
> suggestions for speeding it up in the tracker is to just remove that and work on raw bytes (FFS) This is valid, because UTF-8 was designed to make this valid. The UTF-8 encoding of a comma, 0x2C (also the ASCII encoding of a comma), does not appear as a part of any other UTF-8 encodings. Same with the UTF-8 encoding of the double quote, 0x22. So scanning for 0x22 and 0x2C bytes, without stopping to decode other UTF…
> and if you're doing a string-compare for some target value that's already UTF-8, you never need to decode UTF-8 for that field at all
Is Go's internal representation of the target string UTF-8?
Re: Golang – encoding/csv: Reading is slow
#40Earlier quoted context omitted.
How about Java? It is quite funny Java version is much faster than Python version even when Python version does use C ? Something fishy going on.
The Java code is defective. It's not checking for double quotes. The CSV format allows for commas inside column values by surrounding with double quotes, and then you can also put double quotes within such values by escaping them as double double quotes. Fix those defects and the Java code will be a little slower. With modern JVMs, Java can occasionally actually be faster than native compiled languages due to dynamic…
A few minutes ago (and after your comment) one of the commenters of that issue tested against Apache Commons CSV and found that Java was 1.9x faster than Go, rather than the original 3x: https://github.com/golang/go/issues/16791#issuecomment-24456...