Golang – encoding/csv: Reading is slow • Hacker News Board

Re: Golang – encoding/csv: Reading is slow

#71

Post by weberc2 » Sat, Sep 03, 2016, 9:31 PM UTC

geofft wrote: post #34

masklinn wrote: post #24

Earlier quoted context omitted.

ignore this, I somehow missed that the poster specifically mentioned Python 3, which does have an encoding-aware CSV module. ~~It's also unclear which version of Python is being used, the Python 2 csv module is byte-based and encoding-unaware which can lead to unexpected behaviours.~~ Go's CSV package apparently only does UTF-8, and suggestions for speeding it up in the tracker is to just remove that and work on raw…

> suggestions for speeding it up in the tracker is to just remove that and work on raw bytes (FFS) This is valid, because UTF-8 was designed to make this valid. The UTF-8 encoding of a comma, 0x2C (also the ASCII encoding of a comma), does not appear as a part of any other UTF-8 encodings. Same with the UTF-8 encoding of the double quote, 0x22. So scanning for 0x22 and 0x2C bytes, without stopping to decode other UTF…

Go's implementation allows the caller to designate any UTF8 character as the delimiter, not just as ascii.

Re: Golang – encoding/csv: Reading is slow

#72

Post by pjmlp » Sat, Sep 03, 2016, 9:36 PM UTC

jakub_h wrote: post #19

jsiep wrote: post #16

Earlier quoted context omitted.

Besides the point that the Java example is not a good one; The JVM is actually a pretty mean piece of software with a lot of optimisation. So while Go could in theory produce faster code then Java I doubt the Go compiler is clever enough to produce faster code then the JVM in a lot scenarios (at the moment).

But Java conceptually has a lot of drawbacks that require the JVM to have screaming performance to compensate for. Almost everything being a "headered" object being probably the worst offender. Even a slightly worse Go compiler is probably well compensated-for by denser data structure layout in the operating memory.

Only true until value types get productified and there are already prototype versions to play with.

Also depending on which JVM SDK is being used (Oracle Hotspot, Oracle Graal, IBM J9, HP, PTG, JET,...), the quality of escape analysis differs but it all boils down to turning those headered objects into plain structs, if possible stack allocated.

Re: Golang – encoding/csv: Reading is slow

#73

Post by pjmlp » Sat, Sep 03, 2016, 9:39 PM UTC

grandinj wrote: post #63

sorokod wrote: post #60

Earlier quoted context omitted.

Note that internally Java represents strings as utf-16

Depends. The API for strings in java is mostly UTF16 but the latest JVM will magically use UTF8 as its internal representation.

Only the Oracle one, it doesn't apply to other vendors.

Re: Golang – encoding/csv: Reading is slow

#74

Post by _ph_ » Sat, Sep 03, 2016, 9:43 PM UTC

petters wrote: post #37

Any CSV reader should be limited only be disk access right? I wrapped together a C++ program solving this problem and got 0.124 seconds. But that does not do quotations etc.

I would guess, if you directly translate your program to Go it would not be much slower. On low level code, Go 1.7 gets quite close to GCC. The point is, that doing the quotations right, especially if you allow non-ascii quotes, eats a lot of performance. So the benchmark is less about the languages involved, but rather the exact algorithms used and capabilities offered.

Re: Golang – encoding/csv: Reading is slow

#75

Post by weberc2 » Sat, Sep 03, 2016, 9:58 PM UTC

geofft wrote: post #44

snissn wrote: post #38

Earlier quoted context omitted.

That's cool about utf8 - what downsides are there to not treating utf-8 as raw bytes?

The big things are related to string length not matching byte count. strlen() is O(n) because you have to see how many sequences are actually in the string. More than that, splitting/slicing/indexing a string based on byte offsets doesn't work. For a 100-byte ASCII string, you're guaranteed that you can split it into two 50-byte strings and things will still work: you can output them separately, you can get the total…

To clarify, Letting the length of a UTF-8 string in Go is O(1); it's computed and stored on the string header at creation.

Re: Golang – encoding/csv: Reading is slow

#76

Post by jonlawlor » Sat, Sep 03, 2016, 9:59 PM UTC

endymi0n wrote: post #69

dsymonds wrote: post #66

Earlier quoted context omitted.

The standard regexp package, unlike PCRE, is actually a proper regular expression parser/matcher. Anything doing backtracking is at risk of exponential blowup and isn't safe. https://swtch.com/~rsc/regexp/regexp1.html

which is exactly what I meant with less specialized but safe :)

It is also often much faster than pcre.

Re: Golang – encoding/csv: Reading is slow

#77

Post by Roboprog » Sat, Sep 03, 2016, 9:59 PM UTC

djur wrote: post #15

It seems pretty common for languages to start out with a relatively unoptimized CSV parser (if they have one at all) and then get a faster one contributed by the community once there's enough interest. Ruby had that happen with FasterCSV. The Java comparison here seems inapt, because it doesn't do as much as the other two. It's just a naive "split on commas" implementation that wouldn't handle quoted cells. Really, i…

Indeed. Flip it around: Go-lang comes with CSV support in the standard library, whereas Java requires you to get something like the Ostermiller utilities or an Apache library.

Re: Golang – encoding/csv: Reading is slow

#78

Post by weberc2 » Sat, Sep 03, 2016, 10:00 PM UTC

wongarsu wrote: post #25

justincormack wrote: post #20

Earlier quoted context omitted.

It also seems to support UTF8, which is perhaps common now. (CSV doesnt tell you what encoding it is, in the old days it was 95% ASCII).

How would a CSV parser even break UTF8 encoding (by accident)? All CSV control characters (comma, doubleqote and newline) map to the same codepoints in ASCII and UTF8, and no non-ASCII UTF8 character uses any ASCII codepoint in it's encoding.

Go lets you use any code point as the delimiter.

Re: Golang – encoding/csv: Reading is slow

#79

Post by jonlawlor » Sat, Sep 03, 2016, 10:01 PM UTC

SeanDav wrote: post #31

Effectively one is comparing library performance here and not language performance. Granted, that line can get very blurry indeed, but in this case this says very little about golang the language and far more about a current implementation of one of the golang libraries.

It is kind of both; go doesn't allow some approaches in native go code that can make it slower than other languages. (I love go, but that is my experience.)

In this case the choice to use utf-8 everywhere, including in the csv delimiters, is making it slower.

Re: Golang – encoding/csv: Reading is slow

#80

Post by Roboprog » Sat, Sep 03, 2016, 10:02 PM UTC

petters wrote: post #37

Any CSV reader should be limited only be disk access right? I wrapped together a C++ program solving this problem and got 0.124 seconds. But that does not do quotations etc.

Good point. But assume if you run a series of tests on the same file, it's in RAM. (discard the time of the first run or two, assume data source - network, DB, file - makes access time moot)