Golang – encoding/csv: Reading is slow • Hacker News Board

Re: Golang – encoding/csv: Reading is slow

#41

Post by jjawssd » Sat, Sep 03, 2016, 6:51 PM UTC

aartur wrote: post #12

zephyrfalcon wrote: post #2

Python's csv module uses an internal module _csv which is written in C. So I'm not sure it's all that surprising that a Go implementation is a bit slower.

I run the benchmark using PyPy (which doesn't have this C extension) and got a result about 20% slower compared to CPython (ie. still faster than Go). EDIT. I also did a funny thing and replaced the CPython C _csv.so extensions with pure Python version _csv.py, from PyPy. It run about 80 (eighty) times slower. It shows what wonders does JIT do (at least to some code).

Would be a great experiment to Cythonize PyPy's _csv.so

Re: Golang – encoding/csv: Reading is slow

#42

Post by [deleted] » Sat, Sep 03, 2016, 6:58 PM UTC

snissn wrote: post #38

geofft wrote: post #34

Earlier quoted context omitted.

> suggestions for speeding it up in the tracker is to just remove that and work on raw bytes (FFS) This is valid, because UTF-8 was designed to make this valid. The UTF-8 encoding of a comma, 0x2C (also the ASCII encoding of a comma), does not appear as a part of any other UTF-8 encodings. Same with the UTF-8 encoding of the double quote, 0x22. So scanning for 0x22 and 0x2C bytes, without stopping to decode other UTF…

That's cool about utf8 - what downsides are there to not treating utf-8 as raw bytes?

[deleted]

Re: Golang – encoding/csv: Reading is slow

#43

Post by masklinn » Sat, Sep 03, 2016, 7:01 PM UTC

paulddraper wrote: post #39

geofft wrote: post #34

Earlier quoted context omitted.

> suggestions for speeding it up in the tracker is to just remove that and work on raw bytes (FFS) This is valid, because UTF-8 was designed to make this valid. The UTF-8 encoding of a comma, 0x2C (also the ASCII encoding of a comma), does not appear as a part of any other UTF-8 encodings. Same with the UTF-8 encoding of the double quote, 0x22. So scanning for 0x22 and 0x2C bytes, without stopping to decode other UTF…

> and if you're doing a string-compare for some target value that's already UTF-8, you never need to decode UTF-8 for that field at all Is Go's internal representation of the target string UTF-8?

> Is Go's internal representation of the target string UTF-8?

Kinda but kinda not, a Go string is actually an arbitrary bag of bytes, but some API (such as unicode/utf8 or `range` to iterate on codepoints — runes in Go parlance) assume it's proper UTF8.

Re: Golang – encoding/csv: Reading is slow

#44

Post by geofft » Sat, Sep 03, 2016, 7:06 PM UTC

snissn wrote: post #38

geofft wrote: post #34

Earlier quoted context omitted.

> suggestions for speeding it up in the tracker is to just remove that and work on raw bytes (FFS) This is valid, because UTF-8 was designed to make this valid. The UTF-8 encoding of a comma, 0x2C (also the ASCII encoding of a comma), does not appear as a part of any other UTF-8 encodings. Same with the UTF-8 encoding of the double quote, 0x22. So scanning for 0x22 and 0x2C bytes, without stopping to decode other UTF…

That's cool about utf8 - what downsides are there to not treating utf-8 as raw bytes?

The big things are related to string length not matching byte count. strlen() is O(n) because you have to see how many sequences are actually in the string. More than that, splitting/slicing/indexing a string based on byte offsets doesn't work. For a 100-byte ASCII string, you're guaranteed that you can split it into two 50-byte strings and things will still work: you can output them separately, you can get the total length by adding strlen() on each half, you can find a character by doing strchr() on each half, etc. For a 100-byte valid UTF-8 string, splitting it into two 50-byte strings will possibly get you an invalid string, because a character could be split in half. So strlen() (even a UTF-8-correct strlen()) and strchr() don't compose. Outputting a string in two halves works properly as long as the receiver buffers its input, and is willing to wait to reconstruct a partial character.

A related problem is that in older UNIX terminals, pressing backspace would delete one byte, not one character. Newer UNIX kernels have code in the terminal implementation to decode UTF-8 enough to backspace an entire character.

Re: Golang – encoding/csv: Reading is slow

#45

Post by endymi0n » Sat, Sep 03, 2016, 7:09 PM UTC

On a related note, also the Go stdlib regex package is pretty naive and imperformant compared to a full blown and modern backtracking PCRE implementation (at 1/10 the LOC and complexity) - same thing goes for the reflection based JSON package (which is still kinda "fast enough").

The focus wasn't so much on performance but on initial completeness, good interface, versatility, clarity and simplicity - with faster or more specialized implementations left to the community.

There might be different opinions about that, but I personally like the approach of having a solid and ordered programming pocket knife - that also doesn't replace a Katana for cutting.

Re: Golang – encoding/csv: Reading is slow

#46

Post by deno » Sat, Sep 03, 2016, 7:09 PM UTC

Better than node.js

    import * as csv from 'csv-parse';
    import * as fs from 'fs';
    
    type Line = [string,string,string,string,string,string];
    
    const parser = new csv.Parser({});
    
    parser.on('data', (line: Line) => { 
    if (line[0] === '42') {
            console.dir(line);
        } 
    });
    
    fs.createReadStream('mock_data.csv').pipe(parser);
    
    $ /usr/bin/time node parse_csv.js
    43.61user 0.85system 0:45.61elapsed 97%CPU (0avgtext+0avgdata 60076maxresident)k
    
    $ node --version
    v6.4.0

Edit: Using fast-csv

    24.28user 0.20system 0:24.58elapsed 99%CPU (0avgtext+0avgdata 91780maxresident)k

Re: Golang – encoding/csv: Reading is slow

#47

Post by halayli » Sat, Sep 03, 2016, 7:13 PM UTC

djur wrote: post #15

It seems pretty common for languages to start out with a relatively unoptimized CSV parser (if they have one at all) and then get a faster one contributed by the community once there's enough interest. Ruby had that happen with FasterCSV. The Java comparison here seems inapt, because it doesn't do as much as the other two. It's just a naive "split on commas" implementation that wouldn't handle quoted cells. Really, i…

I feel you're trying to defend go without much objectivity. Such performance gap needs to be addressed properly instead of saying it's pretty good already.

It doesn't sound right if go takes 5 hours to finish csv parsing job while Python takes 2.5 hrs.

Re: Golang – encoding/csv: Reading is slow

#48

Post by twotwotwo » Sat, Sep 03, 2016, 7:16 PM UTC

As someone notes on the bug, if you were rolling your own, there are some other things you could do--return a [][]byte that's a pointer to its internal buffer, only usable until the next row is read.

Making a version of encoding/csv that retains most of its features (custom delimiters, handling backslashes and quoting and \r) but streams like that would be a fun open source project for someone who likes Making Things Go Fast.

Re: Golang – encoding/csv: Reading is slow

#49

Post by masklinn » Sat, Sep 03, 2016, 7:16 PM UTC

deno wrote: post #46

Better than node.js import * as csv from 'csv-parse'; import * as fs from 'fs'; type Line = [string,string,string,string,string,string]; const parser = new csv.Parser({}); parser.on('data', (line: Line) => { if (line[0] === '42') { console.dir(line); } }); fs.createReadStream('mock_data.csv').pipe(parser); $ /usr/bin/time node parse_csv.js 43.61user 0.85system 0:45.61elapsed 97%CPU (0avgtext+0avgdata 60076maxresident…

csv-parse is hardly the only CSV parser for node, and it is by far the slowest: https://github.com/phihag/csv-speedtest (csv2json depends on csv-parse, so it's unsurprising that it's even slower)

Re: Golang – encoding/csv: Reading is slow

#50

Post by masklinn » Sat, Sep 03, 2016, 7:20 PM UTC

jjawssd wrote: post #41

aartur wrote: post #12

Earlier quoted context omitted.

I run the benchmark using PyPy (which doesn't have this C extension) and got a result about 20% slower compared to CPython (ie. still faster than Go). EDIT. I also did a funny thing and replaced the CPython C _csv.so extensions with pure Python version _csv.py, from PyPy. It run about 80 (eighty) times slower. It shows what wonders does JIT do (at least to some code).

Would be a great experiment to Cythonize PyPy's _csv.so

That sounds completely worthless, PyPy doesn't need a Cython version, and its library is a Python version of CPython's native csv module.