Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Golang – encoding/csv: Reading is slow

github.com

Post reply on HN
Print view View original

61–70 of 102 posts

Re: Golang – encoding/csv: Reading is slow

#61

Post by deno »

masklinn wrote: post #49
deno wrote: post #46

Better than node.js import * as csv from 'csv-parse'; import * as fs from 'fs'; type Line = [string,string,string,string,string,string]; const parser = new csv.Parser({}); parser.on('data', (line: Line) => { if (line[0] === '42') { console.dir(line); } }); fs.createReadStream('mock_data.csv').pipe(parser); $ /usr/bin/time node parse_csv.js 43.61user 0.85system 0:45.61elapsed 97%CPU (0avgtext+0avgdata 60076maxresident…

csv-parse is hardly the only CSV parser for node, and it is by far the slowest: https://github.com/phihag/csv-speedtest (csv2json depends on csv-parse, so it's unsurprising that it's even slower)

The fastest streaming example on that list (csv-parser) is still 5× slower than Python.

Re: Golang – encoding/csv: Reading is slow

#62

Post by merb »

dalke wrote: post #40
nradov wrote: post #14

Earlier quoted context omitted.

The Java code is defective. It's not checking for double quotes. The CSV format allows for commas inside column values by surrounding with double quotes, and then you can also put double quotes within such values by escaping them as double double quotes. Fix those defects and the Java code will be a little slower. With modern JVMs, Java can occasionally actually be faster than native compiled languages due to dynamic…

A few minutes ago (and after your comment) one of the commenters of that issue tested against Apache Commons CSV and found that Java was 1.9x faster than Go, rather than the original 3x: https://github.com/golang/go/issues/16791#issuecomment-24456...

actually the java one is still amazing since it's a cold jvm. when it would be a big file I would think that java is far ahead of both. with an aggressive jit. maybe pypy is faster than all 3 :D

Re: Golang – encoding/csv: Reading is slow

#63

Post by grandinj »

sorokod wrote: post #60
piinbinary wrote: post #33

Earlier quoted context omitted.

I did the same [0]. It runs almost as fast as the Java implementation. I'd be interested to see how yours works if you are willing to share it. Edit: Plus one that is ~2x faster than Java by avoiding allocations [1]. [0] https://gist.github.com/jmikkola/6ac96ad6d6f66e772c33ec41ed2... [1] https://gist.github.com/jmikkola/7ded8392226b7659c881f5540be...

Note that internally Java represents strings as utf-16

Depends. The API for strings in java is mostly UTF16 but the latest JVM will magically use UTF8 as its internal representation.

Re: Golang – encoding/csv: Reading is slow

#64

Post by chrisseaton »

jjawssd wrote: post #41
aartur wrote: post #12

Earlier quoted context omitted.

I run the benchmark using PyPy (which doesn't have this C extension) and got a result about 20% slower compared to CPython (ie. still faster than Go). EDIT. I also did a funny thing and replaced the CPython C _csv.so extensions with pure Python version _csv.py, from PyPy. It run about 80 (eighty) times slower. It shows what wonders does JIT do (at least to some code).

Would be a great experiment to Cythonize PyPy's _csv.so

PyPy doesn't have a _csv.so - that was the main point of the comment you replied to.

Re: Golang – encoding/csv: Reading is slow

#65

Post by drej »

geofft wrote: post #52
masklinn wrote: post #32

Earlier quoted context omitted.

1. there's nothing useless about it 2. the Python 3 CSV library supports arbitrary codepoints as delimiter, quote character and escape character (if applicable)

> 1. there's nothing useless about it Have you ever seen a "C"SV with a multibyte sequence as a delimiter? I haven't. Even if such a thing exists, the feature is of negative utility if it slows down CSV parsing for everyone else. If you must, write two implementations, and use the slow path if your delimiter is multibyte.

That's precisely my suggestion https://github.com/golang/go/issues/16791#issuecomment-24209...

Moving from runes to bytes in reading gives us a nice speedup - not quite to eliminate the gap, but it's a start. The rest is likely all the memory copies - once the data is read in a buffer, then copied byte by byte into a slice and only then converted into a string, which is another copy, because strings can't be based on pre-existing byte slices (not in the public API that is).

Re: Golang – encoding/csv: Reading is slow

#66

Post by dsymonds »

endymi0n wrote: post #45

On a related note, also the Go stdlib regex package is pretty naive and imperformant compared to a full blown and modern backtracking PCRE implementation (at 1/10 the LOC and complexity) - same thing goes for the reflection based JSON package (which is still kinda "fast enough"). The focus wasn't so much on performance but on initial completeness, good interface, versatility, clarity and simplicity - with faster or m…

The standard regexp package, unlike PCRE, is actually a proper regular expression parser/matcher. Anything doing backtracking is at risk of exponential blowup and isn't safe.

https://swtch.com/~rsc/regexp/regexp1.html

Re: Golang – encoding/csv: Reading is slow

#67

Post by rdtsc »

zephyrfalcon wrote: post #2

Python's csv module uses an internal module _csv which is written in C. So I'm not sure it's all that surprising that a Go implementation is a bit slower.

But not sure if it matters. Go is free to use a C module to load csv files as well. It can use assembly or other tricks as well perhaps.

Re: Golang – encoding/csv: Reading is slow

#68

Post by eternalban »

jakub_h wrote: post #19
jsiep wrote: post #16

Earlier quoted context omitted.

Besides the point that the Java example is not a good one; The JVM is actually a pretty mean piece of software with a lot of optimisation. So while Go could in theory produce faster code then Java I doubt the Go compiler is clever enough to produce faster code then the JVM in a lot scenarios (at the moment).

But Java conceptually has a lot of drawbacks that require the JVM to have screaming performance to compensate for. Almost everything being a "headered" object being probably the worst offender. Even a slightly worse Go compiler is probably well compensated-for by denser data structure layout in the operating memory.

> Java conceptually has a lot of drawbacks

In what sense is an articulated object a "conceptual drawback"? It is a richer object model and SMI and friends had the engineering chops to makes it highly performant.

Re: Golang – encoding/csv: Reading is slow

#69

Post by endymi0n »

dsymonds wrote: post #66
endymi0n wrote: post #45

On a related note, also the Go stdlib regex package is pretty naive and imperformant compared to a full blown and modern backtracking PCRE implementation (at 1/10 the LOC and complexity) - same thing goes for the reflection based JSON package (which is still kinda "fast enough"). The focus wasn't so much on performance but on initial completeness, good interface, versatility, clarity and simplicity - with faster or m…

The standard regexp package, unlike PCRE, is actually a proper regular expression parser/matcher. Anything doing backtracking is at risk of exponential blowup and isn't safe. https://swtch.com/~rsc/regexp/regexp1.html

which is exactly what I meant with less specialized but safe :)

Re: Golang – encoding/csv: Reading is slow

#70

Post by weberc2 »

geofft wrote: post #34
masklinn wrote: post #24

Earlier quoted context omitted.

ignore this, I somehow missed that the poster specifically mentioned Python 3, which does have an encoding-aware CSV module. ~~It's also unclear which version of Python is being used, the Python 2 csv module is byte-based and encoding-unaware which can lead to unexpected behaviours.~~ Go's CSV package apparently only does UTF-8, and suggestions for speeding it up in the tracker is to just remove that and work on raw…

> suggestions for speeding it up in the tracker is to just remove that and work on raw bytes (FFS) This is valid, because UTF-8 was designed to make this valid. The UTF-8 encoding of a comma, 0x2C (also the ASCII encoding of a comma), does not appear as a part of any other UTF-8 encodings. Same with the UTF-8 encoding of the double quote, 0x22. So scanning for 0x22 and 0x2C bytes, without stopping to decode other UTF…

Go's implementation allows the caller to designate any UTF8 character as the delimiter, not just as ascii.
Post reply on HN