Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Golang – encoding/csv: Reading is slow

github.com

Post reply on HN
Print view View original

11–20 of 102 posts

Re: Golang – encoding/csv: Reading is slow

#11

Post by jerf »

0xFFC wrote: post #4
zephyrfalcon wrote: post #2

Python's csv module uses an internal module _csv which is written in C. So I'm not sure it's all that surprising that a Go implementation is a bit slower.

How about Java? It is quite funny Java version is much faster than Python version even when Python version does use C ? Something fishy going on.

There's nothing mysterious about the Java version... that's not a CSV parser. The other two things are truly CSV parsers. (Inasmuch as there is such a thing for such an ill-defined format. (No, the RFC is not determinative.))

It's easy to be faster if you do fundamentally less. Not necessarily wrong, depending on your task, but it's not comparable.

Re: Golang – encoding/csv: Reading is slow

#12

Post by aartur »

zephyrfalcon wrote: post #2

Python's csv module uses an internal module _csv which is written in C. So I'm not sure it's all that surprising that a Go implementation is a bit slower.

I run the benchmark using PyPy (which doesn't have this C extension) and got a result about 20% slower compared to CPython (ie. still faster than Go).

EDIT. I also did a funny thing and replaced the CPython C _csv.so extensions with pure Python version _csv.py, from PyPy. It run about 80 (eighty) times slower. It shows what wonders does JIT do (at least to some code).

Re: Golang – encoding/csv: Reading is slow

#13

Post by __s »

0xFFC wrote: post #4
zephyrfalcon wrote: post #2

Python's csv module uses an internal module _csv which is written in C. So I'm not sure it's all that surprising that a Go implementation is a bit slower.

How about Java? It is quite funny Java version is much faster than Python version even when Python version does use C ? Something fishy going on.

The java source is using str.split(',') which is completely different than Python's offering of dialect/delimiter/etc https://github.com/python/cpython/blob/master/Lib/csv.py#L24

Re: Golang – encoding/csv: Reading is slow

#14

Post by nradov »

0xFFC wrote: post #4
zephyrfalcon wrote: post #2

Python's csv module uses an internal module _csv which is written in C. So I'm not sure it's all that surprising that a Go implementation is a bit slower.

How about Java? It is quite funny Java version is much faster than Python version even when Python version does use C ? Something fishy going on.

The Java code is defective. It's not checking for double quotes. The CSV format allows for commas inside column values by surrounding with double quotes, and then you can also put double quotes within such values by escaping them as double double quotes. Fix those defects and the Java code will be a little slower.

With modern JVMs, Java can occasionally actually be faster than native compiled languages due to dynamic optimization at runtime.

Re: Golang – encoding/csv: Reading is slow

#15

Post by djur »

It seems pretty common for languages to start out with a relatively unoptimized CSV parser (if they have one at all) and then get a faster one contributed by the community once there's enough interest. Ruby had that happen with FasterCSV.

The Java comparison here seems inapt, because it doesn't do as much as the other two. It's just a naive "split on commas" implementation that wouldn't handle quoted cells. Really, if Go's CSV reader is only 200% slower than that and 50% slower than Python's optimized C implementation, that's pretty good already.

Re: Golang – encoding/csv: Reading is slow

#16

Post by jsiep »

0xFFC wrote: post #4
zephyrfalcon wrote: post #2

Python's csv module uses an internal module _csv which is written in C. So I'm not sure it's all that surprising that a Go implementation is a bit slower.

How about Java? It is quite funny Java version is much faster than Python version even when Python version does use C ? Something fishy going on.

Besides the point that the Java example is not a good one; The JVM is actually a pretty mean piece of software with a lot of optimisation. So while Go could in theory produce faster code then Java I doubt the Go compiler is clever enough to produce faster code then the JVM in a lot scenarios (at the moment).

Re: Golang – encoding/csv: Reading is slow

#17

Post by chrisper »

baq wrote: post #7
chrisper wrote: post #3

Earlier quoted context omitted.

You should contribute this knowledge to the GitHub issue.

ultimately it's of academic interest - the end user doesn't care what precisely is going on under the hood.

I thought it is rather part of the solution/answer to the issue. It is expected to be slower, so it's not necessarily broken. Kind of it's a feature, not a bug.

Re: Golang – encoding/csv: Reading is slow

#18

Post by chrisper »

jlarocco wrote: post #6
chrisper wrote: post #3

Earlier quoted context omitted.

You should contribute this knowledge to the GitHub issue.

A good idea, but I think it's common knowledge. Lots of modules in the stdlib are written in C.

Well, it was news to me. Maybe because I only use Python casually.

Re: Golang – encoding/csv: Reading is slow

#19

Post by jakub_h »

jsiep wrote: post #16
0xFFC wrote: post #4

Earlier quoted context omitted.

How about Java? It is quite funny Java version is much faster than Python version even when Python version does use C ? Something fishy going on.

Besides the point that the Java example is not a good one; The JVM is actually a pretty mean piece of software with a lot of optimisation. So while Go could in theory produce faster code then Java I doubt the Go compiler is clever enough to produce faster code then the JVM in a lot scenarios (at the moment).

But Java conceptually has a lot of drawbacks that require the JVM to have screaming performance to compensate for. Almost everything being a "headered" object being probably the worst offender. Even a slightly worse Go compiler is probably well compensated-for by denser data structure layout in the operating memory.

Re: Golang – encoding/csv: Reading is slow

#20

Post by justincormack »

djur wrote: post #15

It seems pretty common for languages to start out with a relatively unoptimized CSV parser (if they have one at all) and then get a faster one contributed by the community once there's enough interest. Ruby had that happen with FasterCSV. The Java comparison here seems inapt, because it doesn't do as much as the other two. It's just a naive "split on commas" implementation that wouldn't handle quoted cells. Really, i…

It also seems to support UTF8, which is perhaps common now. (CSV doesnt tell you what encoding it is, in the old days it was 95% ASCII).
Post reply on HN