So, where's the Perl (+ CPAN...) version for comparison?
I guess "Nobody cares about your dead religion" :-)
Still, it would probably be faster, at the expense of being unreadable.
Golang – encoding/csv: Reading is slow
81–90 of 102 posts
Re: Golang – encoding/csv: Reading is slow
#82As someone notes on the bug, if you were rolling your own, there are some other things you could do--return a [][]byte that's a pointer to its internal buffer, only usable until the next row is read. Making a version of encoding/csv that retains most of its features (custom delimiters, handling backslashes and quoting and \r) but streams like that would be a fun open source project for someone who likes Making Things…
I did this a couple of months ago and got a >5x speedup. It's at the expense of dropping quoting though, so no commas or newlines can be in the input data.
Re: Golang – encoding/csv: Reading is slow
#83Python's csv module uses an internal module _csv which is written in C. So I'm not sure it's all that surprising that a Go implementation is a bit slower.
But not sure if it matters. Go is free to use a C module to load csv files as well. It can use assembly or other tricks as well perhaps.
There's a performance cost to calling between c and Go, and sharing memory between the two makes for hard to predict GC behavior. I doubt it would be faster than a pure Go implementation.
Re: Golang – encoding/csv: Reading is slow
#84Earlier quoted context omitted.
I thought it is rather part of the solution/answer to the issue. It is expected to be slower, so it's not necessarily broken. Kind of it's a feature, not a bug.
If one language is 50% slower in file reading and a simple state mashine that iterates over an array, that's not a feature. Either the implementation or the compiler is lacking some optimization.
The implementation is suboptimal, and Go has some overhead compared to C.
Re: Golang – encoding/csv: Reading is slow
#85Earlier quoted context omitted.
But Java conceptually has a lot of drawbacks that require the JVM to have screaming performance to compensate for. Almost everything being a "headered" object being probably the worst offender. Even a slightly worse Go compiler is probably well compensated-for by denser data structure layout in the operating memory.
> Java conceptually has a lot of drawbacks In what sense is an articulated object a "conceptual drawback"? It is a richer object model and SMI and friends had the engineering chops to makes it highly performant.
What do you mean by "a richer object model". The drawbacks are principally that the programmer has less control over allocs and layout.
Re: Golang – encoding/csv: Reading is slow
#86Earlier quoted context omitted.
All 3 support UTF8. The difference with the Go implementation is the mostly useless capability to have a utf-8 multibyte item as the delimiter.
1. there's nothing useless about it 2. the Python 3 CSV library supports arbitrary codepoints as delimiter, quote character and escape character (if applicable)
>1. there's nothing useless about it
Opportunity cost. It slows down the parsing for support of something that nobody has ever seen in the wild (not to mention it doesn't even match the name of the format, but let's get past that since we already use ; | and others).
Plus I can't even imagine a use case that would make it a good idea to use that over a simpler delimiter, or even the special purpose ASCII delimiter character. Can you?
Re: Golang – encoding/csv: Reading is slow
#87On a related note, also the Go stdlib regex package is pretty naive and imperformant compared to a full blown and modern backtracking PCRE implementation (at 1/10 the LOC and complexity) - same thing goes for the reflection based JSON package (which is still kinda "fast enough"). The focus wasn't so much on performance but on initial completeness, good interface, versatility, clarity and simplicity - with faster or m…
On differing opinions: I prefer having a Katana, an automatic shotgun, and a spy drone, and leaving the pocket knife at home.
In other words....C/C++ :) I just try not to blow off my leg.
Re: Golang – encoding/csv: Reading is slow
#88The Java code is not a CSV parser. I added the results of using Apache Commons CSV to the GitHub thread. After using that, Python was actually by far the fastest. Hooray for performance sensitive code in C :)
FYI, I later went back and ran PyPy, which implements csv in pure Python. Even including start-up time, it was nearly at CPython speed.
Re: Golang – encoding/csv: Reading is slow
#89On the CSV Game benchmark, the Go csv reader is around 10x slower than the fastest: https://bitbucket.org/ewanhiggs/csv-game
Re: Golang – encoding/csv: Reading is slow
#90Effectively one is comparing library performance here and not language performance. Granted, that line can get very blurry indeed, but in this case this says very little about golang the language and far more about a current implementation of one of the golang libraries.
It is kind of both; go doesn't allow some approaches in native go code that can make it slower than other languages. (I love go, but that is my experience.) In this case the choice to use utf-8 everywhere, including in the csv delimiters, is making it slower.
This isn't what makes CSV parsing slow, there's nothing about Go that requires you to deal in UTF8.