Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Golang – encoding/csv: Reading is slow

github.com

Post reply on HN
Print view View original

51–60 of 102 posts

Re: Golang – encoding/csv: Reading is slow

#51

Post by elmigranto »

deno wrote: post #46

Better than node.js import * as csv from 'csv-parse'; import * as fs from 'fs'; type Line = [string,string,string,string,string,string]; const parser = new csv.Parser({}); parser.on('data', (line: Line) => { if (line[0] === '42') { console.dir(line); } }); fs.createReadStream('mock_data.csv').pipe(parser); $ /usr/bin/time node parse_csv.js 43.61user 0.85system 0:45.61elapsed 97%CPU (0avgtext+0avgdata 60076maxresident…

Probably has to do with using `try/catch` whis is not omptimized by V8. Different parser is 10 times faster on my machine.

https://www.npmjs.com/package/csv-parser

Edit: `fast-csv` seems to be using a lot of `RegExp`s on each iteration which can't be that fast compared to csv-parser which seems to simply go over each symbol (state machine?).

Re: Golang – encoding/csv: Reading is slow

#52

Post by geofft »

masklinn wrote: post #32
coldtea wrote: post #23

Earlier quoted context omitted.

All 3 support UTF8. The difference with the Go implementation is the mostly useless capability to have a utf-8 multibyte item as the delimiter.

1. there's nothing useless about it 2. the Python 3 CSV library supports arbitrary codepoints as delimiter, quote character and escape character (if applicable)

> 1. there's nothing useless about it

Have you ever seen a "C"SV with a multibyte sequence as a delimiter? I haven't.

Even if such a thing exists, the feature is of negative utility if it slows down CSV parsing for everyone else. If you must, write two implementations, and use the slow path if your delimiter is multibyte.

Re: Golang – encoding/csv: Reading is slow

#53

Post by deno »

masklinn wrote: post #49
deno wrote: post #46

Better than node.js import * as csv from 'csv-parse'; import * as fs from 'fs'; type Line = [string,string,string,string,string,string]; const parser = new csv.Parser({}); parser.on('data', (line: Line) => { if (line[0] === '42') { console.dir(line); } }); fs.createReadStream('mock_data.csv').pipe(parser); $ /usr/bin/time node parse_csv.js 43.61user 0.85system 0:45.61elapsed 97%CPU (0avgtext+0avgdata 60076maxresident…

csv-parse is hardly the only CSV parser for node, and it is by far the slowest: https://github.com/phihag/csv-speedtest (csv2json depends on csv-parse, so it's unsurprising that it's even slower)

I chose the most popular one on npm because Go and Python are using stdlib.

Re: Golang – encoding/csv: Reading is slow

#54

Post by lcarlson »

This may be a bit off topic but I've found sqlite to be quite a powerful csv parser. Once posted you can manipulate the data in lots of ways. When you're working with reports that need to get back into some sort of table format, it's very intuitive and easy for SQL people.

Re: Golang – encoding/csv: Reading is slow

#55

Post by elmigranto »

deno wrote: post #53
masklinn wrote: post #49

Earlier quoted context omitted.

csv-parse is hardly the only CSV parser for node, and it is by far the slowest: https://github.com/phihag/csv-speedtest (csv2json depends on csv-parse, so it's unsurprising that it's even slower)

I chose the most popular one on npm because Go and Python are using stdlib.

But you still wrote "faster than node.js" and not "faster that most popular npm module" (which aren't always of a great quality or performance-oriented).

Re: Golang – encoding/csv: Reading is slow

#56

Post by heroprotagonist »

halayli wrote: post #47
djur wrote: post #15

It seems pretty common for languages to start out with a relatively unoptimized CSV parser (if they have one at all) and then get a faster one contributed by the community once there's enough interest. Ruby had that happen with FasterCSV. The Java comparison here seems inapt, because it doesn't do as much as the other two. It's just a naive "split on commas" implementation that wouldn't handle quoted cells. Really, i…

I feel you're trying to defend go without much objectivity. Such performance gap needs to be addressed properly instead of saying it's pretty good already. It doesn't sound right if go takes 5 hours to finish csv parsing job while Python takes 2.5 hrs.

Well, you are free to address it if it bothers you. The typical response to this of _I don't want to_ or _I shouldn't have to_ seems a bit naive when working with open source projects. The issue has only been on the tracker for two weeks, and it has the 'HelpWanted' tag, so it's not like they're opposed to improving the speed here.

If you're going to throw out specific numbers, you should probably get them, or at least the ratios, correct.

Numbers from the tracker are:

Go: avg 1.489 secs Python: avg 0.933 secs

If you'd like to test this on a really large dataset to come up with how long it would take for Python to perform the same operation when Go requires 5 hours, that might be a bit more useful. If we just look at the available data, then the _extrapolation_ for Python would not be 2.5 hours. There's still a gap, but there's no need to exaggerate.

Re: Golang – encoding/csv: Reading is slow

#58

Post by deno »

elmigranto wrote: post #51
deno wrote: post #46

Better than node.js import * as csv from 'csv-parse'; import * as fs from 'fs'; type Line = [string,string,string,string,string,string]; const parser = new csv.Parser({}); parser.on('data', (line: Line) => { if (line[0] === '42') { console.dir(line); } }); fs.createReadStream('mock_data.csv').pipe(parser); $ /usr/bin/time node parse_csv.js 43.61user 0.85system 0:45.61elapsed 97%CPU (0avgtext+0avgdata 60076maxresident…

Probably has to do with using `try/catch` whis is not omptimized by V8. Different parser is 10 times faster on my machine. https://www.npmjs.com/package/csv-parser Edit: `fast-csv` seems to be using a lot of `RegExp`s on each iteration which can't be that fast compared to csv-parser which seems to simply go over each symbol (state machine?).

    4.95user 0.19system 0:05.22elapsed 98%CPU (0avgtext+0avgdata 29704maxresident)k
A lot better but that’s still 5× slower than Python.

Re: Golang – encoding/csv: Reading is slow

#59

Post by deno »

elmigranto wrote: post #55
deno wrote: post #53

Earlier quoted context omitted.

I chose the most popular one on npm because Go and Python are using stdlib.

But you still wrote "faster than node.js" and not "faster that most popular npm module" (which aren't always of a great quality or performance-oriented).

Yup[1]. Sorry!

[1] https://meta.wikimedia.org/wiki/Cunningham%27s_Law

Re: Golang – encoding/csv: Reading is slow

#60

Post by sorokod »

piinbinary wrote: post #33
tmaly wrote: post #29

I wrote my own in Go that is blazing fast using bytes. I know the data is ascii so I was able to use that to my advantage.

I did the same [0]. It runs almost as fast as the Java implementation. I'd be interested to see how yours works if you are willing to share it. Edit: Plus one that is ~2x faster than Java by avoiding allocations [1]. [0] https://gist.github.com/jmikkola/6ac96ad6d6f66e772c33ec41ed2... [1] https://gist.github.com/jmikkola/7ded8392226b7659c881f5540be...

Note that internally Java represents strings as utf-16
Post reply on HN