Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Golang – encoding/csv: Reading is slow

github.com

Post reply on HN
Print view View original

91–100 of 102 posts

Re: Golang – encoding/csv: Reading is slow

#91

Post by weberc2 »

twotwotwo wrote: post #48

As someone notes on the bug, if you were rolling your own, there are some other things you could do--return a [][]byte that's a pointer to its internal buffer, only usable until the next row is read. Making a version of encoding/csv that retains most of its features (custom delimiters, handling backslashes and quoting and \r) but streams like that would be a fun open source project for someone who likes Making Things…

That guy here. I'm also interested in rolling a version that only supports standard delimiters so I can forego rune parsing. Rune parsing accounts for about 30% of the processing; not sure how much a bytes implementation could save, but I'm hopeful.

Re: Golang – encoding/csv: Reading is slow

#92

Post by mook »

geofft wrote: post #34
masklinn wrote: post #24

Earlier quoted context omitted.

ignore this, I somehow missed that the poster specifically mentioned Python 3, which does have an encoding-aware CSV module. ~~It's also unclear which version of Python is being used, the Python 2 csv module is byte-based and encoding-unaware which can lead to unexpected behaviours.~~ Go's CSV package apparently only does UTF-8, and suggestions for speeding it up in the tracker is to just remove that and work on raw…

> suggestions for speeding it up in the tracker is to just remove that and work on raw bytes (FFS) This is valid, because UTF-8 was designed to make this valid. The UTF-8 encoding of a comma, 0x2C (also the ASCII encoding of a comma), does not appear as a part of any other UTF-8 encodings. Same with the UTF-8 encoding of the double quote, 0x22. So scanning for 0x22 and 0x2C bytes, without stopping to decode other UTF…

For string comparisons, wouldn't case sensitivity and normalization be an issue in some contexts?

Re: Golang – encoding/csv: Reading is slow

#93

Post by weberc2 »

deno wrote: post #46

Better than node.js import * as csv from 'csv-parse'; import * as fs from 'fs'; type Line = [string,string,string,string,string,string]; const parser = new csv.Parser({}); parser.on('data', (line: Line) => { if (line[0] === '42') { console.dir(line); } }); fs.createReadStream('mock_data.csv').pipe(parser); $ /usr/bin/time node parse_csv.js 43.61user 0.85system 0:45.61elapsed 97%CPU (0avgtext+0avgdata 60076maxresident…

You don't need to drop quote handling in order to stream, but handling quotes is a pain.

Re: Golang – encoding/csv: Reading is slow

#94

Post by burntsushi »

weberc2 wrote: post #75
geofft wrote: post #44

Earlier quoted context omitted.

The big things are related to string length not matching byte count. strlen() is O(n) because you have to see how many sequences are actually in the string. More than that, splitting/slicing/indexing a string based on byte offsets doesn't work. For a 100-byte ASCII string, you're guaranteed that you can split it into two 50-byte strings and things will still work: you can output them separately, you can get the total…

To clarify, Letting the length of a UTF-8 string in Go is O(1); it's computed and stored on the string header at creation.

To clarify even more: that length is the number of bytes (or UTF-8 code units) in the string. It doesn't corresponding to the number of characters (which one may either consider to be Unicode codepoints, or more technically correct, Unicode grapheme clusters).

If you want to count the number of codepoints in a string (called "rune" in Go), then you need to do so explicitly: https://golang.org/pkg/unicode/utf8/#RuneCountInString

Re: Golang – encoding/csv: Reading is slow

#95

Post by masklinn »

deno wrote: post #61
masklinn wrote: post #49

Earlier quoted context omitted.

csv-parse is hardly the only CSV parser for node, and it is by far the slowest: https://github.com/phihag/csv-speedtest (csv2json depends on csv-parse, so it's unsurprising that it's even slower)

The fastest streaming example on that list (csv-parser) is still 5× slower than Python.

Which I'd assume has to do with the overhead of dispatching a ton of asynchronous events for relatively little parsed data rather than the intrinsic speed of node, the fastest synchronous parsers of the list are about on-par with Python.

Re: Golang – encoding/csv: Reading is slow

#96

Post by masklinn »

petters wrote: post #37

Any CSV reader should be limited only be disk access right? I wrapped together a C++ program solving this problem and got 0.124 seconds. But that does not do quotations etc.

> Any CSV reader should be limited only be disk access right?

Depends on the speed of your storage subsystem. If you're working from RAM or from a fast PCIe SSD, you'll probably bottleneck in the encoding validation and actual parsing.

Re: Golang – encoding/csv: Reading is slow

#97

Post by weberc2 »

burntsushi wrote: post #94
weberc2 wrote: post #75

Earlier quoted context omitted.

To clarify, Letting the length of a UTF-8 string in Go is O(1); it's computed and stored on the string header at creation.

To clarify even more: that length is the number of bytes (or UTF-8 code units) in the string. It doesn't corresponding to the number of characters (which one may either consider to be Unicode codepoints, or more technically correct, Unicode grapheme clusters). If you want to count the number of codepoints in a string (called "rune" in Go), then you need to do so explicitly: https://golang.org/pkg/unicode/utf8/#RuneCo…

Touché

Re: Golang – encoding/csv: Reading is slow

#98

Post by [deleted] »

burntsushi wrote: post #94
weberc2 wrote: post #75

Earlier quoted context omitted.

To clarify, Letting the length of a UTF-8 string in Go is O(1); it's computed and stored on the string header at creation.

To clarify even more: that length is the number of bytes (or UTF-8 code units) in the string. It doesn't corresponding to the number of characters (which one may either consider to be Unicode codepoints, or more technically correct, Unicode grapheme clusters). If you want to count the number of codepoints in a string (called "rune" in Go), then you need to do so explicitly: https://golang.org/pkg/unicode/utf8/#RuneCo…

[deleted]

Re: Golang – encoding/csv: Reading is slow

#99

Post by [deleted] »

burntsushi wrote: post #94
weberc2 wrote: post #75

Earlier quoted context omitted.

To clarify, Letting the length of a UTF-8 string in Go is O(1); it's computed and stored on the string header at creation.

To clarify even more: that length is the number of bytes (or UTF-8 code units) in the string. It doesn't corresponding to the number of characters (which one may either consider to be Unicode codepoints, or more technically correct, Unicode grapheme clusters). If you want to count the number of codepoints in a string (called "rune" in Go), then you need to do so explicitly: https://golang.org/pkg/unicode/utf8/#RuneCo…

[deleted]

Re: Golang – encoding/csv: Reading is slow

#100

Post by sp332 »

ktRolster wrote: post #35
wongarsu wrote: post #25

Earlier quoted context omitted.

How would a CSV parser even break UTF8 encoding (by accident)? All CSV control characters (comma, doubleqote and newline) map to the same codepoints in ASCII and UTF8, and no non-ASCII UTF8 character uses any ASCII codepoint in it's encoding.

I've seen one break because of the byte-order marker that sometimes gets added to UTF-8. I don't remember the details of why that broke it, just remember that it worked fine on everything except that.

UTF-8 doesn't have a "byte order", so I thought it wouldn't have a byte-order mark. But apparently some software adds one anyway. https://en.wikipedia.org/wiki/UTF-8#Byte_order_mark
Post reply on HN