How fast can a BufferedReader read lines in Java?
1–10 of 90 posts
Re: How fast can a BufferedReader read lines in Java?
#2So what's the reason for this? Is it maybe because of some unicode shenanigans? Java characters are 16bit iirc, and strings have some forty bytes of constant overhead.
Re: How fast can a BufferedReader read lines in Java?
#3Java is... java.
I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision).
Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.
Re: How fast can a BufferedReader read lines in Java?
#4I don't know what Java's BufferedReader is doing, but it's probably not the optimal thing in terms of IO throughput. I would blame the algorithm long before blaming anything inherent about the JVM.
Erlang is another language where "naive" IO is kind of slow. https://github.com/bbense/beatwc/ is a project someone did to test various methods of doing IO in Erlang/Elixir, and their performance for a line-counting task, relative to the Unix wc(1) command.
It's interesting to see which approaches are faster. Yes, parallelism gains you a bit, but a much larger win comes from avoiding the stutter-stop effect of cutting the read buffer off whenever you hit a newline. Instead, the read buffer should be the same size as your IO source's optimal read-chunk size (a disk block; a TCP huge packet), and you should grab a whole buffer-ful of lines at a time, do a pattern-matching binary scan to collect all the indices of the newlines, and then use those indices to part the buffer out as slice references.
This achieves quite a dramatic speedup, since most of the time you don't need movable copies of the lines, and can copy the line (or more likely just part of it) yourself when you need to hold onto it.
This approach is probably also already built in to Java's "better" IO libraries, like NIO.
Re: How fast can a BufferedReader read lines in Java?
#5So what's the reason for this? Is it maybe because of some unicode shenanigans? Java characters are 16bit iirc, and strings have some forty bytes of constant overhead.
I'm no Java ninja, but a few things jump out of https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c... :
- at least one heap allocation for every line. After it finds the EOL it first uses 'new String' followed by '.toString()
- the C++ version will almost certainly be backing on to memchr() behind the scenes, which will be using SIMD instructions where it makes sense (e.g. large enough scan size, probably true in this case). the Java version is a manual bytewise-coded loop.
- the C++ version is reusing its output buffer, no reallocations assuming the same string length or less
No idea about encodings in Java, maybe that is playing a role too
Re: How fast can a BufferedReader read lines in Java?
#6So what's the reason for this? Is it maybe because of some unicode shenanigans? Java characters are 16bit iirc, and strings have some forty bytes of constant overhead.
I'm no Java ninja, but a few things jump out of https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c... : - at least one heap allocation for every line. After it finds the EOL it first uses 'new String' followed by '.toString() - the C++ version will almost certainly be backing on to memchr() behind the scenes, which will be using SIMD instructions where it makes sense (e.g. large enough scan size, probably t…
Yes the reuse of the buffer in C++ seems likely and would probably explain a large part of the difference, but I don't know enough of the std::string implementation to be sure about that.
Re: How fast can a BufferedReader read lines in Java?
#7Java is... java. I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision). Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.
Why didn't you just write it using the NDK?
Re: How fast can a BufferedReader read lines in Java?
#8So what's the reason for this? Is it maybe because of some unicode shenanigans? Java characters are 16bit iirc, and strings have some forty bytes of constant overhead.
[deleted]
Re: How fast can a BufferedReader read lines in Java?
#9The post does nothing to explain how and why, it just throws a couple of outputs from a non specified machine and does no comparison.
It has no baseline and no specs. For all I know, he could have got his 0.5 GB/sec on ab old Pentium II processor.
There is no analysis.
I am perplexed.
Re: How fast can a BufferedReader read lines in Java?
#10Eh, he didn't use NIO. BufferedReader is an ancient Java relic. Like reading from STDIN in c, it's not made to be fast, it's there for convenience and backwards compatibility.
Read a file using something like Vert.X, which is optimized for speed. I'm 100% confident it will be faster than the naive c approach