How fast can a BufferedReader read lines in Java? • Hacker News Board

Re: How fast can a BufferedReader read lines in Java?

#31

Post by tom_mellior » Sat, Jul 27, 2019, 3:40 PM UTC

zaphar wrote: post #27

jnordwick wrote: post #19

Impressive bad for a professor. Even worse for someone who says on their page "I like crazy fast code." He certainly doesn't seem to know how to produce it (or if you believe he does - he's being intentionally obtuse or lazy). Using the simplest, slowest, oldest method to read as file line by line from a time when there wasn't such a thing a high performance Java (trading systems are built in Java that rival c++ perf…

Alternatively if you took his class then you you probably learned that BufferedReader is not the most performant way to read lines from IO and probably learned the correct alternatives. If you work at some of the places java commonly gets used then you'll see BufferedReader all over the place. This article isn't a critique of Java. It's a warning about BufferedReader. If you do high performance java you'll be hiring…

If the article's point was that better alternatives exist, it should have made that point by mentioning those better alternatives and ideally benchmarking them as well.

I agree with others that this particular article comes across as very lazy and not up to Lemire's usual standard.

Re: How fast can a BufferedReader read lines in Java?

#32

Post by Zach_the_Lizard » Sat, Jul 27, 2019, 3:46 PM UTC

adrianN wrote: post #17

TheChaplain wrote: post #13

Earlier quoted context omitted.

Myself I've worked with gigabyte sized CSV files without issues, so it was likely your implementation rather than the fault of the Java language.

Did you do it in 128 megs of RAM?

Depending on what needs to be done with the CSV files, it's very possible to do it in 128MB of RAM. For example, if we need to read the rows, transform them a bit, and then write to another file, we can read up to N rows, transform them, and then write them. That should result in bounded memory consumption because only up to N rows need to be kept. Similar strategies are possible if the rows are used as input to an ETL job, calling a Web service with the results of parsing the file, etc.

Editing a file gets trickier, though it's not impossible. Maybe using a [piece table](https://en.wikipedia.org/wiki/Piece_table) plus some smart buffering the file can keep memory consumption below some constant, letting it function for large files, but with the downside of lower performance for files larger than whatever the constant is?

Re: How fast can a BufferedReader read lines in Java?

#33

Post by ubu7737 » Sat, Jul 27, 2019, 3:53 PM UTC

This is absurd, the original platform libraries do not account for the fastest use-cases in any specialized IO case.

Java NIO channel should have been used for this. It was demonstrated back in the early 2000s with the "Grand Canyon" demo achieving very good throughput for its time, and it's still the gold standard.

Re: How fast can a BufferedReader read lines in Java?

#34

Post by [deleted] » Sat, Jul 27, 2019, 3:56 PM UTC

derefr wrote: post #4

I don't know what Java's BufferedReader is doing, but it's probably not the optimal thing in terms of IO throughput. I would blame the algorithm long before blaming anything inherent about the JVM. Erlang is another language where "naive" IO is kind of slow. https://github.com/bbense/beatwc/ is a project someone did to test various methods of doing IO in Erlang/Elixir, and their performance for a line-counting task,…

[deleted]

Re: How fast can a BufferedReader read lines in Java?

#35

Post by cozos » Sat, Jul 27, 2019, 3:57 PM UTC

jnordwick wrote: post #19

Impressive bad for a professor. Even worse for someone who says on their page "I like crazy fast code." He certainly doesn't seem to know how to produce it (or if you believe he does - he's being intentionally obtuse or lazy). Using the simplest, slowest, oldest method to read as file line by line from a time when there wasn't such a thing a high performance Java (trading systems are built in Java that rival c++ perf…

As someone who has only known BufferedReader for his entire life, I found this comment to be extremely informative. Thanks!

Re: How fast can a BufferedReader read lines in Java?

#36

Post by ubu7737 » Sat, Jul 27, 2019, 3:58 PM UTC

derefr wrote: post #4

I don't know what Java's BufferedReader is doing, but it's probably not the optimal thing in terms of IO throughput. I would blame the algorithm long before blaming anything inherent about the JVM. Erlang is another language where "naive" IO is kind of slow. https://github.com/bbense/beatwc/ is a project someone did to test various methods of doing IO in Erlang/Elixir, and their performance for a line-counting task,…

Java NIO does exactly what you describe. IO is chunked optimally, the entire buffer is pulled, the API exposes a select() mechanism just like any libc, and the user is expected to frame its own received data.

edit: Lemire is showing a lack of what Fowler describes as "mechanical sympathy".

https://martinfowler.com/articles/lmax.html#QueuesAndTheirLa...

Re: How fast can a BufferedReader read lines in Java?

#37

Post by micmus » Sat, Jul 27, 2019, 4:00 PM UTC

derefr wrote: post #11

derefr wrote: post #4

I don't know what Java's BufferedReader is doing, but it's probably not the optimal thing in terms of IO throughput. I would blame the algorithm long before blaming anything inherent about the JVM. Erlang is another language where "naive" IO is kind of slow. https://github.com/bbense/beatwc/ is a project someone did to test various methods of doing IO in Erlang/Elixir, and their performance for a line-counting task,…

I'm reminded to add, in the vein of the author's complaint, that there is a similar ridiculousness in Erlang land, that cannot be circumnavigated so easily: reading/writing to zlib-compressed files using Erlang's file:open(..., [compressed]) option—or generating/parsing zlib-compressed ETF binaries using erlang:binary_to_term(..., [compressed]))—holds [the moral equivalent of†] a global lock . Only one process can be…

That's probably not true in the recent versions - in OTP 20 the zlib integration was reworked and is now based on NIFs (similar to Java's JNI) instead of port drivers.

Re: How fast can a BufferedReader read lines in Java?

#38

Post by tawy12345 » Sat, Jul 27, 2019, 4:10 PM UTC

I'm amazed at how upset some commenters are about a blog post that did a toy experiment and didn't actually make any strong claims.

I'm actually a stickler about good benchmarks - it riles me when people draw sweeping conclusions from poorly-designed experiments. Lemire is actually one of the good ones. If you want something more fully developed than a blog post, read one of his papers.

I personally really enjoy his blog because of this - he's good at picking interesting exploratory experiments that provide some insight, without trying to over-generalize from the results. If you read his conclusion, the point is that there is a good probability that even relatively simple programs are CPU-bound. His experiment supports that point. My experience also matches that - I've seen a lot of data processing code that could be I/O bound in theory (i.e. a perfect implementation could max out CPU or network) but is CPU bound in practice. Usually because of string manipulation, regexes, or any number of other things.

> This is not the best that Java can do: Java can ingest data much faster. However, my results suggest that on modern systems, Java file parsing might be frequently processor-bound, as opposed to system bound. That is, you can buy much better disks and network cards, and your system won’t go any faster. Unless, of course, you have really good Java engineers.

Re: How fast can a BufferedReader read lines in Java?

#39

Post by jnordwick » Sat, Jul 27, 2019, 4:33 PM UTC

tawy12345 wrote: post #38

I'm amazed at how upset some commenters are about a blog post that did a toy experiment and didn't actually make any strong claims. I'm actually a stickler about good benchmarks - it riles me when people draw sweeping conclusions from poorly-designed experiments. Lemire is actually one of the good ones. If you want something more fully developed than a blog post, read one of his papers. I personally really enjoy his…

I can make a lot of things cpu bound with crappy implementation. I'm not going to write a blog post about any of them.

Re: How fast can a BufferedReader read lines in Java?

#40

Post by ricardobeat » Sat, Jul 27, 2019, 4:41 PM UTC

elFarto wrote: post #28

The first issue I can see with that code is it's not doing what he expects. He does this to read the file into a StringBuffer: bf.lines().forEach(s -> sb.append(s)); However, this ends up reading all the lines into one giant line, since the String's that lines() produces have the newline character stripped. This leads to the second lines() call to read a 23MB line (the file produced by gen.py). This is less than opti…

That's a pretty big error if you're correct. What does it say about the language when a CS professor falls for this on a 40 line file?