Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How fast can a BufferedReader read lines in Java?

lemire.me

Post reply on HN
Print view View original

11–20 of 90 posts

Re: How fast can a BufferedReader read lines in Java?

#11

Post by derefr »

derefr wrote: post #4

I don't know what Java's BufferedReader is doing, but it's probably not the optimal thing in terms of IO throughput. I would blame the algorithm long before blaming anything inherent about the JVM. Erlang is another language where "naive" IO is kind of slow. https://github.com/bbense/beatwc/ is a project someone did to test various methods of doing IO in Erlang/Elixir, and their performance for a line-counting task,…

I'm reminded to add, in the vein of the author's complaint, that there is a similar ridiculousness in Erlang land, that cannot be circumnavigated so easily: reading/writing to zlib-compressed files using Erlang's file:open(..., [compressed]) option—or generating/parsing zlib-compressed ETF binaries using erlang:binary_to_term(..., [compressed]))—holds [the moral equivalent of†] a global lock. Only one process can be zlib-compressing or zlib-decompressing a chunk of data at once, no matter how many cores your system has.

This means that, even when your data set compresses so well that you'd theoretically gain a ton of speed by having the data streamed from disk compressed, and then decompressed during parsing—this doesn't apply in practice, since you're introducing an artificial bottleneck in your IO reads.

I'm not actually sure if this is a bug in Erlang, per se, or if it's just the intended behavior and compressed file IO was never intended to be used for performance, only for e.g. embedded devices with tiny ROMs.

(If people here think it's a bug, I'll probably go to the effort at some point to profile the performance impact and submit it as a bug on https://bugs.erlang.org.)

† What it's actually doing, is that all zlib compression/decompression passes get sent to a zlib "port driver" as messages. Port drivers can handle multiple requests in-flight at once (they're the in-process equivalent to sockets—each Erlang process's port against the port-driver is its own "connection") but the zlib port driver shim is coded to expose zlib as a single-threaded, blocking, request-response style of server, rather than one that accepts connections in parallel and instantiates a separate zlib context for each separate connection it receives.

Re: How fast can a BufferedReader read lines in Java?

#12

Post by aristus »

znpy wrote: post #9

The post does nothing to explain how and why, it just throws a couple of outputs from a non specified machine and does no comparison. It has no baseline and no specs. For all I know, he could have got his 0.5 GB/sec on ab old Pentium II processor. There is no analysis. I am perplexed.

Lemire is one of the leading experts on string matching and the author of several core libraries you probably use every day.

edit fine, so instead maybe click on the links in the post to see that this article is just one of a series. He's probably tired of copy-pasting the specs of his reference hardware (Skylake https://arxiv.org/pdf/1902.08318.pdf) since all he's concerned about is the relative performance of different software.

There is a difference between "I'm being dumb because I don't know what I'm doing" and "I'm being lazy because I've done it 1,000 times and the target audience knows what I mean".

Re: How fast can a BufferedReader read lines in Java?

#13

Post by TheChaplain »

nottorp wrote: post #3

Java is... java. I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision). Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.

Myself I've worked with gigabyte sized CSV files without issues, so it was likely your implementation rather than the fault of the Java language.

Re: How fast can a BufferedReader read lines in Java?

#14

Post by [deleted] »

nottorp wrote: post #3

Java is... java. I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision). Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.

[deleted]

Re: How fast can a BufferedReader read lines in Java?

#15

Post by adrianN »

aristus wrote: post #12
znpy wrote: post #9

The post does nothing to explain how and why, it just throws a couple of outputs from a non specified machine and does no comparison. It has no baseline and no specs. For all I know, he could have got his 0.5 GB/sec on ab old Pentium II processor. There is no analysis. I am perplexed.

Lemire is one of the leading experts on string matching and the author of several core libraries you probably use every day. edit fine, so instead maybe click on the links in the post to see that this article is just one of a series. He's probably tired of copy-pasting the specs of his reference hardware (Skylake https://arxiv.org/pdf/1902.08318.pdf ) since all he's concerned about is the relative performance of diff…

Then they should know enough to give at least some theories that can explain the difference and tell us something more about the test setup.

Re: How fast can a BufferedReader read lines in Java?

#16

Post by saagarjha »

aristus wrote: post #12
znpy wrote: post #9

The post does nothing to explain how and why, it just throws a couple of outputs from a non specified machine and does no comparison. It has no baseline and no specs. For all I know, he could have got his 0.5 GB/sec on ab old Pentium II processor. There is no analysis. I am perplexed.

Lemire is one of the leading experts on string matching and the author of several core libraries you probably use every day. edit fine, so instead maybe click on the links in the post to see that this article is just one of a series. He's probably tired of copy-pasting the specs of his reference hardware (Skylake https://arxiv.org/pdf/1902.08318.pdf ) since all he's concerned about is the relative performance of diff…

That doesn’t excuse him from needing to describe the hardware he ran the benchmark on.

Re: How fast can a BufferedReader read lines in Java?

#17

Post by adrianN »

TheChaplain wrote: post #13
nottorp wrote: post #3

Java is... java. I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision). Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.

Myself I've worked with gigabyte sized CSV files without issues, so it was likely your implementation rather than the fault of the Java language.

Did you do it in 128 megs of RAM?

Re: How fast can a BufferedReader read lines in Java?

#18

Post by lanna »

aristus wrote: post #12
znpy wrote: post #9

The post does nothing to explain how and why, it just throws a couple of outputs from a non specified machine and does no comparison. It has no baseline and no specs. For all I know, he could have got his 0.5 GB/sec on ab old Pentium II processor. There is no analysis. I am perplexed.

Lemire is one of the leading experts on string matching and the author of several core libraries you probably use every day. edit fine, so instead maybe click on the links in the post to see that this article is just one of a series. He's probably tired of copy-pasting the specs of his reference hardware (Skylake https://arxiv.org/pdf/1902.08318.pdf ) since all he's concerned about is the relative performance of diff…

https://en.wikipedia.org/wiki/Argument_from_authority

Re: How fast can a BufferedReader read lines in Java?

#19

Post by jnordwick »

Impressive bad for a professor. Even worse for someone who says on their page "I like crazy fast code." He certainly doesn't seem to know how to produce it (or if you believe he does - he's being intentionally obtuse or lazy).

Using the simplest, slowest, oldest method to read as file line by line from a time when there wasn't such a thing a high performance Java (trading systems are built in Java that rival c++ performance), then complaining about the performance.

It is probably mostly in the GC and copying. Each call to readline is going to new off a string and copy into it. Also the conversion from bytes to string needs to go through a Unicode conversion and check with probably another copy in there somewhere. I wouldn't be surprised if bufferedreader did some more allocations and copying too. Some of the Java libraries especially from early Java aren't implemented for performance but for simplicity. Java probably has the most readable standard library of any language.

He should reimplement it as a byte buffer with nio channel. For all of Java's good points, it's standard library can really suck for performance ever though hotspot can produce excellent code and it doesn't need to be that way. I've seen huge systems that never GC or have a scheduled GC once a week.

If you take his class, go ask the university for your money back. Hopefully her puts more effort into thanks, but I doubt it.

Re: How fast can a BufferedReader read lines in Java?

#20

Post by VRay »

TheChaplain wrote: post #13
nottorp wrote: post #3

Java is... java. I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision). Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.

Myself I've worked with gigabyte sized CSV files without issues, so it was likely your implementation rather than the fault of the Java language.

Googles "parse csv Java"

Copies and pasted top answer that buffers the whole thing into the heap before parsing

Blames Java/hardware when this doesn't scale

Post reply on HN