Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How fast can a BufferedReader read lines in Java?

lemire.me

Post reply on HN
Print view View original

51–60 of 90 posts

Re: How fast can a BufferedReader read lines in Java?

#51

Post by djhworld »

nullwasamistake wrote: post #10

Eh, he didn't use NIO. BufferedReader is an ancient Java relic. Like reading from STDIN in c, it's not made to be fast, it's there for convenience and backwards compatibility. Read a file using something like Vert.X, which is optimized for speed. I'm 100% confident it will be faster than the naive c approach

Do you have example of alternatives for the BufferedReader with the NIO APIs?

I do a lot of work with large GZIP that are read line by line using the standard IO (i.e. GzipInputStrem(FileInputStream)) etc) but your comment has really made me second guess my choice of doing that...

Re: How fast can a BufferedReader read lines in Java?

#52

Post by gpderetta »

tom_mellior wrote: post #31
zaphar wrote: post #27

Earlier quoted context omitted.

Alternatively if you took his class then you you probably learned that BufferedReader is not the most performant way to read lines from IO and probably learned the correct alternatives. If you work at some of the places java commonly gets used then you'll see BufferedReader all over the place. This article isn't a critique of Java. It's a warning about BufferedReader. If you do high performance java you'll be hiring…

If the article's point was that better alternatives exist, it should have made that point by mentioning those better alternatives and ideally benchmarking them as well. I agree with others that this particular article comes across as very lazy and not up to Lemire's usual standard.

That's not the point of the article. The point of the article is that many programs are in fact CPU bound, contrary to the often repeated claims and you do actually need to do optimization work to saturate your IO; even in lanuages considered fast, like c++ and java, the straightforward implementation is often suboptimal.

Re: How fast can a BufferedReader read lines in Java?

#53

Post by [deleted] »

d2mw wrote: post #5
adrianN wrote: post #2

So what's the reason for this? Is it maybe because of some unicode shenanigans? Java characters are 16bit iirc, and strings have some forty bytes of constant overhead.

I'm no Java ninja, but a few things jump out of https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c... : - at least one heap allocation for every line. After it finds the EOL it first uses 'new String' followed by '.toString() - the C++ version will almost certainly be backing on to memchr() behind the scenes, which will be using SIMD instructions where it makes sense (e.g. large enough scan size, probably t…

[deleted]

Re: How fast can a BufferedReader read lines in Java?

#54

Post by dionian »

Zach_the_Lizard wrote: post #24
nottorp wrote: post #3

Java is... java. I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision). Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.

> Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up Let us all hope that Project Valhalla, the effort to add value types to Java, comes to a swift completion. It would be very helpful in these sorts of scenarios. Though at this point I'd wonder (as you did) why I'm using Java in the firs…

I'm a little confused, trying to parse an entire file in memory instead of streaming it is one of the mistakes I might make in my first year or two of development. parsing 80k lines of CSV in java is pretty easy as long as you write your code to be efficient and release memory line by line

Re: How fast can a BufferedReader read lines in Java?

#55

Post by hinkley »

elFarto wrote: post #28

The first issue I can see with that code is it's not doing what he expects. He does this to read the file into a StringBuffer: bf.lines().forEach(s -> sb.append(s)); However, this ends up reading all the lines into one giant line, since the String's that lines() produces have the newline character stripped. This leads to the second lines() call to read a 23MB line (the file produced by gen.py). This is less than opti…

Is it possible he edited the code? I don’t see the append in the blog post.

Re: How fast can a BufferedReader read lines in Java?

#57

Post by pjmlp »

aristus wrote: post #12
znpy wrote: post #9

The post does nothing to explain how and why, it just throws a couple of outputs from a non specified machine and does no comparison. It has no baseline and no specs. For all I know, he could have got his 0.5 GB/sec on ab old Pentium II processor. There is no analysis. I am perplexed.

Lemire is one of the leading experts on string matching and the author of several core libraries you probably use every day. edit fine, so instead maybe click on the links in the post to see that this article is just one of a series. He's probably tired of copy-pasting the specs of his reference hardware (Skylake https://arxiv.org/pdf/1902.08318.pdf ) since all he's concerned about is the relative performance of diff…

Yet he misses the fact that Java isn't defined by a single implementation and the standard library reference doesn't dictate how each Java compliant implementation is required to provide BufferedReader behaviour.

Re: How fast can a BufferedReader read lines in Java?

#58

Post by pjmlp »

nottorp wrote: post #3

Java is... java. I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision). Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.

Android gives bad name to Java with its Dalvik and ART partial implementations.

Re: How fast can a BufferedReader read lines in Java?

#60

Post by soup10 »

elFarto wrote: post #28

The first issue I can see with that code is it's not doing what he expects. He does this to read the file into a StringBuffer: bf.lines().forEach(s -> sb.append(s)); However, this ends up reading all the lines into one giant line, since the String's that lines() produces have the newline character stripped. This leads to the second lines() call to read a 23MB line (the file produced by gen.py). This is less than opti…

this still does a fair amount of processing, curious what the throughput is with just straight reads into a char[] from FileReader
Post reply on HN