Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How fast can a BufferedReader read lines in Java?

lemire.me

Post reply on HN
Print view View original

81–90 of 90 posts

Re: How fast can a BufferedReader read lines in Java?

#81

Post by pvg »

ricardobeat wrote: post #78
bhaak wrote: post #77

Earlier quoted context omitted.

There are several simple data transformation steps. There's nothing inherently wrong with these steps. The compiler can't read your mind and guess what you wanted to do. How would this be possible to be prevented with any programming language, existing or hypothetical? Even if the meta information that the appended string doesn't contain any newlines would be passed on, the second call to lines() would rather use is…

To give an example: if the lines() method returned instances of a Line class, the string buffer and it's append() method would be aware and still create a collection of lines instead of one long string. Not necessarily nice and would probably create issues in many other interfaces, but problems like this _can_ be solved with careful language and API design. Note that his end goal was to 'read lines' and not use any p…

That would be a very clunky approach to put it mildly - the explicit design intent of BufferedReader is to let you read lines without caring what the line separator is - it's a lossy transformation. You also can't subclass String. You'd be adding something very unwieldy for an edge case that is directly contrary to the purpose of the API. The 'mistake' (such as it is) of the author here is simply using the wrong API - there are shorter, more direct ways to read a file into a string without any of these contortions.

    data = new String(Files.readAllBytes(Paths.get(filename)));

Re: How fast can a BufferedReader read lines in Java?

#82

Post by pjmlp »

Zach_the_Lizard wrote: post #24
nottorp wrote: post #3

Java is... java. I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision). Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.

> Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up Let us all hope that Project Valhalla, the effort to add value types to Java, comes to a swift completion. It would be very helpful in these sorts of scenarios. Though at this point I'd wonder (as you did) why I'm using Java in the firs…

You can already play with Valhalla.

https://mail.openjdk.java.net/pipermail/valhalla-dev/2019-Ju...

Re: How fast can a BufferedReader read lines in Java?

#83

Post by bhaak »

ricardobeat wrote: post #78
bhaak wrote: post #77

Earlier quoted context omitted.

There are several simple data transformation steps. There's nothing inherently wrong with these steps. The compiler can't read your mind and guess what you wanted to do. How would this be possible to be prevented with any programming language, existing or hypothetical? Even if the meta information that the appended string doesn't contain any newlines would be passed on, the second call to lines() would rather use is…

To give an example: if the lines() method returned instances of a Line class, the string buffer and it's append() method would be aware and still create a collection of lines instead of one long string. Not necessarily nice and would probably create issues in many other interfaces, but problems like this _can_ be solved with careful language and API design. Note that his end goal was to 'read lines' and not use any p…

That still comes down to an understanding of the API.

I don't think that appending a SingleLineString to a String object would obviously add a newline. The String/StringBuffer/StringBuilder classes in Java certainly don't work that way.

If you were adding to some other object like an instance of some text representation class, this could be different. But it would still not be 100% safe to deduce just by the API itself without looking at the documentation.

Re: How fast can a BufferedReader read lines in Java?

#85

Post by nottorp »

nottorp wrote: post #3

Java is... java. I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision). Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.

Interesting comments so let's have some answers:

1. 128 M ram total memory, 20-30 M free at best. CPU was slow and we were running off a SD card. Not exactly an enterprise monster.

2. Of course I was parsing it line by line. But running OOM triggered the garbage collector so many times that it took like 3-4 minutes to finish. I didn't mean that the system actually stopped because of running out of ram.

3. Why didn't you do it with the NDK/some other solution? Because the PM just gave up on the feature when it wasn't done in a day. It was just real time search completion whith we didn't really need there, it was just "nice to have".

Re: How fast can a BufferedReader read lines in Java?

#86

Post by nottorp »

kuschku wrote: post #42
nottorp wrote: post #3

Java is... java. I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision). Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.

I’m parsing much larger datasets than those on android, in less than 18M RAM usage. You just have to use NIO and use direct buffers instead of naively reading and allocating strings.

So what's the point of using Java then? Of course I can do it in a non GC language using a buffer that holds one line and as much memory as needed for the data I want to retain...

Re: How fast can a BufferedReader read lines in Java?

#87

Post by kuschku »

nottorp wrote: post #86
kuschku wrote: post #42

Earlier quoted context omitted.

I’m parsing much larger datasets than those on android, in less than 18M RAM usage. You just have to use NIO and use direct buffers instead of naively reading and allocating strings.

So what's the point of using Java then? Of course I can do it in a non GC language using a buffer that holds one line and as much memory as needed for the data I want to retain...

Direct Buffers are a Java API to directly use C buffers and access them in a fast way. They're the fastest way to handle memory in Java.

Re: How fast can a BufferedReader read lines in Java?

#88

Post by AtlasBarfed »

VRay wrote: post #20
TheChaplain wrote: post #13

Earlier quoted context omitted.

Myself I've worked with gigabyte sized CSV files without issues, so it was likely your implementation rather than the fault of the Java language.

Googles "parse csv Java" Copies and pasted top answer that buffers the whole thing into the heap before parsing Blames Java/hardware when this doesn't scale

Bad string allocations can easily swamp the JVM, with things like loops adding a single character to a string with a string allocation each loop, and other naive approaches turning what should be a small reused buffer into gigabytes of heap.

The JVM does a remarkable job of just-good-enough don't-worry-about-GC, but anyone who is a paid programmer needs to understand rudimentary aspects of GC, or your stuff will go sideways fast in production.

Re: How fast can a BufferedReader read lines in Java?

#89

Post by AtlasBarfed »

nottorp wrote: post #86
kuschku wrote: post #42

Earlier quoted context omitted.

I’m parsing much larger datasets than those on android, in less than 18M RAM usage. You just have to use NIO and use direct buffers instead of naively reading and allocating strings.

So what's the point of using Java then? Of course I can do it in a non GC language using a buffer that holds one line and as much memory as needed for the data I want to retain...

Not having to worry about memory for 99% of the time.

Having to do research to optimize I/O is not just a java thing. As stated before in this thread, the C stdlib isn't historically great either, and a lot of languages piggyback on that.

Modern I/O is almost always a different library and you need to do reasearch.

Re: How fast can a BufferedReader read lines in Java?

#90

Post by nottorp »

AtlasBarfed wrote: post #88
VRay wrote: post #20

Earlier quoted context omitted.

Googles "parse csv Java" Copies and pasted top answer that buffers the whole thing into the heap before parsing Blames Java/hardware when this doesn't scale

Bad string allocations can easily swamp the JVM, with things like loops adding a single character to a string with a string allocation each loop, and other naive approaches turning what should be a small reused buffer into gigabytes of heap. The JVM does a remarkable job of just-good-enough don't-worry-about-GC, but anyone who is a paid programmer needs to understand rudimentary aspects of GC, or your stuff will go s…

Bad string allocations can easily swamp the JVM

Like String.split() and whatever BufferedReader does, I guess. It's not like i was doing byte by byte manipulation. The GC just triggered too often.

Post reply on HN