How fast can a BufferedReader read lines in Java?

Post by peq » Sat, Jul 27, 2019, 7:03 PM UTC

elFarto wrote: post #28

The first issue I can see with that code is it's not doing what he expects. He does this to read the file into a StringBuffer: bf.lines().forEach(s -> sb.append(s)); However, this ends up reading all the lines into one giant line, since the String's that lines() produces have the newline character stripped. This leads to the second lines() call to read a 23MB line (the file produced by gen.py). This is less than opti…

    public void readString(String data) throws IOException {
        int lastIdx = 0;
        for (int idx = data.indexOf('\n'); idx > -1; idx = data.indexOf('\n', lastIdx)) {
            parseLine(subSequenceView(data, lastIdx, idx));
            lastIdx = idx + 1;
        }
        parseLine(subSequenceView(data, lastIdx, data.length()));
    }

    CharSequence subSequenceView(CharSequence base, int beginIndex, int endIndex) {
        return new StringView(base, beginIndex, endIndex - beginIndex);
    }

    static class StringView implements CharSequence {
        final CharSequence base;
        final int offset;
        final int length;

        StringView(CharSequence base, int offset, int length) {
            if (length = 0");
            this.base = base;
            this.offset = offset;
            this.length = length;
        }

        @Override
        public char charAt(int n) {
            if (n = length)
                throw new IndexOutOfBoundsException(n);
            return base.charAt(offset + n);
        }

        @Override
        public int length() {
            return length;
        }

        @Override
        public CharSequence subSequence(int beginIndex, int endIndex) {
            return new StringView(base, offset + beginIndex, endIndex - beginIndex);
        }
    }

Post by pvg » Sat, Jul 27, 2019, 7:24 PM UTC

chvid wrote: post #62

At least two problems with the java code. Concatenation of strings using the plus operator creates a new string and copies the content of the old, that pushes the complexity of the code from o(n) to o(n2) where n is the number of lines. Secondly order is not guaranteed with the for each operation on streams. The correct way to do it is using collect(Collectors.joining(“\n”)) or straight forward imperative style (with…

Post by tom_mellior » Sat, Jul 27, 2019, 7:49 PM UTC

gpderetta wrote: post #52

tom_mellior wrote: post #31

Earlier quoted context omitted.

If the article's point was that better alternatives exist, it should have made that point by mentioning those better alternatives and ideally benchmarking them as well. I agree with others that this particular article comes across as very lazy and not up to Lemire's usual standard.

That's not the point of the article. The point of the article is that many programs are in fact CPU bound, contrary to the often repeated claims and you do actually need to do optimization work to saturate your IO; even in lanuages considered fast, like c++ and java, the straightforward implementation is often suboptimal.

Post by vbezhenar » Sat, Jul 27, 2019, 7:52 PM UTC

Post by _old_dude_ » Sat, Jul 27, 2019, 7:54 PM UTC

djhworld wrote: post #51

nullwasamistake wrote: post #10

Eh, he didn't use NIO. BufferedReader is an ancient Java relic. Like reading from STDIN in c, it's not made to be fast, it's there for convenience and backwards compatibility. Read a file using something like Vert.X, which is optimized for speed. I'm 100% confident it will be faster than the naive c approach

Do you have example of alternatives for the BufferedReader with the NIO APIs? I do a lot of work with large GZIP that are read line by line using the standard IO (i.e. GzipInputStrem(FileInputStream)) etc) but your comment has really made me second guess my choice of doing that...

  Path path = ...
  var count = 0;
  try(var channel = FileChannel.open(path)) {
    var buffer = ByteBuffer.allocateDirect(8192);
    while(channel.read(buffer) != -1) {
      while(buffer.hasRemaining()) {
        if (buffer.get() == '\n') {
          count++;
        }
      }
      buffer.clear();
    }
  }
  System.out.println(count);