21–30 of 90 posts
Eh, he didn't use NIO. BufferedReader is an ancient Java relic. Like reading from STDIN in c, it's not made to be fast, it's there for convenience and backwards compatibility. Read a file using something like Vert.X, which is optimized for speed. I'm 100% confident it will be faster than the naive c approach
The post does nothing to explain how and why, it just throws a couple of outputs from a non specified machine and does no comparison. It has no baseline and no specs. For all I know, he could have got his 0.5 GB/sec on ab old Pentium II processor. There is no analysis. I am perplexed.
Lemire is one of the leading experts on string matching and the author of several core libraries you probably use every day. edit fine, so instead maybe click on the links in the post to see that this article is just one of a series. He's probably tired of copy-pasting the specs of his reference hardware (Skylake https://arxiv.org/pdf/1902.08318.pdf ) since all he's concerned about is the relative performance of diff…
So what's the reason for this? Is it maybe because of some unicode shenanigans? Java characters are 16bit iirc, and strings have some forty bytes of constant overhead.
I'm no Java ninja, but a few things jump out of https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c... : - at least one heap allocation for every line. After it finds the EOL it first uses 'new String' followed by '.toString() - the C++ version will almost certainly be backing on to memchr() behind the scenes, which will be using SIMD instructions where it makes sense (e.g. large enough scan size, probably t…
Java is... java. I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision). Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.
Let us all hope that Project Valhalla, the effort to add value types to Java, comes to a swift completion. It would be very helpful in these sorts of scenarios.
Though at this point I'd wonder (as you did) why I'm using Java in the first place given the resource constraints of the system, I've had some success using byte[] arrays and using `yourInputStreamHere.read(someBuffer, yourOffset, YOUR_PAGE_SIZE)` in a few less constrained but relatively more performance critical areas.
Depending on your exact needs, this can sometimes result in more or less constant memory consumption; it's possible to reuse the array on each read. You can also take it a step further by finding the indexes where you would split the array (e.g. the index of the next comma for a CSV) and using methods that operate on an array plus start and end indexes. That sort of strategy can allow you to avoid additional copies the array, at the cost of somewhat annoying and less maintainable code that is definitely not Java-esque.
There are also less severe solutions that won't buffer the whole file in memory, using StringBuilder or other techniques, etc. depending on what you're doing.
This all might be for naught though; I'm assuming the records have to be parsed into some structure, so tricks like the above might not be good enough given Java's seemingly insatiable hunger for heap space.
While I'm not the language's greatest fan, this is one area where Go has been able to shine. Having value types as a possibility makes solving this sort of problem a typically less expensive proposition.
Earlier quoted context omitted.
Lemire is one of the leading experts on string matching and the author of several core libraries you probably use every day. edit fine, so instead maybe click on the links in the post to see that this article is just one of a series. He's probably tired of copy-pasting the specs of his reference hardware (Skylake https://arxiv.org/pdf/1902.08318.pdf ) since all he's concerned about is the relative performance of diff…
That doesn’t excuse him from needing to describe the hardware he ran the benchmark on.
I'm not saying that's happening here, but it's a basic fact when writing benchmarks that you have to actually test something real and not a transient property of the program after the compiler has had its chance to be really smart.
Impressive bad for a professor. Even worse for someone who says on their page "I like crazy fast code." He certainly doesn't seem to know how to produce it (or if you believe he does - he's being intentionally obtuse or lazy). Using the simplest, slowest, oldest method to read as file line by line from a time when there wasn't such a thing a high performance Java (trading systems are built in Java that rival c++ perf…
If you work at some of the places java commonly gets used then you'll see BufferedReader all over the place. This article isn't a critique of Java. It's a warning about BufferedReader. If you do high performance java you'll be hiring exceptional java engineers who will not be using BufferedReader and you won't have this problem. If you are a run of the mill java shop you probably don't work with code that isn't using BufferedReader and this article is for you when you are debugging a pathological performance problem that suddenly reared it's head in production.
This professor seems like a good one to me.
bf.lines().forEach(s -> sb.append(s));
The fastest version I managed to write was:
public void readString5(String data) throws IOException {
int lastIdx = 0;
for (int idx = data.indexOf('\n'); idx > -1; idx = data.indexOf('\n', lastIdx))
{
parseLine(data.substring(lastIdx, idx));
lastIdx = idx+1;
}
parseLine(data.substring(lastIdx));
}
Impressive bad for a professor. Even worse for someone who says on their page "I like crazy fast code." He certainly doesn't seem to know how to produce it (or if you believe he does - he's being intentionally obtuse or lazy). Using the simplest, slowest, oldest method to read as file line by line from a time when there wasn't such a thing a high performance Java (trading systems are built in Java that rival c++ perf…
Alternatively if you took his class then you you probably learned that BufferedReader is not the most performant way to read lines from IO and probably learned the correct alternatives. If you work at some of the places java commonly gets used then you'll see BufferedReader all over the place. This article isn't a critique of Java. It's a warning about BufferedReader. If you do high performance java you'll be hiring…
I can only evaluate him on what I just read, and it isn't very good.
So what's the reason for this? Is it maybe because of some unicode shenanigans? Java characters are 16bit iirc, and strings have some forty bytes of constant overhead.