41–50 of 90 posts
The first issue I can see with that code is it's not doing what he expects. He does this to read the file into a StringBuffer: bf.lines().forEach(s -> sb.append(s)); However, this ends up reading all the lines into one giant line, since the String's that lines() produces have the newline character stripped. This leads to the second lines() call to read a 23MB line (the file produced by gen.py). This is less than opti…
Where is the second lines call?
Java is... java. I was once working on an Android app on a cheap custom board with 128 M ram (don't ask why Android on a single function custom board, wasn't my decision). Among other things, I had to parse a 80000 line csv file. Splitting and the rest of the processing created so many temporary strings the system ran out of ram. We eventually gave up.
You just have to use NIO and use direct buffers instead of naively reading and allocating strings.
The first issue I can see with that code is it's not doing what he expects. He does this to read the file into a StringBuffer: bf.lines().forEach(s -> sb.append(s)); However, this ends up reading all the lines into one giant line, since the String's that lines() produces have the newline character stripped. This leads to the second lines() call to read a 23MB line (the file produced by gen.py). This is less than opti…
I'm confused. Where is the second lines call?
Some of the blame for that probably lies with his headline choice, but he clearly states at the end of this post:
""" This is not the best that Java can do: Java can ingest data much faster. However, my results suggest that on modern systems, Java file parsing might be frequently processor-bound, as opposed to system bound. That is, you can buy much better disks and network cards, and your system won’t go any faster. Unless, of course, you have really good Java engineers.
Many firms probably just throw more hardware at the problem. """
It's not about this piece of code. It's not even about Java In the previous post he mentions at the start of this one, he pointed out:
""" These results suggest that reading a text file in C++ could be CPU bound in that sense that buying an even faster disk would not speed up your single-threaded throughput. """
So, I take his point to be that one shouldn't make assumptions about performance. Rough performance scales -- such as have been posted here many times (e.g. [1]) -- make great rules of thumb for implementation choices or as a guide for where to look first for bottlenecks. To optimize in the real world, though, you're best served using real measurements.
[1] https://www.prowesscorp.com/computer-latency-at-a-human-scal...
Earlier quoted context omitted.
Lemire is one of the leading experts on string matching and the author of several core libraries you probably use every day. edit fine, so instead maybe click on the links in the post to see that this article is just one of a series. He's probably tired of copy-pasting the specs of his reference hardware (Skylake https://arxiv.org/pdf/1902.08318.pdf ) since all he's concerned about is the relative performance of diff…
Then they should know enough to give at least some theories that can explain the difference and tell us something more about the test setup.
Earlier quoted context omitted.
Lemire is one of the leading experts on string matching and the author of several core libraries you probably use every day. edit fine, so instead maybe click on the links in the post to see that this article is just one of a series. He's probably tired of copy-pasting the specs of his reference hardware (Skylake https://arxiv.org/pdf/1902.08318.pdf ) since all he's concerned about is the relative performance of diff…
Then he should really know better. Not sure why you think the sentence you have written is even an argument.
I'm appalled at this attitude of entitlement, people don't owe us anything when publishing free content on the web. It's ok to suggest a change or ask for more details, but keep it respectful.
The first issue I can see with that code is it's not doing what he expects. He does this to read the file into a StringBuffer: bf.lines().forEach(s -> sb.append(s)); However, this ends up reading all the lines into one giant line, since the String's that lines() produces have the newline character stripped. This leads to the second lines() call to read a 23MB line (the file produced by gen.py). This is less than opti…
That's a pretty big error if you're correct. What does it say about the language when a CS professor falls for this on a 40 line file?
What does it say about you that you're asking a leading question in this way?
Earlier quoted context omitted.
I'm confused. Where is the second lines call?
Lines 19 and 27 in the source: https://github.com/lemire/Code-used-on-Daniel-Lemire-s-blog/...
You're right, the hidden scanFile()...makes no sense.
It's stripping all the new lines from the input.
Why? What is going on?
Earlier quoted context omitted.
Lines 19 and 27 in the source: https://github.com/lemire/Code-used-on-Daniel-Lemire-s-blog/...
What the heck? You're right, the hidden scanFile()...makes no sense. It's stripping all the new lines from the input. Why? What is going on?
Not the best method to use, probably better to use the Files.readString method.