71–80 of 90 posts
Honestly, it seems that nearly everyone here is missing his point. Some of the blame for that probably lies with his headline choice, but he clearly states at the end of this post: """ This is not the best that Java can do: Java can ingest data much faster. However, my results suggest that on modern systems, Java file parsing might be frequently processor-bound, as opposed to system bound. That is, you can buy much b…
Earlier quoted context omitted.
That's a pretty big error if you're correct. What does it say about the language when a CS professor falls for this on a 40 line file?
Getting benchmarks right is difficult, even for a CS professor. The language doesn't even enter there. What does it say about you that you're asking a leading question in this way?
I don't know what Java's BufferedReader is doing, but it's probably not the optimal thing in terms of IO throughput. I would blame the algorithm long before blaming anything inherent about the JVM. Erlang is another language where "naive" IO is kind of slow. https://github.com/bbense/beatwc/ is a project someone did to test various methods of doing IO in Erlang/Elixir, and their performance for a line-counting task,…
Java NIO does exactly what you describe. IO is chunked optimally, the entire buffer is pulled, the API exposes a select() mechanism just like any libc, and the user is expected to frame its own received data. edit: Lemire is showing a lack of what Fowler describes as "mechanical sympathy". https://martinfowler.com/articles/lmax.html#QueuesAndTheirLa...
The first issue I can see with that code is it's not doing what he expects. He does this to read the file into a StringBuffer: bf.lines().forEach(s -> sb.append(s)); However, this ends up reading all the lines into one giant line, since the String's that lines() produces have the newline character stripped. This leads to the second lines() call to read a 23MB line (the file produced by gen.py). This is less than opti…
That's a pretty big error if you're correct. What does it say about the language when a CS professor falls for this on a 40 line file?
So what's the reason for this? Is it maybe because of some unicode shenanigans? Java characters are 16bit iirc, and strings have some forty bytes of constant overhead.
I'm no Java ninja, but a few things jump out of https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c... : - at least one heap allocation for every line. After it finds the EOL it first uses 'new String' followed by '.toString() - the C++ version will almost certainly be backing on to memchr() behind the scenes, which will be using SIMD instructions where it makes sense (e.g. large enough scan size, probably t…
As a result std::string was disgustingly slow compared to C code.
Completely wrong. It is like asserting something about C based on GCC specific behaviour. Java is not a single language implementation.
Agreed. I think you are being downvoted because you forgot to paste in your benchmark results.
So now I have to go out and do benchmarks across all these implementations just to prove they aren't the same?!?
https://en.m.wikipedia.org/wiki/List_of_Java_virtual_machine...
Earlier quoted context omitted.
Getting benchmarks right is difficult, even for a CS professor. The language doesn't even enter there. What does it say about you that you're asking a leading question in this way?
I'm always interested in thoughts about language design and programming in general. We have fifty+ years of commercial software development and it still sucks. This caught my eye as the kind of error that shouldn't even be allowed to happen.
How would this be possible to be prevented with any programming language, existing or hypothetical?
Even if the meta information that the appended string doesn't contain any newlines would be passed on, the second call to lines() would rather use is as an optimization hint instead of raising an error.
Earlier quoted context omitted.
I'm always interested in thoughts about language design and programming in general. We have fifty+ years of commercial software development and it still sucks. This caught my eye as the kind of error that shouldn't even be allowed to happen.
There are several simple data transformation steps. There's nothing inherently wrong with these steps. The compiler can't read your mind and guess what you wanted to do. How would this be possible to be prevented with any programming language, existing or hypothetical? Even if the meta information that the appended string doesn't contain any newlines would be passed on, the second call to lines() would rather use is…
Eh, he didn't use NIO. BufferedReader is an ancient Java relic. Like reading from STDIN in c, it's not made to be fast, it's there for convenience and backwards compatibility. Read a file using something like Vert.X, which is optimized for speed. I'm 100% confident it will be faster than the naive c approach
Do you have example of alternatives for the BufferedReader with the NIO APIs? I do a lot of work with large GZIP that are read line by line using the standard IO (i.e. GzipInputStrem(FileInputStream)) etc) but your comment has really made me second guess my choice of doing that...
You should be able to hack something together that feeds zero copy buffers into Netty compression handler. Maybe using Netty or Vert.X file API, or maybe just raw NIO2.
I'm not sure how fast this would be, but my gut says "very". Netty can easily saturate 40 Gigabit Ethernet lines, and file IO should have less overhead. That's ~5 gigabytes a second.
It's going to be a good bit of coding for sure. Vert.X/Netty/NIO2 are all async and pretty low level. They're generally 1 thread per core, and along with SSD read patterns you're probably best off reading files in parallel, one per core. Might not be worth the effort.
You may want to look into ZStandard as well. It's Superior to Gzip in most ways when you need something fast but decent.
Eh, he didn't use NIO. BufferedReader is an ancient Java relic. Like reading from STDIN in c, it's not made to be fast, it's there for convenience and backwards compatibility. Read a file using something like Vert.X, which is optimized for speed. I'm 100% confident it will be faster than the naive c approach
Do you have a quick github project example?