Ooh fun, Advent of Code chaser!
A fair comparison between languages should include the make and build times. I haven't used Java / Maven for years, and I'm reminded why, heading into minute 2 of downloads for './mvnw clean verify'.
The One Billion Row Challenge
31–40 of 366 posts
Re: The One Billion Row Challenge
#32Ooh fun, Advent of Code chaser! A fair comparison between languages should include the make and build times. I haven't used Java / Maven for years, and I'm reminded why, heading into minute 2 of downloads for './mvnw clean verify'.
Java build times are very fast. You are just measuring your internet speed here.
(Also, gradle is faster as a build tool for incremental compilation)
Re: The One Billion Row Challenge
#33> Each contender will be run five times in a row. The slowest and the fastest runs are discarded. The mean value of the remaining three runs is the result for that contender and will be added to the leaderboard. Shouldn’t he take the single fastest time, assuming file (and JDK) being in file cache is controlled for?
This is done to simulate real-world performance. Your binary is not the only binary in the system and other services may be running as well. So fastest time is the happiest path and slowest is the unluckiest. The range of remaining three is what you expect to get 99% of the time on a real world system.
Any production where you're routinely scanning a text file with a million records is probably a batch process, and I'd be shocked if the usual performance wasn't much closer to the worst case than the average.
Re: The One Billion Row Challenge
#34Earlier quoted context omitted.
Rather than read the file into memory, memory mapping can be used.
You wouldn't want to do this for a huge file. A very fast solution would use a small number of buffers and io_uring (or equivalent), keeping the page table and cache footprint small.
Yeah so I had a discussion on Twitter about this, turns out 12GB is small enough to fit into memory, and the author runs submissions by running a solution 5 times in a row, so using direct IO actually hurts because having the kernel cache is a way to enforce the file is in memory for the 4 runs after. I have a direct IO solution with SIMD string search and double parsing, just in C++ (using libraries). It runs in 6 seconds on my 24 core linux box (NVMe).
Code: https://github.com/rockwotj/1brc
Discussion on Filesystem cache: https://x.com/rockwotj/status/1742168024776430041?s=20
Re: The One Billion Row Challenge
#35Ooh fun, Advent of Code chaser! A fair comparison between languages should include the make and build times. I haven't used Java / Maven for years, and I'm reminded why, heading into minute 2 of downloads for './mvnw clean verify'.
Java build times are very fast. You are just measuring your internet speed here. (Also, gradle is faster as a build tool for incremental compilation)
...very relative
Re: The One Billion Row Challenge
#36Ooh fun, Advent of Code chaser! A fair comparison between languages should include the make and build times. I haven't used Java / Maven for years, and I'm reminded why, heading into minute 2 of downloads for './mvnw clean verify'.
Don't need maven
> No external dependencies may be used
Re: The One Billion Row Challenge
#37Earlier quoted context omitted.
Rather than read the file into memory, memory mapping can be used.
You wouldn't want to do this for a huge file. A very fast solution would use a small number of buffers and io_uring (or equivalent), keeping the page table and cache footprint small.
What is the downside of memory mapping in this scenario? Shouldn't the page table properly handle the case of doing a single sequential read over a range of pages? Accessing the contents of a file doesn't seem like something caching would matter for. Do you mean that reading of sequential pages will keep adding to the cache compared to reading from a single page? That seems like a similar thing as before where they will be the first things gone from the cache anyways.
Re: The One Billion Row Challenge
#38> Each contender will be run five times in a row. The slowest and the fastest runs are discarded. The mean value of the remaining three runs is the result for that contender and will be added to the leaderboard. Shouldn’t he take the single fastest time, assuming file (and JDK) being in file cache is controlled for?
Why?
The author does explicitly want all the files to be cached in memory for the later runs: https://x.com/gunnarmorling/status/1742181941409882151?s=20
Re: The One Billion Row Challenge
#39Ooh fun, Advent of Code chaser! A fair comparison between languages should include the make and build times. I haven't used Java / Maven for years, and I'm reminded why, heading into minute 2 of downloads for './mvnw clean verify'.
Java build times are very fast. You are just measuring your internet speed here. (Also, gradle is faster as a build tool for incremental compilation)
Gradle is faster than what? Than maven? Maybe. But not than Go or Cargo.