Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The One Billion Row Challenge

morling.dev

Post reply on HN
Print view View original

41–50 of 366 posts

Re: The One Billion Row Challenge

#41

Post by pclmulqdq »

rockwotj wrote: post #34
pclmulqdq wrote: post #27

Earlier quoted context omitted.

You wouldn't want to do this for a huge file. A very fast solution would use a small number of buffers and io_uring (or equivalent), keeping the page table and cache footprint small.

Yeah so I had a discussion on Twitter about this, turns out 12GB is small enough to fit into memory, and the author runs submissions by running a solution 5 times in a row, so using direct IO actually hurts because having the kernel cache is a way to enforce the file is in memory for the 4 runs after. I have a direct IO solution with SIMD string search and double parsing, just in C++ (using libraries). It runs in 6 s…

I missed the "5 times in a row." If you do that, yeah, keeping the whole thing in memory is far better.

Re: The One Billion Row Challenge

#42

Post by lifthrasiir »

rockwotj wrote: post #34
pclmulqdq wrote: post #27

Earlier quoted context omitted.

You wouldn't want to do this for a huge file. A very fast solution would use a small number of buffers and io_uring (or equivalent), keeping the page table and cache footprint small.

Yeah so I had a discussion on Twitter about this, turns out 12GB is small enough to fit into memory, and the author runs submissions by running a solution 5 times in a row, so using direct IO actually hurts because having the kernel cache is a way to enforce the file is in memory for the 4 runs after. I have a direct IO solution with SIMD string search and double parsing, just in C++ (using libraries). It runs in 6 s…

> double parsing

In case you haven't noticed yet, the input format guarantees exactly one fractional digit, so you can read a single signed integer followed by `.` and one digit instead.

Re: The One Billion Row Challenge

#43

Post by nneonneo »

userbinator wrote: post #23
worthless-trash wrote: post #2

Interesting challenge, shame its only java. Can't wait till people start hand rolling their own JVM bytecode.

Alternatively, "must be written in Java" can be interpreted to mean "must use the JVM to begin execution", and you can clearly spawn another process from Java...

Just use JNA to allocate an RWX buffer with mmap, unpack a bit of clever code into the buffer, and forge a native function pointer to call it :)

Re: The One Billion Row Challenge

#44

Post by wokwokwok »

pkolaczk wrote: post #39
kaba0 wrote: post #32

Earlier quoted context omitted.

Java build times are very fast. You are just measuring your internet speed here. (Also, gradle is faster as a build tool for incremental compilation)

Gradle is faster than what? Than maven? Maybe. But not than Go or Cargo.

Itself.

> gradle is faster as a build tool for incremental compilation

Implicit:

> …than it is building from scratch, where it needs to download lots of stuff

I mean, yes, saying Java builds are fast does seem a bit “rolls eyes, yes technically by loc when the compiler is actually running” …but, ^_^! let’s not start banging on about how great cargo/rust compile times are… they’re really terrible once procedural macros are used, or sys dependencies invoke some heinous c dependency build like autoconf and lots of crates do… and there’s still a subpar incremental compilation story.

So, you know. Eh. Live and let live. Gradle isn’t that bad.

Re: The One Billion Row Challenge

#46

Post by chii »

mrkeen wrote: post #36
vessenes wrote: post #31

Ooh fun, Advent of Code chaser! A fair comparison between languages should include the make and build times. I haven't used Java / Maven for years, and I'm reminded why, heading into minute 2 of downloads for './mvnw clean verify'.

Don't need maven > No external dependencies may be used

maven is just a build tool. It creates a single uberjar which you run with the JDK directly - you don't ship the maven project.

But the real meaning is that you're not allowed to use external libraries, rather than build tool related dependency.

Re: The One Billion Row Challenge

#47

Post by wokwokwok »

I suppose it defeats the spirit of the game to, knowing your worst run is discarded, calculate the results on the first run by whatever slow method you want, save them somewhere useful, and just read the results and print them out on the following runs?

Or at the very least, convert the input into a more convenient binary format for the following runs.

Re: The One Billion Row Challenge

#49

Post by wiseowise »

vessenes wrote: post #31

Ooh fun, Advent of Code chaser! A fair comparison between languages should include the make and build times. I haven't used Java / Maven for years, and I'm reminded why, heading into minute 2 of downloads for './mvnw clean verify'.

Why would you clean?

“I discard cache and it is so slow, aarghhh”

Post reply on HN