Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The One Billion Row Challenge

morling.dev

Post reply on HN
Print view View original

11–20 of 366 posts

Re: The One Billion Row Challenge

#11

Post by lifthrasiir »

> Q: Can I make assumptions on the names of the weather stations showing up in the data set?

> A: No, while only a fixed set of station names is used by the data set generator, any solution should work with arbitrary UTF-8 station names (for the sake of simplicity, names are guaranteed to contain no `;` character).

I'm unsure if it's intentional or not, but this essentially means that the submission should be correct for all inputs, but can and probably should be tuned for the particular input regenerated by `create_measurements.sh`. I can imagine submissions with a perfect hash function tuned for given set of stations, for example.

Re: The One Billion Row Challenge

#13

Post by pbh101 »

> Each contender will be run five times in a row. The slowest and the fastest runs are discarded. The mean value of the remaining three runs is the result for that contender and will be added to the leaderboard.

Shouldn’t he take the single fastest time, assuming file (and JDK) being in file cache is controlled for?

Re: The One Billion Row Challenge

#15

Post by Animats »

kriskrunch wrote: post #10
worthless-trash wrote: post #2

Interesting challenge, shame its only java. Can't wait till people start hand rolling their own JVM bytecode.

Check out the discussion[0], looks like there are submissions in several languages. Go, Rust, Python, and C++, to name a few [0] https://github.com/gunnarmorling/1brc/discussions

It looks like the problem is dominated by reading in the data file. Some fast solutions just read the whole file into memory.

Re: The One Billion Row Challenge

#17

Post by fny »

Animats wrote: post #15
kriskrunch wrote: post #10

Earlier quoted context omitted.

Check out the discussion[0], looks like there are submissions in several languages. Go, Rust, Python, and C++, to name a few [0] https://github.com/gunnarmorling/1brc/discussions

It looks like the problem is dominated by reading in the data file. Some fast solutions just read the whole file into memory.

Rather than read the file into memory, memory mapping can be used.

Re: The One Billion Row Challenge

#18

Post by Hugsun »

lifthrasiir wrote: post #11

> Q: Can I make assumptions on the names of the weather stations showing up in the data set? > A: No, while only a fixed set of station names is used by the data set generator, any solution should work with arbitrary UTF-8 station names (for the sake of simplicity, names are guaranteed to contain no `;` character). I'm unsure if it's intentional or not, but this essentially means that the submission should be correct…

Given this requirement, they would be wise to have the test data be different from the example data. That would prevent overfitting optimizations.
Post reply on HN