Earlier quoted context omitted.
Why would I use Hadoop for such a small number of rows…?
1 billion is small for hadoop?
If it fits on one computer it's not a hadoop problem.
The One Billion Row Challenge
21–30 of 366 posts
Re: The One Billion Row Challenge
#22I don't understand, it should be pretty easy. A rolling average with BigDecimal would probably be sufficient but a scientific lib might be better for a rolling average or more than a hundred million numbers. https://stackoverflow.com/questions/277309/java-floating-poi...
The difficulty is creating the fastest implementation. If you look at the results of the submissions so far you’ll see a big difference in duration, between 11 seconds and more than 4 minutes.
11 seconds seems pretty impressive for a 12Gb file. Would be interesting to know what programming language could do it faster. For a database comparison you’d probably want to include loading the data into your database for a fair comparrison.
Re: The One Billion Row Challenge
#23Interesting challenge, shame its only java. Can't wait till people start hand rolling their own JVM bytecode.
Alternatively, "must be written in Java" can be interpreted to mean "must use the JVM to begin execution", and you can clearly spawn another process from Java...
Re: The One Billion Row Challenge
#24Earlier quoted context omitted.
It looks like the problem is dominated by reading in the data file. Some fast solutions just read the whole file into memory.
Rather than read the file into memory, memory mapping can be used.
But would that really be faster when you need to read every byte of a file?
I thought memory mapping solved a different problem.
Re: The One Billion Row Challenge
#25[deleted]
Re: The One Billion Row Challenge
#26> Q: Can I make assumptions on the names of the weather stations showing up in the data set? > A: No, while only a fixed set of station names is used by the data set generator, any solution should work with arbitrary UTF-8 station names (for the sake of simplicity, names are guaranteed to contain no `;` character). I'm unsure if it's intentional or not, but this essentially means that the submission should be correct…
Given this requirement, they would be wise to have the test data be different from the example data. That would prevent overfitting optimizations.
I'm not sure that it is overfitting to optimize your code for the test set. The requirement is just that you don't break compatibility for arbitrary names in the process.
This sort of thing shows up all the time in the real world - where, for example, 90% of traffic will hit one endpoint. Or 90% of a database is one specific table. Discovering and microoptimizing for the common case is an important skill.
Re: The One Billion Row Challenge
#27Earlier quoted context omitted.
It looks like the problem is dominated by reading in the data file. Some fast solutions just read the whole file into memory.
Rather than read the file into memory, memory mapping can be used.
You wouldn't want to do this for a huge file. A very fast solution would use a small number of buffers and io_uring (or equivalent), keeping the page table and cache footprint small.
Re: The One Billion Row Challenge
#28Earlier quoted context omitted.
Given this requirement, they would be wise to have the test data be different from the example data. That would prevent overfitting optimizations.
I'm not sure that it is overfitting to optimize your code for the test set. The requirement is just that you don't break compatibility for arbitrary names in the process. This sort of thing shows up all the time in the real world - where, for example, 90% of traffic will hit one endpoint. Or 90% of a database is one specific table. Discovering and microoptimizing for the common case is an important skill.
That's why I was so unsure. I think though, that it is generally better to have the input generator seeded and do not disclose the chosen seed in advance, which prevents overfitting to a single input and yet encourages overfitting to a much bigger class of inputs.
Re: The One Billion Row Challenge
#29> Each contender will be run five times in a row. The slowest and the fastest runs are discarded. The mean value of the remaining three runs is the result for that contender and will be added to the leaderboard. Shouldn’t he take the single fastest time, assuming file (and JDK) being in file cache is controlled for?
This is done to simulate real-world performance. Your binary is not the only binary in the system and other services may be running as well. So fastest time is the happiest path and slowest is the unluckiest.
The range of remaining three is what you expect to get 99% of the time on a real world system.
Re: The One Billion Row Challenge
#30Interesting challenge, shame its only java. Can't wait till people start hand rolling their own JVM bytecode.
Alternatively, "must be written in Java" can be interpreted to mean "must use the JVM to begin execution", and you can clearly spawn another process from Java...
Another rule was no external dependencies
I guess you could from Java itself write a new binary and then run that binary, but it would be against the spirit of the challenge.