The One Billion Row Challenge

Post by kaba0 » Thu, Jan 04, 2024, 10:13 PM UTC

troupo wrote: post #310

kaba0 wrote: post #291

Earlier quoted context omitted.

It has always been “an imperative code that outputs the declarative configuration that can be used for the build”. Which is a very reasonable thing to do, but unfortunately most people fail to understand that a println in the config file’s global scope is different than inside a closure for a task description.

> but unfortunately most people fail to understand that a println in the config file’s global scope is different than inside a closure for a task description. Literally no one complaining about gradle being slow is doing that. Allmost all of gradle's problems come not from people using it, but from Gradle itself. I mean, you said it yourself: in 15 years the state of their API docs is "okayish". But somehow people ar…

Post by gerikson » Thu, Jan 04, 2024, 10:51 PM UTC

lessbergstein wrote: post #174

gerikson wrote: post #119

Earlier quoted context omitted.

I profiled my attempt, actually reading each line is the bottleneck.

Perl is always going to be much faster than Java at tasks like this. Use stdin and chomp() instead of reading each line explicitly. This is really a small, trivial task for a perl script. Even with a billion lines this is nothing for a modern cpu and perl.

Post by corlinpalmer » Thu, Jan 04, 2024, 11:04 PM UTC

Post by tauchunfall » Fri, Jan 05, 2024, 12:38 AM UTC

troupo wrote: post #159

kaba0 wrote: post #75

Earlier quoted context omitted.

What step are we talking about? Javac itself is absolutely on the same order of magnitude speed as Go per loc, while rust is significantly slower (which makes sense, the latter is a properly optimizing compiler with heavy static analysis, while the former two just spews out java byte code/machine code). Gradle with a daemon is also pretty fast, you are just probably used to some complex project with hundreds of depen…

> Gradle with a daemon is also pretty fast I've yet to see a project where Gradle daemon a) does anything useful and b) is acutally used by gradle itself (instead of seemingly doing everything from scratch, no idea what it does in the seconds it takes for it to start up).

Post by pbh101 » Fri, Jan 05, 2024, 2:24 AM UTC

bayindirh wrote: post #29

pbh101 wrote: post #13

> Each contender will be run five times in a row. The slowest and the fastest runs are discarded. The mean value of the remaining three runs is the result for that contender and will be added to the leaderboard. Shouldn’t he take the single fastest time, assuming file (and JDK) being in file cache is controlled for?

This is done to simulate real-world performance. Your binary is not the only binary in the system and other services may be running as well. So fastest time is the happiest path and slowest is the unluckiest. The range of remaining three is what you expect to get 99% of the time on a real world system.