The One Billion Row Challenge

Post by Algunenano » Thu, Jan 04, 2024, 12:29 PM UTC

solarized wrote: post #123

Just for fun. Speed testing awk vs Java. awk -F';' '{ station = $1 temperature = $2 sum[station] += temperature count[station]++ if (temperature max[station] || count[station] == 1) { max[station] = temperature } } END { for (s in sum) { mean = sum[s] / count[s] printf "{%s=%.1f/%.1f/%.1f", s, min[s], mean, max[s] printf (s == PROCINFO["sorted_in"][length(PROCINFO["sorted_in"])] ? "}\n" : ", ") } }' measurement.txt

    time clickhouse local -q "SELECT concat('{', arrayStringConcat(groupArray(v), ', '), '}')
    FROM
    (
        SELECT concat(station, '=', min(t), '/', max(t), '/', avg(t)) AS v
        FROM file('measurements.txt', 'CSV', 'station String, t Float32')
        GROUP BY station
        ORDER BY station ASC
    )
    SETTINGS format_csv_delimiter = ';', max_threads = 8" >/dev/null

    real 0m15.201s
    user 2m16.124s
    sys 0m2.351

Post by pkolaczk » Thu, Jan 04, 2024, 12:37 PM UTC

kaba0 wrote: post #167

pkolaczk wrote: post #150

Earlier quoted context omitted.

So maybe I was just unlucky with gradle and lucky with cargo. A project that is a mixture of 20k LoC Scala and 300k LoC Java took 6 minutes to compile, and incremental was still several tens of seconds. Cargo/Rust cold compiles a project of 1M LoC (all dependencies) in about 1:30 on the same hardware and incremental is like 2-4 seconds. As for precedural macros - yes they can be slow to compile but so are Java annota…

Scala is a much different case, it has one of the most advanced type systems, many implicit scoping rules etc. Even a java change that might affect the scala codebase could result in a slow incremental build. Also, the build system may not have been as modular as it could. In my experience, java annotations gets processed very fast.

Post by anonymoushn » Thu, Jan 04, 2024, 12:42 PM UTC

londons_explore wrote: post #144

rockwotj wrote: post #128

Earlier quoted context omitted.

A few things: Disk access can certainly be parallelized, and NVMe is blazing fast, so the bottleneck is more the CPU than disk. There are systems that are built around around modern hardware that realize this (redpanda.com, where I work is one such example) Parsing is a lot of the compute time and some SIMD tricks like SWAR for finding delimiters can be helpful. Stringzilla is a cool library if you want to see a clea…

I would hope that any reasonably performant implementation would be faster not only than NVMe, but also faster than CPU to RAM data transfers. The AMD EPYC-Milan in the test server supports memory reads at 150 gigabytes/sec, but thats a 32 core machine, and our test only gets 8 of those cores, so we probably can't expect more than 37 gigabytes per second of read bandwidth. The total file is ~12 gigabytes, so we shoul…

Post by troupo » Thu, Jan 04, 2024, 12:51 PM UTC

kaba0 wrote: post #168

troupo wrote: post #156

Earlier quoted context omitted.

> Then someone in your team failed learning the bare minimum to write a sane gradle file Then someone in over 15 years and 8 versions gradle failed to make a system that doesn't require "learning to write sane gradle files" to make sure that it's only two orders of magnitude and not five orders of magnitude slower than it can be. > and are putting imperative stuff into the configuration phase Has nothing to do with t…

It is your own failure if you beat the tool over yourself, for not having learnt it.

Post by blackoil » Thu, Jan 04, 2024, 1:10 PM UTC

pkolaczk wrote: post #155

kaba0 wrote: post #75

Earlier quoted context omitted.

What step are we talking about? Javac itself is absolutely on the same order of magnitude speed as Go per loc, while rust is significantly slower (which makes sense, the latter is a properly optimizing compiler with heavy static analysis, while the former two just spews out java byte code/machine code). Gradle with a daemon is also pretty fast, you are just probably used to some complex project with hundreds of depen…

> Javac itself is absolutely on the same order of magnitude speed as Go per loc Not really when it has to run on a cold JVM. And before it warms up, it’s already done. Then you are in territory of keeping the compiler process between the runs, but that is a memory hog and also not always realiable (gradle often decides to run a fresh daemon for whatever reason). So theoretically, in lab conditions yes, in practice no…