161–170 of 366 posts
Earlier quoted context omitted.
Java build times are very fast. You are just measuring your internet speed here. (Also, gradle is faster as a build tool for incremental compilation)
Gradle is faster than what? Than maven? Maybe. But not than Go or Cargo.
- Read with O_DIRECT (but compare it with the mmap approach). I know O_DIRECT gets much hate, but this could be one of the rare cases where it helps.
- Use a simple array with sentinel and linear search for the station names. This is dumb, but if the number of stations is small enough this could beat the hash. (In the back of my head I have a rule of thumb that linear search is faster for up to 1000 elements, but I'm not sure anymore where I got this from).
Earlier quoted context omitted.
1 billion is small for hadoop?
Anything that fits in RAM on one machine is easily too small for Hadoop. In those cases, the overhead of Hadoop is going to make it get destroyed by a single beefy machine. The only times where this might not be the case is when you're doing a crazy amount of computation relative to the data you have. Note that you can easily reach 1TB of RAM on (enterprise) commodity hardware now, and SSDs are pretty fast too. Old b…
> Q: Can I make assumptions on the names of the weather stations showing up in the data set? > A: No, while only a fixed set of station names is used by the data set generator, any solution should work with arbitrary UTF-8 station names (for the sake of simplicity, names are guaranteed to contain no `;` character). I'm unsure if it's intentional or not, but this essentially means that the submission should be correct…
Earlier quoted context omitted.
What step are we talking about? Javac itself is absolutely on the same order of magnitude speed as Go per loc, while rust is significantly slower (which makes sense, the latter is a properly optimizing compiler with heavy static analysis, while the former two just spews out java byte code/machine code). Gradle with a daemon is also pretty fast, you are just probably used to some complex project with hundreds of depen…
> Gradle with a daemon is also pretty fast I've yet to see a project where Gradle daemon a) does anything useful and b) is acutally used by gradle itself (instead of seemingly doing everything from scratch, no idea what it does in the seconds it takes for it to start up).
Unfortunately I have no time to code this up but things that I would try to make it fast: - Read with O_DIRECT (but compare it with the mmap approach). I know O_DIRECT gets much hate, but this could be one of the rare cases where it helps. - Use a simple array with sentinel and linear search for the station names. This is dumb, but if the number of stations is small enough this could beat the hash. (In the back of my…
When you want real speed like this and are happy with insane complexity, code that writes and self-compiles a hash function based on the data can make sense.
Earlier quoted context omitted.
Itself. > gradle is faster as a build tool for incremental compilation Implicit: > …than it is building from scratch, where it needs to download lots of stuff I mean, yes, saying Java builds are fast does seem a bit “rolls eyes, yes technically by loc when the compiler is actually running” …but, ^_^! let’s not start banging on about how great cargo/rust compile times are… they’re really terrible once procedural macro…
So maybe I was just unlucky with gradle and lucky with cargo. A project that is a mixture of 20k LoC Scala and 300k LoC Java took 6 minutes to compile, and incremental was still several tens of seconds. Cargo/Rust cold compiles a project of 1M LoC (all dependencies) in about 1:30 on the same hardware and incremental is like 2-4 seconds. As for precedural macros - yes they can be slow to compile but so are Java annota…
In my experience, java annotations gets processed very fast.
Earlier quoted context omitted.
Then someone in your team failed learning the bare minimum to write a sane gradle file, and are putting imperative stuff into the configuration phase. For big projects, compiling only what’s necessary is a huge time win.
> Then someone in your team failed learning the bare minimum to write a sane gradle file Then someone in over 15 years and 8 versions gradle failed to make a system that doesn't require "learning to write sane gradle files" to make sure that it's only two orders of magnitude and not five orders of magnitude slower than it can be. > and are putting imperative stuff into the configuration phase Has nothing to do with t…
first search result for averaging streaming data: https://nestedsoftware.com/2018/03/20/calculating-a-moving-a... so you just walk the file and read a chunk, update the averages and move on. the resource usage should be 0.000 nothing and speed should be limited by your disk IO.
actually i think you can also just average each chunk and then add it to existing data. like read N rows(say all have one location to keep it simple), average the data from the chunk, update/save min and max, move on to next chunk, do the same but now update the average by adding to existing/previously computed average and divide by two. the result will be the same - disk IO will be the most limiting aspect. this "ch…
I.e. avg of {22.5, 23, 24} = 23.17... But:
1. 22.5
2. (22.5 + 23)/2 = 22.75
3. (22.75 + 24)/2 = 23.375
Just for fun. Speed testing awk vs Java. awk -F';' '{ station = $1 temperature = $2 sum[station] += temperature count[station]++ if (temperature max[station] || count[station] == 1) { max[station] = temperature } } END { for (s in sum) { mean = sum[s] / count[s] printf "{%s=%.1f/%.1f/%.1f", s, min[s], mean, max[s] printf (s == PROCINFO["sorted_in"][length(PROCINFO["sorted_in"])] ? "}\n" : ", ") } }' measurement.txt