41–50 of 75 posts
Obviously not the point of the post, but this doesn't seem like a very reasonable thing to calculate, statistically speaking. Also, I wonder if there is some way to use branch-and-bound to look at fewer combinations.
It's well known that Rust's default HashMap hashing function is slow because it's designed to be safe against DoS [1]. If you want your HashMap to be faster, just use a faster hashing function like ahash[2], which can be up to 40% faster.
[1] https://doc.rust-lang.org/book/ch08-03-hash-maps.html#hashin...
Earlier quoted context omitted.
articles like this aren't pointless, i don't think it is meant to be framed as a fair comparison of "rust-the-language is 5 orders of magnitude faster than python-the-language". this article does offer examples of how to use a profiler and some ideas of what kind of things can be bottlenecks and how to eliminate them, and also helps people who only ever work with very slow programming tech stacks to understand what k…
Intentional or not, it does come off as kind of “disruptive” in the negative sense of the word, ie “you’re all doing it wrong in field X. Let me show you how to work faster”. But in a real scenario there are often so many other constraints that “optimized code” is far from the top. Salaries are also expensive, so if you factor in the lifetime cost of writing and maintaining a bit of exotic Rust for an exotic problem,…
good engineering should figure out what the main constraints are, and solve for those, while doing a "good enough" job for everything else and no more -- like you allude to, the main bottlenecks to solve for are usually things like reducing engineering effort, reducing schedule, reducing long term maintenance cost of the overall system (not just this isolated component) - especially how easy it is to hire someone who can understand and fix the thing.
but, this article isn't a "how to make good whole-system engineering tradeoffs for the long-term benefit or your employer / client" article, it's an article showing how to write fast rust code.
But I'd have thought there must be heuristics to help narrow the search.
For example you could calculate how each question correlates to all the other questions[1]. If you picked the set of k questions with the best correlation, then you have a good sense of which will be in the actual "k-corr set".
You could widen your search a little bit, but by chopping out any questions which correlate poorly to the overall score, you narrow your search greatly, going from 200 choose 5 to 10 choose 5 is a speedup factor of a million, and you've probably considered all the sets that are likely to be in the final bucket.
It's been a while since I've worked in the domain, but OP might also want to check out https://en.wikipedia.org/wiki/Item_response_theory and https://en.wikipedia.org/wiki/Classical_test_theory.
[1] https://en.wikipedia.org/wiki/Point-biserial_correlation_coe...
The author found that HashMap::get was taking most of the time and didn't swap the hashing function?? It's well known that Rust's default HashMap hashing function is slow because it's designed to be safe against DoS [1]. If you want your HashMap to be faster, just use a faster hashing function like ahash[2], which can be up to 40% faster. [1] https://doc.rust-lang.org/book/ch08-03-hash-maps.html#hashin... [2] https:/…
> Note that everywhere we refer to HashMap is actually an alias for fxhash::FxHashMap, which is just std::collections::HashMap with a more efficient hashing algorithm
> the point of this post isn’t to compare highly-optimized Python to highly-optimized Rust. The point is to compare “standard-Jupyter-notebook” Python to highly-optimized Rust. I guess the title gets clicks, but I'm curious how good python gets. I'm under the impression pandas is pretty fast despite it being python
Pandas can be pretty fast, but DuckDB and Polars are both faster than Pandas. DuckDB supports vectorized and parallelized operations on Pandas dataframes, while Polars is written in Rust. I feel though the killer is that inner loop where dataframe operations are being performed across a large number of iterations, and there's significant overhead there. For-loops are usually not the most performant solution in Python…
Basically "Analyzing Data 180,000x Faster with AOT strongly typed compiled language" versus using a dynamically typed interpreted one.
Basically "Analyzing Data 180,000x Faster with AOT strongly typed compiled language" versus using a dynamically typed interpreted one.
The post deals a lot both with strings and maps with strings as keys. One idea is to intern these strings using an interer like [1] which returns string keys as monotonically increasing integers starting at 0. Then instead of a map you can just have a regular vector with the interned key as the index into the vector. This gives you the map functionality with very good performance.
Comparing highly optimized code (including total algorithm rewrite and relying on unsafe and SIMD operations) without doing the same on the other side is a pointless exercise. It's like showing how much faster you can get your handcrafted assembly code to run vs a bash script.