31–40 of 88 posts
docs: https://docs.greptime.com/user-guide/coprocessor-and-scripti...
code: https://github.com/GreptimeTeam/greptimedb/tree/develop/src/...
Earlier quoted context omitted.
Numpy is often faster because it’s often using highly optimized simd, or makes use of BLAS/Fortran/LAPACK/MKL/CuBLAS implementations. A pure rust implementation will likely always be slower by virtue of not using the same tightly designed optimized code. Side note: A fun implementation detail of numpy is that after you install it from pypi, it does a user side compile of some of the modules on first import. Which mea…
And note that for example on Apple M1 it's essentially impossible to beat an implementation that uses Apple's Accelerate library for things like matrix multiplication, because Apple uses undocumented instructions unavailable to the public in that library.
Deprecated since version 1.20: The native libraries on macOS, provided by Accelerate, are not fit for use in NumPy since they have bugs that cause wrong output under easily reproducible conditions. If the vendor fixes those bugs, the library could be reinstated, but until then users compiling for themselves should use another linear algebra library or use the built-in (but slower) default, see the next section.
This is truly a fantastic combination -- implement the logic in Rust and use it in Python. GreptimeDB also implements a similar functionality that allows writing Python script to do post-process of SQL query results, with the help of RustPython and Arrow. Maybe this combination can bring a sweet point between performance and efficiency. docs: https://docs.greptime.com/user-guide/coprocessor-and-scripti... code: https…
CREATE FUNCTION pymax (a integer, b integer)
RETURNS integer
AS $$
if a > b:
return a
return b
$$ LANGUAGE plpython3u;
This is truly a fantastic combination -- implement the logic in Rust and use it in Python. GreptimeDB also implements a similar functionality that allows writing Python script to do post-process of SQL query results, with the help of RustPython and Arrow. Maybe this combination can bring a sweet point between performance and efficiency. docs: https://docs.greptime.com/user-guide/coprocessor-and-scripti... code: https…
Python 3 can be executed server side by Postgres as well CREATE FUNCTION pymax (a integer, b integer) RETURNS integer AS $$ if a > b: return a return b $$ LANGUAGE plpython3u; https://www.postgresql.org/docs/current/plpython.html
I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get. If you test a single array operation with NumPy then it will compare favourably to a low level implementat…
Including another for would spell out to "for for example"
I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get. If you test a single array operation with NumPy then it will compare favourably to a low level implementat…
NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scrip…
Earlier quoted context omitted.
NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scrip…
How do you handle it in numpy if you want something like ((x - y)/2 + z) * w all elementwise over the arrays? Naively that's 3 intermediate unused arrays being created.
* Numba: https://numba.pydata.org/
* JAX: https://jax.readthedocs.io/en/latest/notebooks/quickstart.ht...
Earlier quoted context omitted.
How do you handle it in numpy if you want something like ((x - y)/2 + z) * w all elementwise over the arrays? Naively that's 3 intermediate unused arrays being created.
There's plenty of CPU and GPU numpy accelerators available. * Numba: https://numba.pydata.org/ * JAX: https://jax.readthedocs.io/en/latest/notebooks/quickstart.ht...
I've been through the journey with compiling extensions to Python in lower languages (primarily C and C++) and it's really worth evaluating the benefit. In general, the control that it gives you over memory layout and chaining of operations make it very difficult to say from toy examples what benefit you'll get. If you test a single array operation with NumPy then it will compare favourably to a low level implementat…
NumPy is insanely fast for most use cases, I've (grudgingly) come to the conclusion that if NumPy doesn't hack it then I should re-think the problem or my way of solving it rather than to try to optimize that particular bit of code if it isn't meant for something that is going to be run in production on a large number of machines. Likely there are better uses of my time. It's interesting how what is nominally a scrip…
I'd recommend using Numba first, though.