I have said multiple times that Rust has an incredible potential in the data analysis world. And Weld is a great example.
Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM
11–20 of 117 posts
Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM
#12whats the benefit of rust over julia or C for computation?
in this particular case it's to be a drop in replacement for the python datascientist, the rest is irrelevant I believe
Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM
#13There's also RustPython, a Rust implementation of CPython 3.5+: https://news.ycombinator.com/item?id=20686580 > https://github.com/RustPython/RustPython
Is this basically what Cython and PyPy are trying to do, but with Rust?
Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM
#14I have said multiple times that Rust has an incredible potential in the data analysis world. And Weld is a great example.
If only you could get paid for porting open source libraries, maintaining them and trying to get a community to use it
Have fun entertaining your Patreon
Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM
#15whats the benefit of rust over julia or C for computation?
None
On the other side you'll gain all the instability that a young language suffer from, like in 6 months it could be impossible to compile this code and you'll need to refactor it
It is also heavily used only at Mozilla , which is not a strong guarantee of longevity
Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM
#16Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM
#17Very bizarre there is no discussion of numba here, which has been around and used widely for many years, achieves faster speedups than this, and also emits an LLVM IR that is likely a much better starting point for developing a “universal” scientific computing IR than doing yet another thing that further complicates it with fairly needless involvement of Rust. https://numba.pydata.org/
Hi, I am the interviewer. I think I saw numba once but forgot about it. I will check it and probably ask to interview them too. We are preparing interviews about RAPIDS and other similar projects too.
Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM
#18Earlier quoted context omitted.
Is this basically what Cython and PyPy are trying to do, but with Rust?
PyPy is a JIT compiler. RustPython is an interpreter.
And Cython is an AOT compiler for a superset of Python.
RustPython seems to be modestly aiming for a reimplementation of CPython.
Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM
#19I have said multiple times that Rust has an incredible potential in the data analysis world. And Weld is a great example.
If only you could get paid for porting open source libraries, maintaining them and trying to get a community to use it Have fun entertaining your Patreon
True that, hence the reason why I do it so little. In fact I personally do it out of pure boredom.
Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM
#20"the first implementation was in Scala, which was chosen because of its algebraic data types and powerful pattern matching. This made writing the optimizer, which is the core part of the compiler, very easy. Our original optimizer was based on the design of Catalyst, which is Spark SQL’s extensible optimizer. We moved away from Scala because it was too difficult to embed a JVM-based language into other runtimes and languages."
There is an important "contemporary history of computing" article to write about the evolution of the Spark project from "let's build a distributed filesystem for MapReduce in Java because we read those early Google papers" to "SQL is the right model for working with data so DataFrames" to "meet data scientists where they are: Python (and R)" to "make machine learning easy" and now to "LLVM, but for crunching big numeric arrays".