61–70 of 117 posts
whats the benefit of rust over julia or C for computation?
The computation here is actually not done in Rust. The Rust code is performing stages of compilation of the original source code, into an intermediate representation that LLVM finishes compiling. The fully compiled code is what does the computation.
Earlier quoted context omitted.
The computation here is actually not done in Rust. The Rust code is performing stages of compilation of the original source code, into an intermediate representation that LLVM finishes compiling. The fully compiled code is what does the computation.
SIMD / intrinsics?
This post combines pretty much every technology I'm obsessed with right now: Python, Rust, Pandas, Numpy, and LLVM. Yess!!!
Just a word of caution, always obsess with product and customer needs first :) In ML/data science tech first normally won’t end up well
"the first implementation was in Scala, which was chosen because of its algebraic data types and powerful pattern matching. This made writing the optimizer, which is the core part of the compiler, very easy. Our original optimizer was based on the design of Catalyst, which is Spark SQL’s extensible optimizer. We moved away from Scala because it was too difficult to embed a JVM-based language into other runtimes and l…
In addition to the JVM, Scala has had JS [1] and native (via LLVM) [2] targets for years.
(And that's not even mentioning any second-order compilations; e.g. Scala -> JVM bytecode -> native)
There's a number of reasons to not choose Scala, but portability is far from one of them.
"the first implementation was in Scala, which was chosen because of its algebraic data types and powerful pattern matching. This made writing the optimizer, which is the core part of the compiler, very easy. Our original optimizer was based on the design of Catalyst, which is Spark SQL’s extensible optimizer. We moved away from Scala because it was too difficult to embed a JVM-based language into other runtimes and l…
Very interesting. Do you have any references to share relevant to the article you suggest to be written?
Earlier quoted context omitted.
Very interesting. Do you have any references to share relevant to the article you suggest to be written?
No, I based on attending SparkConf between 2015 and 2017. You could probably assemble half of it just by reading summaries of Matei's keynotes.
Very bizarre there is no discussion of numba here, which has been around and used widely for many years, achieves faster speedups than this, and also emits an LLVM IR that is likely a much better starting point for developing a “universal” scientific computing IR than doing yet another thing that further complicates it with fairly needless involvement of Rust. https://numba.pydata.org/
Numba is amazing. +1 for numba
Earlier quoted context omitted.
Just a word of caution, always obsess with product and customer needs first :) In ML/data science tech first normally won’t end up well
It's important to enjoy your work, which is - among other causes - about having right tools. Also, some of us actually get to have some influence over what language we write our projects in.
Earlier quoted context omitted.
Not to sound like a member of the Rust evangelism strike force, but after using Rust for a couple years, I don't have any desire to go back to C - sum types alone are worth the switch to me, not to mention iterators, concurrency story, etc.
C++ has sum types as of several years ago with std::variant.
You would have to keep everything in variants, or wrap/unwrap manually all over the place to get similar functionality.
And C has tagged unions.