Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

notamonadtutorial.com

Post reply on HN
Print view View original

51–60 of 117 posts

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#51

Post by vmchale »

spenrose wrote: post #20

"the first implementation was in Scala, which was chosen because of its algebraic data types and powerful pattern matching. This made writing the optimizer, which is the core part of the compiler, very easy. Our original optimizer was based on the design of Catalyst, which is Spark SQL’s extensible optimizer. We moved away from Scala because it was too difficult to embed a JVM-based language into other runtimes and l…

Not that you should replace Rust with Haskell, but Haskell would've been a better choice than Scala.

It has its own runtime, but it's not difficult to call Haskell code from C or ATS or whatever.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#53

Post by tclancy »

shpongled wrote: post #25
dlphn___xyz wrote: post #2

whats the benefit of rust over julia or C for computation?

Not to sound like a member of the Rust evangelism strike force, but after using Rust for a couple years, I don't have any desire to go back to C - sum types alone are worth the switch to me, not to mention iterators, concurrency story, etc.

I can’t decide if Rust Evangelism Strike Force sounds like an awful or awesome cartoon.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#54

Post by vmchale »

the_duke wrote: post #23
axegon_ wrote: post #11

I have said multiple times that Rust has an incredible potential in the data analysis world. And Weld is a great example.

Weld is a compiler/JIT/runtime though, something Rust is very well suited for, and which is very different code from data analysis/ML. I think Julia is a more interesting language for this space, with the built in matrix support, easier prototyping, a REPL, etc...

Lots of languages are well-suited to compilers. See e.g. Haskell accelerate.

I don't really know why you'd use Rust instead of a GC language from the ML family.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#56

Post by vmchale »

jdc wrote: post #3
dlphn___xyz wrote: post #2

whats the benefit of rust over julia or C for computation?

from TFA: >> We chose Rust because: >> It has a very minimal runtime (essentially just bounds checks on arrays) and is easy to embed into other languages such as Java and Python >> It contains functional programming paradigms such as pattern matching that make writing code such as pattern matching compiler optimizations easier >> It has a great community and high quality packages (called “crates” in Rust) that made d…

Did they have to write the runtime in the same language?

They could've used an ML with GC and it would've been better (for a compiler).

It doesn't really have any functional programming paradigms. Pattern matching is present in imperative languages like past versions of ATS

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#59

Post by pard68 »

sieabahlpark wrote: post #40
westurner wrote: post #6

There's also RustPython, a Rust implementation of CPython 3.5+: https://news.ycombinator.com/item?id=20686580 > https://github.com/RustPython/RustPython

Why didn't they call it Rython

Convention wound lean towards calling it RPython

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#60

Post by mlthoughts2018 »

marmaduke wrote: post #48
mlthoughts2018 wrote: post #10

Very bizarre there is no discussion of numba here, which has been around and used widely for many years, achieves faster speedups than this, and also emits an LLVM IR that is likely a much better starting point for developing a “universal” scientific computing IR than doing yet another thing that further complicates it with fairly needless involvement of Rust. https://numba.pydata.org/

Did you read the article? If you know of Numba works, you know it can't just pick up different functions from sklearn and scipy and do interprocedural optimization (IPO). For Numba to do that, it'd need all functions involved to be written in Numba @jit style, whereas Weld would work directly on the pre-existing functions. Rust is just a IPO driver of sorts here. I'm not critizing Numba btw, I use it regularly, but y…

I don’t agree that the purpose of the article is misaligned from my criticism. This is based on reading the article.
Post reply on HN