Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

notamonadtutorial.com

Post reply on HN
Print view View original

21–30 of 117 posts

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#21

Post by onei »

asjw wrote: post #15
dlphn___xyz wrote: post #2

whats the benefit of rust over julia or C for computation?

None On the other side you'll gain all the instability that a young language suffer from, like in 6 months it could be impossible to compile this code and you'll need to refactor it It is also heavily used only at Mozilla , which is not a strong guarantee of longevity

The Rust developers are working hard to ensure that no breaking changes ever appear in the language. As such old code would always be compilable, even if there's better ways to do it with modern features.

Rust learns the lessons that older languages taught us. Why would you think they'd fall into such an obvious pitfall given how long they took to get to v1?

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#22

Post by jcranmer »

asjw wrote: post #15
dlphn___xyz wrote: post #2

whats the benefit of rust over julia or C for computation?

None On the other side you'll gain all the instability that a young language suffer from, like in 6 months it could be impossible to compile this code and you'll need to refactor it It is also heavily used only at Mozilla , which is not a strong guarantee of longevity

Rust guarantees stability on any feature in the stable release of the compiler. And anything in the standard library will remain in there in perpetuity, even if deprecated.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#23

Post by the_duke »

axegon_ wrote: post #11

I have said multiple times that Rust has an incredible potential in the data analysis world. And Weld is a great example.

Weld is a compiler/JIT/runtime though, something Rust is very well suited for, and which is very different code from data analysis/ML.

I think Julia is a more interesting language for this space, with the built in matrix support, easier prototyping, a REPL, etc...

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#24

Post by FreakLegion »

unbalancedparen wrote: post #17
mlthoughts2018 wrote: post #10

Very bizarre there is no discussion of numba here, which has been around and used widely for many years, achieves faster speedups than this, and also emits an LLVM IR that is likely a much better starting point for developing a “universal” scientific computing IR than doing yet another thing that further complicates it with fairly needless involvement of Rust. https://numba.pydata.org/

Hi, I am the interviewer. I think I saw numba once but forgot about it. I will check it and probably ask to interview them too. We are preparing interviews about RAPIDS and other similar projects too.

Numba is the option used in Lectures in Quantitative Economics with Python, posted and highly upvoted here yesterday: https://news.ycombinator.com/item?id=21022620.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#25

Post by shpongled »

dlphn___xyz wrote: post #2

whats the benefit of rust over julia or C for computation?

Not to sound like a member of the Rust evangelism strike force, but after using Rust for a couple years, I don't have any desire to go back to C - sum types alone are worth the switch to me, not to mention iterators, concurrency story, etc.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#26

Post by laughinghan »

yorwba wrote: post #18
sp332 wrote: post #16

Earlier quoted context omitted.

PyPy is a JIT compiler. RustPython is an interpreter.

And Cython is an AOT compiler for a superset of Python. RustPython seems to be modestly aiming for a reimplementation of CPython.

Maybe "dialect" would be more accurate than "superset"? I don't think Cython is technically a superset of Python, since I think runtime metaprogramming features like __dict__ and monkey-patching are significantly altered or restricted?

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#27

Post by tmostak »

Also worth checking out OmniSci (formerly MapD), which features an LLVM query compiler to gain large speedups executing SQL on both CPU and GPU: https://github.com/omnisci/omniscidb . And here's a link to a blog post giving a high level overview of the advantages of JIT compilation of queries over an interpreter: https://devblogs.nvidia.com/mapd-massive-throughput-database... .

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#28

Post by noobermin »

dlphn___xyz wrote: post #2

whats the benefit of rust over julia or C for computation?

https://www.jwz.org/doc/cadt.html

Not quite Julia, but for C/Fortran particularly for computational uses, if your issue is multiple array scans, it might be time to look at rolling it yourself and then calling your algo from python.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#29

Post by loeg »

laughinghan wrote: post #26
yorwba wrote: post #18

Earlier quoted context omitted.

And Cython is an AOT compiler for a superset of Python. RustPython seems to be modestly aiming for a reimplementation of CPython.

Maybe "dialect" would be more accurate than "superset"? I don't think Cython is technically a superset of Python, since I think runtime metaprogramming features like __dict__ and monkey-patching are significantly altered or restricted?

Yeah, I was going to make a similar comment. It's a dialect of CPython, and certainly there are extensions required to make it usable. But I'm not sure it is a strict superset of the full Python language.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#30

Post by blts »

mlthoughts2018 wrote: post #10

Very bizarre there is no discussion of numba here, which has been around and used widely for many years, achieves faster speedups than this, and also emits an LLVM IR that is likely a much better starting point for developing a “universal” scientific computing IR than doing yet another thing that further complicates it with fairly needless involvement of Rust. https://numba.pydata.org/

Numba is amazing. +1 for numba
Post reply on HN