Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

notamonadtutorial.com

Post reply on HN
Print view View original

101–110 of 117 posts

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#101

Post by shaklee3 »

fluffything wrote: post #91
shaklee3 wrote: post #72

Earlier quoted context omitted.

Can you elaborate more? What do other languages have over std::variant/visit?

That would be like explaining C++ Concepts to an assembly programmer from the 60s that had never used a "function" as a way of abstracting code. If you really want to know, spend one afternoon learning any programming language with built in support for that (Rust, Ocaml, Haskell, ...). ADTs is one of the first things one learns. In Rust, the features you'd need to learn are enums, patterns, and pattern matching. But…

I know how these things work in Rust, and I'm still failing to see your point. It's not at all as complicated as what you are saying given that you can find many blog articles that explain it succinctly in a couple paragraphs.

It's not at all helpful to say that there's something so complicated on these other languages that you can't possibly get the idea across without using them.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#102

Post by infinite8s »

shaklee3 wrote: post #101
fluffything wrote: post #91

Earlier quoted context omitted.

That would be like explaining C++ Concepts to an assembly programmer from the 60s that had never used a "function" as a way of abstracting code. If you really want to know, spend one afternoon learning any programming language with built in support for that (Rust, Ocaml, Haskell, ...). ADTs is one of the first things one learns. In Rust, the features you'd need to learn are enums, patterns, and pattern matching. But…

I know how these things work in Rust, and I'm still failing to see your point. It's not at all as complicated as what you are saying given that you can find many blog articles that explain it succinctly in a couple paragraphs. It's not at all helpful to say that there's something so complicated on these other languages that you can't possibly get the idea across without using them.

One difference is that std::variant is run-time dispatched and uses memory equal to the max of all the variants, while Rust's sum types could potentially be compile-time dispatched and memory optimized to the exact type being used through various code paths.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#103

Post by infinite8s »

sppalkia wrote: post #31
mlthoughts2018 wrote: post #10

Very bizarre there is no discussion of numba here, which has been around and used widely for many years, achieves faster speedups than this, and also emits an LLVM IR that is likely a much better starting point for developing a “universal” scientific computing IR than doing yet another thing that further complicates it with fairly needless involvement of Rust. https://numba.pydata.org/

I'm one of the developers of Weld -- Numba is indeed very cool and is a great way to compile numerical Python code. Weld performs some additional optimizations specific to data science that Numba doesn't really target right now (e.g., fusing parallel loops across independently written functions, parallelizing hash table operations, etc.). We're also working on adding the ability to call Python functions from within W…

Although to be fair, there is no reason why numba couldn't gain those capabilities, it just hasn't been a focus of the project. It should be possible to build a lightweight modular staging system in python/numba similar to Scala's (https://scala-lms.github.io/) or Lua's (http://terralang.org/).

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#104

Post by pacala »

paulddraper wrote: post #65
spenrose wrote: post #20

"the first implementation was in Scala, which was chosen because of its algebraic data types and powerful pattern matching. This made writing the optimizer, which is the core part of the compiler, very easy. Our original optimizer was based on the design of Catalyst, which is Spark SQL’s extensible optimizer. We moved away from Scala because it was too difficult to embed a JVM-based language into other runtimes and l…

> it was too difficult to embed a JVM-based language into other runtimes and languages In addition to the JVM, Scala has had JS [1] and native (via LLVM) [2] targets for years. (And that's not even mentioning any second-order compilations; e.g. Scala -> JVM bytecode -> native) There's a number of reasons to not choose Scala, but portability is far from one of them. [1] https://www.scala-js.org [2] http://www.scala-na…

The library ecosystem is different between JVM/JS/Native. Porting across runtimes may require more work than just changing the compilation target.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#105

Post by turk73 »

toss1 wrote: post #76
naniwaduni wrote: post #71

Earlier quoted context omitted.

You care.

Are you coding as a hobby or a profession? If you are a professional, you will use the most effective tool for the job - to get results. What tool will produce the best results - schedule, budget, quality, maintainability, scalabi, portability, etc.? Other than outliers that will crush your productivity, or multiply it, your feelings are pretty irrelevant. Similarly, when you get into a racecar, your feelings about y…

I sort of disagree with your main assertion. I do big data for a living and what I have seen is that our architecture is dictated to us from above for reasons of "fashion" not really for any reasons of practicality.

I'm actually looking for a different job for that reason.

We are required to used Java on K8s, Kafka & Cassandra for every single solution big or small because it is fashionable, not because it gets the job done well or for any other reason. I can even demonstrate how a couple of Python scripts and Pandas could do all the same work with far less overhead and achieve the same results. Crickets. Python is not sexy where I am, it is the language of peasants, apparently. Not sure what to make of it all, but that is my reality right now.

Also, I don't think you know anything at all about driving race cars. The driver has a tremendous amount of input into the car's setup because it's his life on the line out on the track. "Adapting to the situation" gets finishes, not wins.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#106

Post by sgillen »

ris wrote: post #100

So... this requires cooperation from the underlying libraries (numpy, pandas...) - what is the likelihood of said libraries adopting this upstream vs Weld having to maintain their own shadow implementations for the foreseeable future? Numpy et al of course already have N python acceleration frameworks hammering at their doorsteps to integrate more closely...

How much cooperation is needed though? It seems to me that all that numpy pandas etc. need to do is maintain a stable API, which they already do AFAIK.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#107

Post by paulddraper »

pacala wrote: post #104
paulddraper wrote: post #65

Earlier quoted context omitted.

> it was too difficult to embed a JVM-based language into other runtimes and languages In addition to the JVM, Scala has had JS [1] and native (via LLVM) [2] targets for years. (And that's not even mentioning any second-order compilations; e.g. Scala -> JVM bytecode -> native) There's a number of reasons to not choose Scala, but portability is far from one of them. [1] https://www.scala-js.org [2] http://www.scala-na…

The library ecosystem is different between JVM/JS/Native. Porting across runtimes may require more work than just changing the compilation target.

That's true. If you want it across all platforms, you are restricted to the Scala ecosystem, and can't use Java and JS ecosystems.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#109

Post by [deleted] »

tempguy9999 wrote: post #97
L0stLink wrote: post #85

Earlier quoted context omitted.

lets generalize it further: People will take chances to bring up what they are passionate about. I can relate to this, it is not unreasonable for me.

No disrespect to you but passion in people can come from deep knowledge of a subject but IME much more commonly comes from lack of experience with alternatives. I've noticed the younger people are the more likely they are to be passionate, which I put down mainly to not knowing any better. Once one has more of the experience that places you higher up where you can see further, suddenly one's own plot of land doesn't…

[deleted]

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#110

Post by L0stLink »

tempguy9999 wrote: post #97
L0stLink wrote: post #85

Earlier quoted context omitted.

lets generalize it further: People will take chances to bring up what they are passionate about. I can relate to this, it is not unreasonable for me.

No disrespect to you but passion in people can come from deep knowledge of a subject but IME much more commonly comes from lack of experience with alternatives. I've noticed the younger people are the more likely they are to be passionate, which I put down mainly to not knowing any better. Once one has more of the experience that places you higher up where you can see further, suddenly one's own plot of land doesn't…

Does that justify the snide remarks? We should show more tolerance, there might not be a tool in existence that does not have its disadvantages along with its advantages. Healthy discussion around facts is beneficial, dismissing people for being passionate is not.
Post reply on HN