Live data from Hacker News

Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

notamonadtutorial.com

81–90 of 117 posts

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#81

Earlier quoted context omitted.

If these things interest you, check out Julia

how do you know someone uses Julia? don't worry they'll tell you.

The same could be said for every Rust fanboy that feels the need to mention Rust whenever an article about C or anything implemented in C is brought up.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#82
post #76

Earlier quoted context omitted.

You care.

Are you coding as a hobby or a profession? If you are a professional, you will use the most effective tool for the job - to get results. What tool will produce the best results - schedule, budget, quality, maintainability, scalabi, portability, etc.? Other than outliers that will crush your productivity, or multiply it, your feelings are pretty irrelevant. Similarly, when you get into a racecar, your feelings about y…

> What tool will produce the best results - schedule, budget, quality, maintainability, scalabi, portability, etc.?

Old, boring, mature tools where the limitations are well understood.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#83
post #76

Earlier quoted context omitted.

You care.

Are you coding as a hobby or a profession? If you are a professional, you will use the most effective tool for the job - to get results. What tool will produce the best results - schedule, budget, quality, maintainability, scalabi, portability, etc.? Other than outliers that will crush your productivity, or multiply it, your feelings are pretty irrelevant. Similarly, when you get into a racecar, your feelings about y…

> Are you coding as a hobby or a profession?

This isn’t mutually exclusive, surely. If anything you’d expect it to be quite closely correlated.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#84
post #79
post #51

Earlier quoted context omitted.

Not that you should replace Rust with Haskell, but Haskell would've been a better choice than Scala. It has its own runtime, but it's not difficult to call Haskell code from C or ATS or whatever.

Not difficult to call from C? How does that work, exactly? Wouldn't you need to properly setup the whole runtime (incl. GC) first?

[deleted]

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#85

Earlier quoted context omitted.

how do you know someone uses Julia? don't worry they'll tell you.

The same could be said for every Rust fanboy that feels the need to mention Rust whenever an article about C or anything implemented in C is brought up.

lets generalize it further: People will take chances to bring up what they are passionate about.

I can relate to this, it is not unreasonable for me.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#86
> The motivation behind Weld is to provide bare-metal performance for applications that rely on existing high-level APIs such as NumPy and Pandas.

With regard to Pandas this makes me pause slightly, since, while pandas contains lots of high quality and high performance implementations, the API of pandas in some places doesn’t feel well-designed (the most obvious example is indexing of data frames via square brackets and the various properties like iloc).

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#87

Earlier quoted context omitted.

Weld is a compiler/JIT/runtime though, something Rust is very well suited for, and which is very different code from data analysis/ML. I think Julia is a more interesting language for this space, with the built in matrix support, easier prototyping, a REPL, etc...

Rust is great, but this is an important comment! We used it to implement Weld's compiler and runtime, but we don't expect data scientists who use languages such as Python, Julia, or R to switch over to it; the idea is that these data scientists continue using APIs in these languages, and under the hood, Weld will perform optimizations and compilation for decreasing execution time (and these "under the hood" component…

Would Weld be able to do a better job if these scientist were using a Rust library instead ?

A lot of people would like to use Rust for data-analysis / machine learning, but there are not really any good batteries-included frameworks for getting started with this.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#88
post #56
post #3

Earlier quoted context omitted.

from TFA: >> We chose Rust because: >> It has a very minimal runtime (essentially just bounds checks on arrays) and is easy to embed into other languages such as Java and Python >> It contains functional programming paradigms such as pattern matching that make writing code such as pattern matching compiler optimizations easier >> It has a great community and high quality packages (called “crates” in Rust) that made d…

Did they have to write the runtime in the same language? They could've used an ML with GC and it would've been better (for a compiler). It doesn't really have any functional programming paradigms. Pattern matching is present in imperative languages like past versions of ATS

> Did they have to write the runtime in the same language?

No, they originally wrote the runtime in C++, but ended up re-implementing it in Rust because the C++ runtime had too many bugs.

> They could've used an ML with GC and it would've been better (for a compiler).

They originally wrote the compiler in Scala with the JVM GC, and they said it was much slower and much harder to embed.

> It doesn't really have any functional programming paradigms. Pattern matching is present in imperative languages like past versions of ATS

When choosing a language for such a project, there are many engineering trade-offs that must be evaluated.

The Weld project has dozens of developers that need onboarding, documentation, examples, tooling, etc. Rust has a lower barrier of entry than ATS. One of the main things Weld does is interfacing with LLVM: this is one of the main things the Rust compiler does and Rust has great libraries for this. Another thing Weld does is interfacing with many dynamic languages (Python, R, etc.). Rust does not just have "C FFI". It also has _a lot_ of great tooling for automatically generating and validating all the boilerplate automatically. Finally, performance and code size of Weld is one of the main advantages over the alternatives. Rust generates reasonable code with LLVM, ATS has its own code machine code generator, which while reasonable, isn't as good.

Finally, it is hard to find people who enjoy writing ATS code. They exist, but are not many. Even when you do find them, they often don't like collaborating with people (I only know one ATS user, vmchale on github, and they don't really like interacting with others). OTOH it is trivial to find lots of people that enjoy writing Rust with others. It doesn't matter if this is due to technical reasons, marketing, or hype, but it's a fact that you have to consider if you want a project to grow fast.

Re: Weld: Accelerating numpy, scikit and pandas as much as 100x with Rust and LLVM

#90

Earlier quoted context omitted.

Not to sound like a member of the Rust evangelism strike force, but after using Rust for a couple years, I don't have any desire to go back to C - sum types alone are worth the switch to me, not to mention iterators, concurrency story, etc.

C++ has sum types as of several years ago with std::variant.

> C++ has sum types as of several years ago with std::variant.

Haha, that's like saying that C has templates because it has the _Generic macro.

Post reply on HN