Flashlight: Fast and flexible machine learning in C++ • Hacker News Board

Re: Flashlight: Fast and flexible machine learning in C++

#61

Post by flemishgun » Sat, Apr 17, 2021, 1:16 AM UTC

dynamite-ready wrote: post #32

Does anyone else think that C++ makes more sense for ML work than Python? I'd been thinking so for years. Both for deployment/performance and data wrangling purposes.

100% agree, and there are a number of efforts in the space. mlpack (https://www.github.com/mlpack/mlpack/), Shogun (https://www.shogun-toolbox.org/), and Shark (https://www.shark-ml.org/) are three that have been around for over a decade now. They're a little niche because C++ is not that popular for data science, but they are generally pretty fast (especially mlpack, which focuses on speed).

Re: Flashlight: Fast and flexible machine learning in C++

#62

Post by flemishgun » Sat, Apr 17, 2021, 1:17 AM UTC

Very cool to see more C++-based machine learning efforts. The language still needs a good dataframe abstraction (maybe XFrame?), but with matrix algebra provided by Armadillo/Eigen and other long-time machine learning libraries like mlpack, Shogun, Shark (and if you want to include C, Darknet), personally I think the future is bright for machine learning in C++---especially for production and deployment applications.

Re: Flashlight: Fast and flexible machine learning in C++

#63

Post by Kranar » Sat, Apr 17, 2021, 1:42 AM UTC

stephc_int13 wrote: post #53

Given the very high computing requirements of Machine Learning, I've always be perplexed by the seemingly widespread and unquestionable preference of Python over native code (typically C/C++). I guess performance was considered less critical than clarity/flexibility. But it seems that people are discovering that complex code tend to be hard to read/modify no matter the language...

I work at a quant firm and we use Python because of how painful it is to build things in C++. Our framework is built in C++ but we then expose all of it to Python using pybind11 (amazing library).

Most quants do not want to learn complex build systems that have quirky behavior on different platforms, wait for very long compile times when making small changes to the code, dense and incomprehensible error messages, and a host of painful problems that one has to consider when writing C++.

Python just works really well on almost all platforms.

The biggest downside of Python is its parallelism, which means there is a lot of hackyness around writing parallel code. In most cases we can break things down and run different tests independently of one another, but in many other cases we have to make use of awkward workarounds, use multiprocessing, and other tricks.

Re: Flashlight: Fast and flexible machine learning in C++

#64

Post by [deleted] » Sat, Apr 17, 2021, 1:52 AM UTC

stephc_int13 wrote: post #53

Given the very high computing requirements of Machine Learning, I've always be perplexed by the seemingly widespread and unquestionable preference of Python over native code (typically C/C++). I guess performance was considered less critical than clarity/flexibility. But it seems that people are discovering that complex code tend to be hard to read/modify no matter the language...

[deleted]

Re: Flashlight: Fast and flexible machine learning in C++

#65

Post by marcinzm » Sat, Apr 17, 2021, 2:36 AM UTC

stephc_int13 wrote: post #53

Given the very high computing requirements of Machine Learning, I've always be perplexed by the seemingly widespread and unquestionable preference of Python over native code (typically C/C++). I guess performance was considered less critical than clarity/flexibility. But it seems that people are discovering that complex code tend to be hard to read/modify no matter the language...

Python is basically a configuration file for the native code that actually does the machine learning. Python is easier to write than native code or most other complex configuration formats (see terraform issues).

Re: Flashlight: Fast and flexible machine learning in C++

#66

Post by v8dev123 » Sat, Apr 17, 2021, 2:43 AM UTC

antpls wrote: post #52

dynamite-ready wrote: post #32

Does anyone else think that C++ makes more sense for ML work than Python? I'd been thinking so for years. Both for deployment/performance and data wrangling purposes.

In my opinion, no, it makes no sense to write ML in C++ : - Python allows for higher level description of algorithms, which means researchers can focus more on the ML stuff and less on low level details. - There is no performance gain in going from Python to C++, because in both cases the models are compiled to specific binary formats to be executed on dedicated hardwares. TensorFlow enables accelerators not only for…

You will see Modern C++ more like Python these days. It has really neat stuff. A comment is not enough to describe this.

Re: Flashlight: Fast and flexible machine learning in C++

#67

Post by maeil » Sat, Apr 17, 2021, 3:06 AM UTC

I sometimes use Talon, a voice control app mostly used by developers. Iirc the developer incorporated Flashlight but encountered sigificant bugs and slow response times to issues and ended up switching to a different framework. At the very least it didn't feel much tested for real-world usage yet.

Re: Flashlight: Fast and flexible machine learning in C++

#68

Post by beltsazar » Sat, Apr 17, 2021, 4:37 AM UTC

antpls wrote: post #52

dynamite-ready wrote: post #32

Does anyone else think that C++ makes more sense for ML work than Python? I'd been thinking so for years. Both for deployment/performance and data wrangling purposes.

In my opinion, no, it makes no sense to write ML in C++ : - Python allows for higher level description of algorithms, which means researchers can focus more on the ML stuff and less on low level details. - There is no performance gain in going from Python to C++, because in both cases the models are compiled to specific binary formats to be executed on dedicated hardwares. TensorFlow enables accelerators not only for…

> There is no performance gain in going from Python to C++

The backend of TF/PyTorch is written in C++ anyway, so the more complex the model, the less time it needs to spend in the glue code (frontend) that is written in Python. Therefore, rewriting complex models in full C++, for example by using TF/PyTorch C++ API, probably won't much improve the performance.

In this paper the author rewrites some ML models in Rust using tch-rs (Rust binding for PyTorch C++ API) and finds the performance not that much better (even some models perform worse):

https://www.aclweb.org/anthology/2020.nlposs-1.4/

Re: Flashlight: Fast and flexible machine learning in C++

#69

Post by sp-matcha » Sat, Apr 17, 2021, 4:56 AM UTC

whimsicalism wrote: post #23

dwrodri wrote: post #21

This seems really cool, but I don't get why they would pour work into this while simultaneously building a C++ front-end for PyTorch[1]. Per the blog post, both frameworks have the goal of empowering ML researchers to iterate on ML models in such a way that it becomes easier to reason about performance than it would be talking to a bunch of dynamically linked object files behind an interpreter. Facebook is a huge com…

Flashlight is much lower level and gives more fine-grained performance control. For instance, I don't think there is really any way to do real-time speech recognition that is fast with PyTorch because of how it is architected. From my understanding, Tensor Comprehensions and Halide are both very tentative research projects. > not having it's direction beholden to the whims of Alphabet (see: Swift for Tensorflow). I d…

Adding to the above - Tensor Comprehensions was path-finding research and is no longer maintained. The git repo is frozen (archived) as a research artifact.

Halide is still quite active, and was used in products at Adobe and Google circa 2016-2017. Not sure about the current state of industry usage though.

Re: Flashlight: Fast and flexible machine learning in C++

#70

Post by mgraczyk » Sat, Apr 17, 2021, 6:00 AM UTC

sp-matcha wrote: post #69

whimsicalism wrote: post #23

Earlier quoted context omitted.

Flashlight is much lower level and gives more fine-grained performance control. For instance, I don't think there is really any way to do real-time speech recognition that is fast with PyTorch because of how it is architected. From my understanding, Tensor Comprehensions and Halide are both very tentative research projects. > not having it's direction beholden to the whims of Alphabet (see: Swift for Tensorflow). I d…

Adding to the above - Tensor Comprehensions was path-finding research and is no longer maintained. The git repo is frozen (archived) as a research artifact. Halide is still quite active, and was used in products at Adobe and Google circa 2016-2017. Not sure about the current state of industry usage though.

Halide is still used by Google in a few different places.