Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning Library for C++

image.diku.dk

Post reply on HN
Print view View original

41–50 of 52 posts

Re: Machine Learning Library for C++

#41

Post by tlarkworthy »

jhartmann wrote: post #37
tlarkworthy wrote: post #25

Earlier quoted context omitted.

honestly you guys are crazy if you think shark is gonna help you learn machine learning. Its ideal for deployment of ML on things like embedded computer, robotics, games etc. where real time learning is required. Machine learning requires alot of experimentation and C++ is a terrible medium for that. There are loads of good machine learning libraries implemented for python and matlab. Pretty much every good paper in…

Actually you forget that performance when you need to train for days at a time is critical, if I use Octave/Matlab/R my current project might take months to train instead of weeks. All my ML code is high performance threaded C++. I recommend you use a good template linear algebra library like Eigen, you can do plenty of experimentation in C++. I find with a set of a few modern libraries and the required experience a…

True that matlab octave and R are all rubbish for performance. I use python + numpy which all delegates to BLAS for the hardcore linear algebra stuff. I don't normally find C++ gains me all that much. You can also do GPU acceleration pretty easy using theano (e.g. http://deeplearning.net/software/theano/tutorial/using_gpu.h...)

So I reckon my GPU accelerated python still beats a C++ pthreads approach, and is alot faster to develop on.

Your mileage may vary, from what you said you probably know what you are doing, maybe GPU is not applicable. I was really replying to the initial comments that said they want to start learning machine learning on a C++ system. Training for days suggests you are doing something hardcore like MCMC/DBN/Guassian Processes, learners should not start there though....

Re: Machine Learning Library for C++

#42

Post by jhartmann »

Schneekatze wrote: post #40
jhartmann wrote: post #38

Earlier quoted context omitted.

You should consider using Eigen for linear algebra, I have personally found it much better performance wise than using bindings to ATLAS or other more standard linear algebra solutions. ML algorithms tend to be multistage (think about the update of weights with momentum in a Neural network for example), and the primitives available in ATLAS or a blas library are really too low level. Eigen since it generates code for…

This is true, even though we would rather switch to Armadillo due to it's easier handling and better high-level behaviour. Right now the linear algebra library we use -ublas- has the same behaviour as Eigen for BLAS1 type expressions. So it tries to generate optimal (non-SSE) code. Only for BLAS2 and 3 we fall back to the ATLAS-routines which has the same performance as Eigen on the interesting problem sizes. //small…

You forget that if you can do the whole weight update in a single shot operation that the data doesn't have to go through the cache multiple times, and at least on the problem sizes I am working on FP bandwidth isn't what kills it, but the memory bandwidth. Back to the NN example: If you can do the matrix multiply and the application of the delta weights in a single loop iteration you get much better cache behavior.

Another thing about code generation, I am also using a hacked version of Eigen as well in a project I'm working on that can do the tanh and derivative of the tanh so the NN activations go quite abit faster since you can generate vectorized code for the whole calculation that will visit the memory location exactly once. While true the calculation of the weight updates is the most time spent, I saw 3-4x speedup in the activation code doing it in a single operation due to better memory access patterns and less loop iterations. Better memory access patterns can also have synergistic effects on other code because there is less cache pollution happening. By being fast and loose and introducing a few other copies of the matrix data in my case, my performance falls off a cliff when it no longer fits in the cpu cache nicely. 10x difference in the particular case I am remembering.

As always performance is part art, part science and perhaps it won't matter as much for the general case, but for my specific implementation and my matrix sizes Eigen has made a measurable difference for me compared to other solutions.

Re: Machine Learning Library for C++

#43

Post by misiti3780 »

c0g wrote: post #22
misiti3780 wrote: post #8

Earlier quoted context omitted.

Implementing the SVM from scratch was time consuming - no?

Yes it was :) but time well spent imo. On reflection I guess I might have had more free time to spend on this than a normal person - I did the SVM as a [small] part of my masters project, so if you're time constrained with a real job and a life then might be best to disregard me.

If you had the quadratic solver, I would think it would be reasonable to add the rest of the code. If you started adding costs, gammas, etc. I would think it would take a while. I spent hours looking at the source code of libSVM at my last job and never really understood what the hell was going on

Re: Machine Learning Library for C++

#44

Post by misiti3780 »

iskander wrote: post #13
misiti3780 wrote: post #8

Earlier quoted context omitted.

Implementing the SVM from scratch was time consuming - no?

The only tricky part would be writing a quadratic solver. Alternatives: either solve a linear SVM using gradient descent (simpler to write), or offload the core of the algorithm to an existing solver like cvxopt. edit: For an example of using cvxopt, check out http://www.mblondel.org/journal/2010/09/19/support-vector-ma...

cool - thanks

Re: Machine Learning Library for C++

#45

Post by pilooch »

Schneekatze wrote: post #36
pilooch wrote: post #28

Very interesting, but as a daily practitioner I am skeptical. First, this is a lot of code! As a C++ machine learning programmer, I am impressed as I know the pain (someone explains why, see comment https://news.ycombinator.com/item?id=5613797 ). Second, it contains a version of Blas and ublas as well as LBFGS and more, much more, coded from scratch as it seems. This seems too much for an ML library, and a lot to mai…

Hi, shark developer here. First of all, we are glad that our library is discussed on this board! We are happy for every feedback we can get! Regarding Performance: we try to get the key algorithms as fast as possible. And for the hardest parts we rely not on ublas, but use optional bindings to ATLAS. Speed was one of the key design criteria. We hope that we achieved that. Clearly this is no guarantee that every algor…

Amazing work, thanks for sharing. Any hint on how it behaves on big datasets ? Over time I've found that scaling up to real world (and industry) datasets requires going custom (or distributed, a-la-mr / hadoop).

Re: Machine Learning Library for C++

#46

Post by jhartmann »

tlarkworthy wrote: post #41
jhartmann wrote: post #37

Earlier quoted context omitted.

Actually you forget that performance when you need to train for days at a time is critical, if I use Octave/Matlab/R my current project might take months to train instead of weeks. All my ML code is high performance threaded C++. I recommend you use a good template linear algebra library like Eigen, you can do plenty of experimentation in C++. I find with a set of a few modern libraries and the required experience a…

True that matlab octave and R are all rubbish for performance. I use python + numpy which all delegates to BLAS for the hardcore linear algebra stuff. I don't normally find C++ gains me all that much. You can also do GPU acceleration pretty easy using theano (e.g. http://deeplearning.net/software/theano/tutorial/using_gpu.h... ) So I reckon my GPU accelerated python still beats a C++ pthreads approach, and is alot fa…

I'm doing deep belief networks with dropout, and don't have access to GPU's with good double precision performance. I used to write graphics device drivers, so GPU computing has a special place in my heart and definitely agree with you there performance wise. It is funny though that my little laptop is hitting training times similar to some papers where people are using low end GPU's though, its amazing what you can do when you pay attention to performance.

I suspect my tuned C++ code will work quite well on a Intel MIC, and that is probably where I'm going to go when I have more resources to throw at the problem. I do know that Theano does use Alex's C++ CUDA code under the covers and I have done lots of reading of some of theano's code looking at implementation details to help developing my code. I just am not a big python (or most scripting languages actually) fan, perhaps I'm just too old school and written C, C++, C# and Java too long. If it doesn't smell or feel like C, I feel like Scotty in Star Trek 4 when he was making the transparent aluminum on the mac.

Re: Machine Learning Library for C++

#47

Post by to3m »

thwest wrote: post #27
to3m wrote: post #23

Earlier quoted context omitted.

LPGL isn't vastly better than the GPL as it makes it difficult to link statically and/or release for closed platforms. Its advocates would probably see these as plus points, but I'm not sure that they're going to increase uptake.

Difficult, but not really a significant part of the challenges for delivering closed source binaries across platforms. On linux, you have to build distribution specific binaries that match the shared library versions in the package manager. On Windows, you generally put all of your shared libraries in your application's folder, since there are plenty of bad actors who install DLLs without versions in the filename to…

I was thinking more of Xbox360, Playstation3, iOS, etc. - none support user-replacable files. I don't even think any even support dynamic linking... )

Re: Machine Learning Library for C++

#48

Post by Schneekatze »

pilooch wrote: post #45
Schneekatze wrote: post #36

Earlier quoted context omitted.

Hi, shark developer here. First of all, we are glad that our library is discussed on this board! We are happy for every feedback we can get! Regarding Performance: we try to get the key algorithms as fast as possible. And for the hardest parts we rely not on ublas, but use optional bindings to ATLAS. Speed was one of the key design criteria. We hope that we achieved that. Clearly this is no guarantee that every algor…

Amazing work, thanks for sharing. Any hint on how it behaves on big datasets ? Over time I've found that scaling up to real world (and industry) datasets requires going custom (or distributed, a-la-mr / hadoop).

Unfortunately, we don't have much experience with industry sized datasets - simply because we don't have them. I know that our SVMs are among the fastest of the world. At least we beat Libsvm and Liblinear(and there certainly on very big datasets!). But we lack support for hadoop/mpi, even though we would like to change that in the future.

Right now I would say that the main focus of shark is research oriented. That is we want to be fast but also modular so that we can still easily exchange different aspects of the algorithms with our own work. As these goals sometime clash, it is hard to claim that we are the fastest, simply because there is for nearly every algorithm some way to improve when you know exactly which combination of model, loss function and training algorithm you use. But we are (hopefully) reasonably fast and certainly want to improve.

Re: Machine Learning Library for C++

#49

Post by eps »

to3m wrote: post #23
jhartmann wrote: post #16

Earlier quoted context omitted.

Flame Suit On / Rant Mode On I actually really don't understand why anyone uses GPL for a library. I've been doing open source for a long long time, and love the GPL. I have code in the Linux kernel, and believe free software AND open source software are great solutions to very real problems in software engineering. Having open code just gives people more options, and I firmly believe it will win over time as far as…

LPGL isn't vastly better than the GPL as it makes it difficult to link statically and/or release for closed platforms. Its advocates would probably see these as plus points, but I'm not sure that they're going to increase uptake.

LGPL basically means that if I modify the libary code, I should open source the changes, but I can build derived products (in GPL sense) from an unchanged lib as much as I want. This static vs dynamic linking debate is a silly pendantry, enforcing which doesn't contribute anything to the GNU's goals or vision. It's really disappointing to see people time on this. If you want to pursue justice - go after the GPL violations. Making one to re-link a binary to force a compliance is a misplaced effort and essentially a waste of everyone's time.

Re: Machine Learning Library for C++

#50

Post by H3g3m0n »

mjn wrote: post #24
jhartmann wrote: post #16

Earlier quoted context omitted.

Flame Suit On / Rant Mode On I actually really don't understand why anyone uses GPL for a library. I've been doing open source for a long long time, and love the GPL. I have code in the Linux kernel, and believe free software AND open source software are great solutions to very real problems in software engineering. Having open code just gives people more options, and I firmly believe it will win over time as far as…

Limiting choice is BAD. The whole reason you should be creating and using free software and OSS is to not weld the hood shut. But you sound like you want to limit choice for users. The whole point of free software from the GNU perspective is to keep options open for users, and not allow downstream devs to "weld the hood shut" on derivatives by adding more restrictions on what users can do with those derivatives. Howe…

If someone is wanting to go with the full GPL I think the AGPL would be a better fit anyway otherwise people can use a webservice to do all the processing. Could even charge for it and not give any in house changes back.
Post reply on HN