Introduction to Support Vector Machines in Machine Learning
1–10 of 90 posts
Re: Introduction to Support Vector Machines in Machine Learning
#2ITT: Whether SVMs are still relevant in the deep learning era. Some junior researchers will say neural networks are all you need. Industry folks will talk about how they still use decision trees.
Personally, I'm quite bullish on the resurgence of SVMs as SOTA. What did it for me was Mikhail Belkin's talk at IAS.[1]
[1] https://m.youtube.com/watch?index=15&list=PLdDZb3TwJPZ5dqqg_...
Re: Introduction to Support Vector Machines in Machine Learning
#3Interestingly, a top Kaggler (Ahmet) just posted a nice contest solution with SVMs for the TReNDS Neuroimaging contest: https://www.kaggle.com/aerdem4/rapids-svm-on-trends-neuroima...
Re: Introduction to Support Vector Machines in Machine Learning
#4Stay away, in my opinion. I spent a year supporting a SVM in a production machine learning application, and it made me wish the ML research community hadn't been so in love with them for so long.
They're the perfect blend of theoretically elegant and practically impractical. Training scales as O(n^3), serialized models are heavyweight, prediction is slow. They're like Gaussian Processes, except warped and without any principled way of choosing the kernel function. Applying them to structured data (mix of categorical & continuous features, missing values) is difficult. The hyperparameters are non-intuitive and tuning them is a black art.
GBMs/Random Forests are a better default choice, and far more performant. Even simpler than that, linear models & generalized linear models are my go-to most of the time. And if you genuinely need the extra predictiveness, deep learning seems like better bang for your buck right now. Fast.ai is a good resource if that's interesting to you.
Re: Introduction to Support Vector Machines in Machine Learning
#5Interestingly, a top Kaggler (Ahmet) just posted a nice contest solution with SVMs for the TReNDS Neuroimaging contest: https://www.kaggle.com/aerdem4/rapids-svm-on-trends-neuroima...
Interestingly they're also popularizing RAPIDS of Nvidia as part of it. Thanks for sharing. Never imagined SVM in such scenario!
Re: Introduction to Support Vector Machines in Machine Learning
#6Interestingly, a top Kaggler (Ahmet) just posted a nice contest solution with SVMs for the TReNDS Neuroimaging contest: https://www.kaggle.com/aerdem4/rapids-svm-on-trends-neuroima...
Nice, Thanks for sharing.
Re: Introduction to Support Vector Machines in Machine Learning
#7Stay away, in my opinion. I spent a year supporting a SVM in a production machine learning application, and it made me wish the ML research community hadn't been so in love with them for so long. They're the perfect blend of theoretically elegant and practically impractical. Training scales as O(n^3), serialized models are heavyweight, prediction is slow. They're like Gaussian Processes, except warped and without any…
Kernel function is simple - Are you in a high dimensional space? If so, choose linear kernel. Else? Choose the most non-linear one you can (usually a guassian or RBF). I suppose quadratic and the other kernals are useful if what your modeling looks like that but in practice that is rare.
Prediction is not that slow with linear SVMs especially not compared to something like K-NN. The main hyperparamaters which matter are the "C" value and maybe class weights if you have recall or precision requirements. The C value is something that should be grid-searched, but you might as well be grid-searching everything that matters on every ML algorithm and in this regard SVMs are fast to iterate over (because the C value is all that matters).
Applying categorical and continuous features is not difficult if you choose to do it in anything more sophisticated than sklearn. Also, pd.get_dummies() exists (though it may lead to that slow prediction you're concerned about)
You're most likely right with GBM or Random Forests - though they can have all sorts of issues with parallelism if you're not on the right kind of system. You talk about linear models but SVMs are usually using linear kernals anyway and are a generalization of linear models (including lasso and ridge regression models).
Re: Introduction to Support Vector Machines in Machine Learning
#8ITT: Whether SVMs are still relevant in the deep learning era. Some junior researchers will say neural networks are all you need. Industry folks will talk about how they still use decision trees. Personally, I'm quite bullish on the resurgence of SVMs as SOTA. What did it for me was Mikhail Belkin's talk at IAS.[1] [1] https://m.youtube.com/watch?index=15&list=PLdDZb3TwJPZ5dqqg_...
Thanks for sharing. You actually shared a full playlist. Won't be able to get up before finishing a few of them.
Re: Introduction to Support Vector Machines in Machine Learning
#9Stay away, in my opinion. I spent a year supporting a SVM in a production machine learning application, and it made me wish the ML research community hadn't been so in love with them for so long. They're the perfect blend of theoretically elegant and practically impractical. Training scales as O(n^3), serialized models are heavyweight, prediction is slow. They're like Gaussian Processes, except warped and without any…
Kernel function is simple - Are you in a high dimensional space? If so, choose linear kernel. Else? Choose the most non-linear one you can (usually a guassian or RBF). I suppose quadratic and the other kernals are useful if what your modeling looks like that but in practice that is rare. Prediction is not that slow with linear SVMs especially not compared to something like K-NN. The main hyperparamaters which matter…
Agreed -- linear SVMs, especially in text processing applications, is the one area where they are a natural fit. All their attributes complement the domain. Linear SVMs also have desirable performance characteristics.
But at that point, they also have a lot in common with linear models. Those also seem practical in that domain (though I have less experience here, tbh). And performant, when using SGD + feature hashing like e.g. vowpal wabbit.
My beef with non-linear kernels and structured data is a longer discussion, but I find kernel methods for structured data (which is usually high-dimension but low-rank -- lots of shared structure between features, shared structure between missingness of features) to be highly problematic.
Re: Introduction to Support Vector Machines in Machine Learning
#10ITT: Whether SVMs are still relevant in the deep learning era. Some junior researchers will say neural networks are all you need. Industry folks will talk about how they still use decision trees. Personally, I'm quite bullish on the resurgence of SVMs as SOTA. What did it for me was Mikhail Belkin's talk at IAS.[1] [1] https://m.youtube.com/watch?index=15&list=PLdDZb3TwJPZ5dqqg_...
I mean NNs are still quite bad at low n tabular data (and they may always be), which is honestly how a lot of real life data is, so there is clearly a need for not a neural network.
I feel like I've seem more tree ensembles in the wild than SVMs, though.