Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What it takes to build great machine learning products

radar.oreilly.com

Post reply on HN
Print view View original

11–20 of 49 posts

Re: What it takes to build great machine learning products

#11

Post by ogrisel »

marshallp wrote: post #10

The article doesn't mention two important things (and instead focuses on being clever - the opposite of what machine learning stands for). First, the deep learning algorithms that automatically create features. Second, the importance of gathering lots of data, or generating it. If you have to be really clever with feature engineering, then what's the point of even calling yourself a machine learning person.

I agree that deep-learning is an interesting approach to learn higher level features. However it's still a long way from being a universal solution: for instance deep learning won't help you solve the machine translation or multi-documents text summarization problems automagically: you still need to find good (hence often task dependent) representation for both your input data and the datastructure you are trying to learn a predictive model for.

Re: What it takes to build great machine learning products

#12

Post by gfodor »

A great and insightful article. A common theme I've seen in practice is folks who have a deep understanding of ML often run straight to applying the most sophisticated algorithms possible on raw data. On the other hand, people who know a bit about ML but understand the domain better start by applying intuition to data cleansing and then follow up with simpler algorithms. Without fail the latter group ends up with better results.

Re: What it takes to build great machine learning products

#13

Post by ogrisel »

srconstantin wrote: post #8
ogrisel wrote: post #5

Very nice article Aria. You quickly mention Pegasos as a scalable alternative to SMO. I agree that this works well for linear models. But despite the claim that Pegasos can be trivially adapted to kernel models I have never seen any implementation of a kernel Pegasos and I don't understand how it's even possible. Have you used Pegasos-style algorithm to fit non linear models? On the other hand there exist alternative…

So...are you saying you need the dual formulation in order to allow a kernel model?

No actually it's not the case. But I don't know how Pegasos can be adapted to use kernels. If you take figure 1 of the paper [1] you will see that the gradient of the objective function is used to update a single weights vector `w` at each step of the projected stochastic gradient descent. In a kernel model, all the support vectors cannot be collapsed into a single weight vector `w`. You would need to handle the kernel expansion against the support vectors explicitly. But then how to select the support vectors out of all the samples from the dataset while keeping the algorithm online? The Pegasos paper does not say mention it.

[1] http://eprints.pascal-network.org/archive/00004062/01/Shalev...

Re: What it takes to build great machine learning products

#14

Post by ma2rten »

Right now NLP is mostly limited to niche applications, like e.g. sentiment analysis and clever products build around it. I actually think the reason is that both natural language processing and machine learning are still in their early days.

Imagine all the applications for consumer products if algorithms would be really able to actually understand language (as far as you can understand something if you are a computer program and not a sentient human being), e.g. for example if we were able to do real text summarization.

I believe this is not only possible, but not as far away as people think. However, to reach that goal we need to let go of that idea that NLP is mostly about clever feature engineering, but instead start building algorithm that derive those features themselves. Part of the problem is how evaluation is setup in NLP. What the best algorithm is, is decided based on who gets the best performance on some dataset. This sounds all nice and objective, but you will always be able to get the best performance if you try enough combinations of features (overfitting the testset) [1]. These small improvements say little about real world performance.

For the NLP people among you this is an interesting paper that tries to do a lot of things different: http://ronan.collobert.com/pub/matos/2008_nlp_icml.pdf

This is the corresponding tutorial, which is quite entertaining as well: http://videolectures.net/nips09_collobert_weston_dlnl/

[1] I think, this is less true for machine translation, where there are more and bigger testsets and less feature engineering going on.

Re: What it takes to build great machine learning products

#15

Post by ma2rten »

gfodor wrote: post #12

A great and insightful article. A common theme I've seen in practice is folks who have a deep understanding of ML often run straight to applying the most sophisticated algorithms possible on raw data. On the other hand, people who know a bit about ML but understand the domain better start by applying intuition to data cleansing and then follow up with simpler algorithms. Without fail the latter group ends up with bet…

Really? I'd think that most ML people are well aware of the importance of data cleansing and feature extraction. Also my experience is that domain knowledge often (but not always - depends on the domain) helps surprisingly little. Feature extraction is mostly an iterative approach anyway: you define some very simple features, you look at the mistakes, you add some features and repeat until you are happy. Ideally you also do some visualization in there somewhere.

Re: What it takes to build great machine learning products

#16

Post by chaostheory »

The article is light on details. imo there are two major things your team needs:

1) Programmers that have the needed math skills, or mathematicians with the needed coding skills

2) A distributed ML framework

Solving problem one is not easy but it's straightforward.

Solving problem two is harder. While there are a lot of open source machine learning projects, almost all of them seem to have a focus of being used by a person and not a program. Moreover very few do distributed processing except for mahout (http://mahout.apache.org/). Mahout is promising but the documentation is still thin, and I'm not sure if it's getting momentum in terms of mind share yet.

Re: What it takes to build great machine learning products

#17

Post by sparsevector »

ogrisel wrote: post #13
srconstantin wrote: post #8

Earlier quoted context omitted.

So...are you saying you need the dual formulation in order to allow a kernel model?

No actually it's not the case. But I don't know how Pegasos can be adapted to use kernels. If you take figure 1 of the paper [1] you will see that the gradient of the objective function is used to update a single weights vector `w` at each step of the projected stochastic gradient descent. In a kernel model, all the support vectors cannot be collapsed into a single weight vector `w`. You would need to handle the kern…

The set of support vectors is just the set of training examples that have non-zero alpha parameters. To implement the gradient update you just evaluate the support vector machine on the example (using the explicit kernel expansion) and then if the example has signed margin less than 1 you add y * eta to the corresponding alpha value.

The difficulty with Pegasos for non linear kernels is the support set quickly becomes very large and so evaluating the model becomes very slow. Note that since the alpha values are not constrained to be non-negative (unlike the standard dual algorithms) the alpha values don't ever get clipped to zero--instead they just slowly converge to zero. It's still (I think) one of the fastest methods in terms of theoretical convergence guarantees but perhaps not as fast as LaSVM or something similar in practice.

However, there's been a more general trend in machine learning to use linear models with lots of features instead of kernel models, partially because of these sort scalability issues.

Re: What it takes to build great machine learning products

#18

Post by brendano »

ma2rten wrote: post #14

Right now NLP is mostly limited to niche applications, like e.g. sentiment analysis and clever products build around it. I actually think the reason is that both natural language processing and machine learning are still in their early days. Imagine all the applications for consumer products if algorithms would be really able to actually understand language (as far as you can understand something if you are a compute…

Careful with the Collobert ICML-2008 paper. It has a very negative reputation among NLP researchers who actually know the area, just for its setup/evaluation. If you're interested in the methods (which I think are interesting), that group's later work is much improved.

Re: What it takes to build great machine learning products

#19

Post by brendano »

ogrisel wrote: post #5

Very nice article Aria. You quickly mention Pegasos as a scalable alternative to SMO. I agree that this works well for linear models. But despite the claim that Pegasos can be trivially adapted to kernel models I have never seen any implementation of a kernel Pegasos and I don't understand how it's even possible. Have you used Pegasos-style algorithm to fit non linear models? On the other hand there exist alternative…

For NLP applications, which I think Aria's article is mostly concerned with, non-linear kernelized classifiers are often little better than linear ones. I think that's one part of the recent interest in SGD-style training algorithms (they work for linear cases nicely, less so for kernelized ones).

[deleted part about kernelizing pegasos, realized i dont know that area]

Re: What it takes to build great machine learning products

#20

Post by jmount »

gfodor wrote: post #12

A great and insightful article. A common theme I've seen in practice is folks who have a deep understanding of ML often run straight to applying the most sophisticated algorithms possible on raw data. On the other hand, people who know a bit about ML but understand the domain better start by applying intuition to data cleansing and then follow up with simpler algorithms. Without fail the latter group ends up with bet…

My definition of a "deep understanding of ML" definitely excludes people who immediately try "the most sophisticated algorithm." Buzzword jockies try all the cool stuff first, but most practitioners I have met try basic statistical methods first. Then when they see what issue they need to overcome they bring in a method designed to help with that issue.
Post reply on HN