What it takes to build great machine learning products • Hacker News Board

Re: What it takes to build great machine learning products

#41

Post by o1iver » Mon, Apr 16, 2012, 8:58 PM UTC

nkh wrote: post #38

Dn_Ab wrote: post #32

Earlier quoted context omitted.

To work with ML stuff you don't need much. You can just download packages and get experience with how to best pick models, choose features and tweak (hyper)parameters. If you want to work on or understand it then you will need math. You need a decent understanding of calculus (mid 1800s level, mulitvariate calculus), a more decent understanding of Linear Algebra (1950s ), information theory (1960s), and probability a…

* If you are doing stuff related to genetic programming then category and type theory have riches to be plundered.* Could you expand this a bit as I don't understand the meaning. Are you saying that if the problem you are working on can be solved with genetic algorithms, then you could blow it away with category and type theory? I don't have a vested interest in either, I am just curious. Thanks.

He said genetic programming not genetic algorithms. For the former, type and category theory are useful!

Re: What it takes to build great machine learning products

#42

Post by ogrisel » Mon, Apr 16, 2012, 10:18 PM UTC

sparsevector wrote: post #17

ogrisel wrote: post #13

Earlier quoted context omitted.

No actually it's not the case. But I don't know how Pegasos can be adapted to use kernels. If you take figure 1 of the paper [1] you will see that the gradient of the objective function is used to update a single weights vector `w` at each step of the projected stochastic gradient descent. In a kernel model, all the support vectors cannot be collapsed into a single weight vector `w`. You would need to handle the kern…

The set of support vectors is just the set of training examples that have non-zero alpha parameters. To implement the gradient update you just evaluate the support vector machine on the example (using the explicit kernel expansion) and then if the example has signed margin less than 1 you add y * eta to the corresponding alpha value. The difficulty with Pegasos for non linear kernels is the support set quickly become…

Thanks for the reply. I was told by twitter that this is similar to the kernel perceptron which I don't know well either. There is a good introduction with python code snippet here:

http://www.mblondel.org/journal/2010/10/31/kernel-perceptron...

However it seems that you need to compute the kernel expansion on the full set of samples (or maybe just the accumulated past samples?): this does not sound very online to me...

Re: What it takes to build great machine learning products

#43

Post by junktest » Mon, Apr 16, 2012, 10:41 PM UTC

Probably try PCA (principal components analysis) to help select the most important features of data, first, before going further in modeling it.

Re: What it takes to build great machine learning products

#44

Post by sparsevector » Mon, Apr 16, 2012, 11:50 PM UTC

ogrisel wrote: post #42

sparsevector wrote: post #17

Earlier quoted context omitted.

The set of support vectors is just the set of training examples that have non-zero alpha parameters. To implement the gradient update you just evaluate the support vector machine on the example (using the explicit kernel expansion) and then if the example has signed margin less than 1 you add y * eta to the corresponding alpha value. The difficulty with Pegasos for non linear kernels is the support set quickly become…

Thanks for the reply. I was told by twitter that this is similar to the kernel perceptron which I don't know well either. There is a good introduction with python code snippet here: http://www.mblondel.org/journal/2010/10/31/kernel-perceptron... However it seems that you need to compute the kernel expansion on the full set of samples (or maybe just the accumulated past samples?): this does not sound very online to me…

It's true you need to compute the kernel dot product between every example you see and every example in the support set (every example that ever previously evaluated to signed margin < 1). Whether it's online depends on your definition of "online" . It's definitely not online in the sense of using memory independent of the number of examples, since you have to keep around the support set. I think there are results showing the support set grows linearly with the size of the training set under reasonable assumptions. However, it is online in the sense that it operates on a stream of data, computing predictions and updates for each example one-by-one. It's also online in the sense that its analysis is based on online learning theory (e.g. mistake / regret bounds). A lot of learning theory papers use "online" in the latter two senses, which is confusing if you expect the former.

Re: What it takes to build great machine learning products

#45

Post by joe_the_user » Tue, Apr 17, 2012, 1:24 AM UTC

gfodor wrote: post #12

A great and insightful article. A common theme I've seen in practice is folks who have a deep understanding of ML often run straight to applying the most sophisticated algorithms possible on raw data. On the other hand, people who know a bit about ML but understand the domain better start by applying intuition to data cleansing and then follow up with simpler algorithms. Without fail the latter group ends up with bet…

That has been my experience with AI programming.

But I would take a more pessimistic interpretation of this.

That is: all our "learning algorithms" has failed to learn and those with some clever heuristics succeed versus the broken methods we so-far have.

Re: What it takes to build great machine learning products

#46

Post by cema » Tue, Apr 17, 2012, 1:46 AM UTC

joe_the_user wrote: post #45

gfodor wrote: post #12

A great and insightful article. A common theme I've seen in practice is folks who have a deep understanding of ML often run straight to applying the most sophisticated algorithms possible on raw data. On the other hand, people who know a bit about ML but understand the domain better start by applying intuition to data cleansing and then follow up with simpler algorithms. Without fail the latter group ends up with bet…

That has been my experience with AI programming. But I would take a more pessimistic interpretation of this. That is: all our "learning algorithms" has failed to learn and those with some clever heuristics succeed versus the broken methods we so-far have.

I would offer a somewhat more optimistic take: humans are still better learners than machines are, and our algorithms still do not capture well the way our thinking (and intuition) works. Really, we have only started with machine learning a few decades ago; catching up with millions of years of evolution can be expected to take a bit longer.

Re: What it takes to build great machine learning products

#47

Post by joe_the_user » Tue, Apr 17, 2012, 2:58 AM UTC

cema wrote: post #46

joe_the_user wrote: post #45

Earlier quoted context omitted.

That has been my experience with AI programming. But I would take a more pessimistic interpretation of this. That is: all our "learning algorithms" has failed to learn and those with some clever heuristics succeed versus the broken methods we so-far have.

I would offer a somewhat more optimistic take: humans are still better learners than machines are, and our algorithms still do not capture well the way our thinking (and intuition) works. Really, we have only started with machine learning a few decades ago; catching up with millions of years of evolution can be expected to take a bit longer.

But I don't really think of that as positive.

Maybe its the pain of my previous AI job talking but when the choice is just an opaque hunch of an expert, it doesn't feel like a victory for human intelligence. A victory for human intelligence looks much like the discovery of a physical law where you both deal with a phenomenon and communicate how someone else can also deal with it.

What is quintessentially human in the modern sense is human beings understanding ourselves rather than

Re: What it takes to build great machine learning products

#48

Post by sireat » Tue, Apr 17, 2012, 1:16 PM UTC

tel wrote: post #29

gfodor wrote: post #12

A great and insightful article. A common theme I've seen in practice is folks who have a deep understanding of ML often run straight to applying the most sophisticated algorithms possible on raw data. On the other hand, people who know a bit about ML but understand the domain better start by applying intuition to data cleansing and then follow up with simpler algorithms. Without fail the latter group ends up with bet…

I think there are essentially two "deep" understandings of ML prevalent today. The first is more common: the ability to do the calculus, algebra, and probability derivations required to design complex ML algorithms combined with the CS knowledge to find/design a good algorithm and the software design skill to actually implement it on real, "big" data. No doubt this is a difficult position to master and those who perf…

Our Data Mining (different field but somewhat related) professor had this quote "All models are wrong, but some models are useful" on the first day of university.

You can build an extremely complicated model that is not useful, where a simpler one might suffice.

Re: What it takes to build great machine learning products

#49

Post by tel » Tue, Apr 17, 2012, 4:18 PM UTC

sireat wrote: post #48

tel wrote: post #29

Earlier quoted context omitted.

I think there are essentially two "deep" understandings of ML prevalent today. The first is more common: the ability to do the calculus, algebra, and probability derivations required to design complex ML algorithms combined with the CS knowledge to find/design a good algorithm and the software design skill to actually implement it on real, "big" data. No doubt this is a difficult position to master and those who perf…

Our Data Mining (different field but somewhat related) professor had this quote "All models are wrong, but some models are useful" on the first day of university. You can build an extremely complicated model that is not useful, where a simpler one might suffice.

I've heard that line referred to as Box's Razor. It's definitely the right heuristic, but it's interesting to see that even if your model is right, you're still in trouble if it's too complex. This is a sort of bias/variance tradeoff.