Abusing hash kernels for wildly unprincipled machine learning
jeremydhoon.github.com
Abusing hash kernels for wildly unprincipled machine learning
1–10 of 29 posts
Re: Abusing hash kernels for wildly unprincipled machine learning
#2This is very similar to the hashing trick used in vowpal wabbit (http://hunch.net/~vw/). I haven't used this python module, but vw is absolutely incredible to use.
Re: Abusing hash kernels for wildly unprincipled machine learning
#3This is very similar to the hashing trick used in vowpal wabbit ( http://hunch.net/~vw/ ). I haven't used this python module, but vw is absolutely incredible to use.
Great point! VW was actually an inspiration for this blog post. I don't believe it handles structured data quite the same way as hashkernel, but it does a great job of accepting both categorical and continuous features. We use VW models to classify all sorts of badness on my team at Facebook (http://www.newscientist.com/article/dn21095-inside-facebooks...).
Re: Abusing hash kernels for wildly unprincipled machine learning
#4I don't understand why this is "wildly unprincipled." Doesn't it have all the guarantees that Weinberger and Shi et al papers describe?
Re: Abusing hash kernels for wildly unprincipled machine learning
#5I don't understand why this is "wildly unprincipled." Doesn't it have all the guarantees that Weinberger and Shi et al papers describe?
This is a great question. The "unprincipled" part comes in the application of hashkernel. For example, any integer-valued fields encountered in structured data are treated as categorical features (whereas floating-point numbers are treated as continuous features). Of course, it's possible that some of these integer-valued fields should be treated as continuous features. When using techniques in ways they were never intended to be used, like treating continuous features as categorical, many of the assumptions that make something "principled" no longer apply.
Re: Abusing hash kernels for wildly unprincipled machine learning
#6This is pretty similar to the approach that many predictive modelers already use: compress data down into 'dummy variables', where each variable represents some attribute. For example, you could convert the variable state into 51 dummy variables, one for california, one for DC, etc. Hashing makes the programming a little easier and helps avoid throwing out data in the long-tail of the distribution when the number of possible values is very high. It's common in NLP exactly because of this.
Hashing comes with costs though. It's much harder to interpret a model with hashed variables because the variables lose meaning. Also, some information may get thrown out; as OP mentions integers may have continuous meaning and categorizing them can really damage the model. If you have a very large dataset it probably won't matter because most modeling methods asymptotically converge (i.e. as you have more and more data the model can learn that 333 and 334 are similar just by seeing enough examples that this is the case), but if you don't you could be throwing out valuable information. In a case like this I suppose the modeler could manually go in and convert his integers to floats to 'tell' the algorithm the data is continuous.
Re: Abusing hash kernels for wildly unprincipled machine learning
#7For someone (me) not particularly knowledgeable on the subject. I'm curious to understand the contrast between this and vowpal wabbit.
Could someone offer an insight into when it would be useful to use this project as opposed to vowpal?
Re: Abusing hash kernels for wildly unprincipled machine learning
#8For someone (me) not particularly knowledgeable on the subject. I'm curious to understand the contrast between this and vowpal wabbit. Could someone offer an insight into when it would be useful to use this project as opposed to vowpal?
First, please don't use hashkernel in production. It's a toy designed to showcase how hash kernels can help us (easily) learn on structured data.
VW is faster, smarter, and more featureful. However, it doesn't know how to learn on structured data.
Re: Abusing hash kernels for wildly unprincipled machine learning
#9Nice writeup, jhoon!
In case anyone else was searching his/her eyes trying to figure out where the "likes" field is actually sanitized out from the the training data set, it's the item.pop("likes") inside the function "is_liked": https://github.com/jeremydhoon/hashkernel/blob/master/fblear...
... Maybe I like long names too much, but my vote would be something longer like "check_if_liked_and_remove_likes" since "is_liked" sounds like a predicate that won't be modifying its input.
Re: Abusing hash kernels for wildly unprincipled machine learning
#10This is pretty similar to the approach that many predictive modelers already use: compress data down into 'dummy variables', where each variable represents some attribute. For example, you could convert the variable state into 51 dummy variables, one for california, one for DC, etc. Hashing makes the programming a little easier and helps avoid throwing out data in the long-tail of the distribution when the number of…
Your point regarding lost meaning is quite salient. It can be useful and enlightening when a learner reports a measure feature importance (such as in random forest models, http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home....).
When you say "hashing makes the programming a little easier," I think you hit the nail on the head. I'm not trying to improve classification accuracy -- my goal was just to make it as easy as possible to learn on arbitrary structured data.