Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Abusing hash kernels for wildly unprincipled machine learning

jeremydhoon.github.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Abusing hash kernels for wildly unprincipled machine learning

#21

Post by rm999 »

bcoates wrote: post #19

I don't know anything about this space, but I have some real-world datasets and love throwing algorithms I don't understand at them. Right now I'm writing classifiers by hand, it'd be awesome to outsource my job to some AI. Does anyone have a good starting point where I can figure out which modern classification algorithms are relevant or even possible given my dataset and computer power available?

This is a pretty decent flowchart to finding appropriate machine learning algorithms for your problem: http://i.imgur.com/iVo4AAw.png

Re: Abusing hash kernels for wildly unprincipled machine learning

#22

Post by cscheid »

apw wrote: post #13
fiatmoney wrote: post #11

Essentially random projection, which has good theoretical justification and comes in handy quite often, for instance, in SVMs [1]. I'd be concerned about using something as naive as CRC32 though, ideally they'd be using a cryptographic-strength hash fn. There's also an entire area of research around "semantic hashing" and local embedding, that starts with such a random projection, and tries to improve the mapping to…

The cited Weinberger article says: Different from random projections, the hashing-trick preserves sparsity and introduces no additional overhead to store projection matrices.

The storage of random projection matrices is puzzling, because there's a well-known trick in always building the projection matrices from a known seed. It gives you the right distribution properties, and none of the storage woes. With a sufficiently fast RNG, this could actually be faster than storing it in main memory because of cache pollution.

Re: Abusing hash kernels for wildly unprincipled machine learning

#23

Post by tomarr »

jhoon wrote: post #10
rm999 wrote: post #6

This is pretty similar to the approach that many predictive modelers already use: compress data down into 'dummy variables', where each variable represents some attribute. For example, you could convert the variable state into 51 dummy variables, one for california, one for DC, etc. Hashing makes the programming a little easier and helps avoid throwing out data in the long-tail of the distribution when the number of…

Your point regarding lost meaning is quite salient. It can be useful and enlightening when a learner reports a measure feature importance (such as in random forest models, http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.... ). When you say "hashing makes the programming a little easier," I think you hit the nail on the head. I'm not trying to improve classification accuracy -- my goal was just to make it…

The hashing is not for security though, so why not keep a store of your hash + key. Again, added overhead but you wouldn't have to hash twice and you could just use the mapping table for debugging, rather than in operational code at the expense of resources.

Re: Abusing hash kernels for wildly unprincipled machine learning

#24

Post by tripzilch »

fiatmoney wrote: post #11

Essentially random projection, which has good theoretical justification and comes in handy quite often, for instance, in SVMs [1]. I'd be concerned about using something as naive as CRC32 though, ideally they'd be using a cryptographic-strength hash fn. There's also an entire area of research around "semantic hashing" and local embedding, that starts with such a random projection, and tries to improve the mapping to…

while cryptographic hashes would fit the bill, there's plenty non-cryptographic hashes with equally good key distribution properties and avalanche behaviour, that are an order of magnitude faster to compute (some, like the FNV hash are even super-easy to implement yourself). These are the hashes used in hash-tables etc, not cryptographic hashes.

Re: Abusing hash kernels for wildly unprincipled machine learning

#25

Post by mhluongo »

rm999 wrote: post #12
jhoon wrote: post #10

Earlier quoted context omitted.

Your point regarding lost meaning is quite salient. It can be useful and enlightening when a learner reports a measure feature importance (such as in random forest models, http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.... ). When you say "hashing makes the programming a little easier," I think you hit the nail on the head. I'm not trying to improve classification accuracy -- my goal was just to make it…

>my goal was just to make it as easy as possible to learn on arbitrary structured data I'd be very careful about throwing arbitrary data at your learner, at least if you don't understand your data well. Oftentimes the predictors and response are not properly separated in the same way they will be during real-world usage (for example, in time); this leads to target leaks, where your model is effectively cheating by us…

Hm, couldn't a hybrid approach deal with this? Eg hash all the data except a few dimensions you think are vital, and add those to the resulting hashed array?

Re: Abusing hash kernels for wildly unprincipled machine learning

#27

Post by mkmkmmmmm »

mhluongo wrote: post #25
rm999 wrote: post #12

Earlier quoted context omitted.

>my goal was just to make it as easy as possible to learn on arbitrary structured data I'd be very careful about throwing arbitrary data at your learner, at least if you don't understand your data well. Oftentimes the predictors and response are not properly separated in the same way they will be during real-world usage (for example, in time); this leads to target leaks, where your model is effectively cheating by us…

Hm, couldn't a hybrid approach deal with this? Eg hash all the data except a few dimensions you think are vital, and add those to the resulting hashed array?

Or location-aware hashing?

Re: Abusing hash kernels for wildly unprincipled machine learning

#28

Post by brendano »

jhoon wrote: post #5
brendano wrote: post #4

I don't understand why this is "wildly unprincipled." Doesn't it have all the guarantees that Weinberger and Shi et al papers describe?

This is a great question. The "unprincipled" part comes in the application of hashkernel. For example, any integer-valued fields encountered in structured data are treated as categorical features (whereas floating-point numbers are treated as continuous features). Of course, it's possible that some of these integer-valued fields should be treated as continuous features. When using techniques in ways they were never i…

eh, it's just binning :)

Re: Abusing hash kernels for wildly unprincipled machine learning

#29

Post by datahipster »

Meh. Diving a little bit more into the results on how well the hash kernels algorithm did with UCI Adult Names data set is mildly disappointing. Take a look at the results (http://archive.ics.uci.edu/ml/machine-learning-databases/adu...) and you'll see that hash kernels rank 14 of 17.

To be fair, I would definitely like to see how this algorithm does with other data sets as well.

All I see this algorithm doing is basically a projection of a highly dimension feature set onto a random n-d projection via a hashing function. Another words, it's not clear to me how an optimal classification boundary can be constructed using this random projection. I feel comfortable with understanding the performance characteristics of techniques such as SVM or vector quantization since they both focus on implementing algorithms that optimally reduce the dimensionality of the feature space. However, random projects are, in my humble opinion, overrated.

Also, there are two additional sources of parameterization that might make it difficult to use this tool: the selection of a hash function (and accounting for the information loss via a random projection) along with the arbitrary hash kernel array size. That makes it somewhat difficult to train and validate a model.

However! I really do like the ideas presented here and look forward into diving into this and other variants. I really appreciate the OP for taking the time to piece together this cool tool and compare it's performance with other algorithms.

Post reply on HN