How to teach a Bayesian spam filter to play chess • Hacker News Board

Re: How to teach a Bayesian spam filter to play chess

#11

Post by zyroth » Fri, Nov 02, 2007, 11:49 PM UTC

pixcavator wrote: post #4

zyroth wrote: post #3

Earlier quoted context omitted.

Bayesian classification works a bit like this: You have a set of inputs and a set of targets. By having seen a history of elements of the powerset of inputs and its manually tagged classes, the bayesian classificator learns how to classify new elements of the powerset of inputs. Thus, a bayesian classifier has to know the classes and the inputs before. It cannot extrapolate from the sets it was trained on, since the…

Are you saying that a neural network can learn addition symbolically? Can you recommend a book or site to read about this?

No, I'm saying that a neural network can learn addition.

See, a bayesian cannot really understand that there are relations between two numbers (like "is bigger than" or "is the following number of"), but a neural network can, since addition is part of a NN.

My personal recommendation on machine learning is 'Pattern Recognition and Machine Learning' by Chris Bishop. But you definately do need a solid mathematical background for that.

Re: How to teach a Bayesian spam filter to play chess

#12

Post by aswanson » Sat, Nov 03, 2007, 12:01 AM UTC

pixcavator wrote: post #8

dpapathanasiou wrote: post #5

Earlier quoted context omitted.

Can you recommend a book or site to read about this? David MacKay wrote a good intro book on Bayesian, neural networks, and related topics: http://www.inference.phy.cam.ac.uk/mackay/itila/

I've looked at the chapter on neurall networks. It does not seem to address operations with symbols, only numbers...

hhm is right. You would end up with a network with connection weights of 1. As long as your sigmoidal transfer functions are biased such that they have no multiplying effect your output would be an addition of the inputs.

Re: How to teach a Bayesian spam filter to play chess

#13

Post by pixcavator » Sat, Nov 03, 2007, 12:01 AM UTC

hhm wrote: post #10

pixcavator wrote: post #8

Earlier quoted context omitted.

I've looked at the chapter on neurall networks. It does not seem to address operations with symbols, only numbers...

You can always encode symbols as numbers, and otherwise. That's what computers do, that's what you do when you do a sum by yourself, and that's what such a neural network should do too. (You should encode the symbols to a binary input, then decode the neural network output to symbols).

So given a function with f('1','1')='2', the computer will figure out that f('1','2')='3', right?

Re: How to teach a Bayesian spam filter to play chess

#14

Post by pixcavator » Sat, Nov 03, 2007, 12:05 AM UTC

aswanson wrote: post #12

pixcavator wrote: post #8

Earlier quoted context omitted.

I've looked at the chapter on neurall networks. It does not seem to address operations with symbols, only numbers...

hhm is right. You would end up with a network with connection weights of 1. As long as your sigmoidal transfer functions are biased such that they have no multiplying effect your output would be an addition of the inputs.

What if the machine does not know that it is dealing with numbers? It's all symbolic: '1'+'1'='2', etc.

Re: How to teach a Bayesian spam filter to play chess

#15

Post by pixcavator » Sat, Nov 03, 2007, 12:07 AM UTC

zyroth wrote: post #11

pixcavator wrote: post #4

Earlier quoted context omitted.

Are you saying that a neural network can learn addition symbolically? Can you recommend a book or site to read about this?

No, I'm saying that a neural network can learn addition. See, a bayesian cannot really understand that there are relations between two numbers (like "is bigger than" or "is the following number of"), but a neural network can, since addition is part of a NN. My personal recommendation on machine learning is 'Pattern Recognition and Machine Learning' by Chris Bishop. But you definately do need a solid mathematical back…

Where would the idea of "is bigger than" or "is the following number of" come from if not from the person who creates the network?

Re: How to teach a Bayesian spam filter to play chess

#16

Post by aswanson » Sat, Nov 03, 2007, 12:18 AM UTC

pixcavator wrote: post #14

aswanson wrote: post #12

Earlier quoted context omitted.

hhm is right. You would end up with a network with connection weights of 1. As long as your sigmoidal transfer functions are biased such that they have no multiplying effect your output would be an addition of the inputs.

What if the machine does not know that it is dealing with numbers? It's all symbolic: '1'+'1'='2', etc.

The neural network doesn't "know" that it is dealing with anything, just as you don't "know" the function your body uses to expand and contract your heart. You could be feeding it stock quotes, rgb pixel values, your daily weight, anything. If the information can reduced to numeric values (it can) the network will determine the relationship in the form of a function.

Re: How to teach a Bayesian spam filter to play chess

#17

Post by machine » Sat, Nov 03, 2007, 12:19 AM UTC

pixcavator wrote: post #14

aswanson wrote: post #12

Earlier quoted context omitted.

hhm is right. You would end up with a network with connection weights of 1. As long as your sigmoidal transfer functions are biased such that they have no multiplying effect your output would be an addition of the inputs.

What if the machine does not know that it is dealing with numbers? It's all symbolic: '1'+'1'='2', etc.

What you are talking about is more along the lines of inductive logic programming, where the goal is to induce general symbolic rules from specific examples. Even with ILP I'm not sure how exactly you could phrase things so that the machine could learn "addition" though. An ILP system could probably learn something like a + b = c implies b + a = c (i.e. commutativity) and other properties of addition from examples.

Re: How to teach a Bayesian spam filter to play chess

#18

Post by pixcavator » Sat, Nov 03, 2007, 12:29 AM UTC

machine wrote: post #17

pixcavator wrote: post #14

Earlier quoted context omitted.

What if the machine does not know that it is dealing with numbers? It's all symbolic: '1'+'1'='2', etc.

What you are talking about is more along the lines of inductive logic programming, where the goal is to induce general symbolic rules from specific examples. Even with ILP I'm not sure how exactly you could phrase things so that the machine could learn "addition" though. An ILP system could probably learn something like a + b = c implies b + a = c (i.e. commutativity) and other properties of addition from examples.

If it can learn these properties, it may be even better! Then it can do all computations. So, can it learn commutativity, and other properties? Would it be smart enough to look for them in the first place?

Re: How to teach a Bayesian spam filter to play chess

#19

Post by pixcavator » Sat, Nov 03, 2007, 12:35 AM UTC

aswanson wrote: post #16

pixcavator wrote: post #14

Earlier quoted context omitted.

What if the machine does not know that it is dealing with numbers? It's all symbolic: '1'+'1'='2', etc.

The neural network doesn't "know" that it is dealing with anything, just as you don't "know" the function your body uses to expand and contract your heart. You could be feeding it stock quotes, rgb pixel values, your daily weight, anything. If the information can reduced to numeric values (it can) the network will determine the relationship in the form of a function.

If the input is numbers then what happens is simply extrapolation (or regression etc). If the input is symbols, then what? The fact that you can associate numbers to the symbols does not help. The extrapolation will be meaningless. Unless you happen to have the correspondence '1'->1, '2'->2, etc,

Re: How to teach a Bayesian spam filter to play chess

#20

Post by machine » Sat, Nov 03, 2007, 12:47 AM UTC

pixcavator wrote: post #18

machine wrote: post #17

Earlier quoted context omitted.

What you are talking about is more along the lines of inductive logic programming, where the goal is to induce general symbolic rules from specific examples. Even with ILP I'm not sure how exactly you could phrase things so that the machine could learn "addition" though. An ILP system could probably learn something like a + b = c implies b + a = c (i.e. commutativity) and other properties of addition from examples.

If it can learn these properties, it may be even better! Then it can do all computations. So, can it learn commutativity, and other properties? Would it be smart enough to look for them in the first place?

I don't know a lot about ILP algorithms, but my understanding of them is that they basically search the space of logical rules for rules that "explain" the examples they are given. More formally, they look for rules from which you could then prove the given examples. So in an ideal world you could give an ILP system a bunch of formulas and it would give you back the Peano axioms or something like that I'm not sure if state of the art ILP systems are good enough to do that in practice or not.