Neural networks are impressive only in that they are able to give any kind of meaningful results at all. In the end, they are only a poor mimicry of real machine intelligence, and not much better, conceptually, than plain old nonlinear regression. Nobody has been able to determine what the structure of a neural network should look like for any given problem (network type, number of nodes, layers, activation functions…
Please, try implementing non-linear regression to understand images. Tell me how it goes. Also, 'nonbody know hows learning is stored'? You very clearly have never worked with neural nets before. Experience is stored in the form of weight values.
Where's the incorrect data stored? How can you fix it? It's in the weight values, somewhere, but you can't go and change the weight values to fix the horse/fish cascade without breaking everything else it knows.
Yes, we know 'where' the data is stored. But it's diffuse, not discrete, so we can't separate it from other data.