Exploring Weight Agnostic Neural Networks
ai.googleblog.com
Exploring Weight Agnostic Neural Networks
1–10 of 67 posts
Re: Exploring Weight Agnostic Neural Networks
#2I wrote a series of Markov chat simulators as a teenager. Often I used a simpler algorithm which ignored the probability weight (all out-links, once learned, given equal probability). These version performed subjectively as well as, if not better, than the versions which tracked the weight of links. I'm not surprised therefore that weight agnostic neural networks can work, too.
Re: Exploring Weight Agnostic Neural Networks
#3I'm pretty sure this was posted a while back (maybe a month or two)?
Re: Exploring Weight Agnostic Neural Networks
#4Previous discussion (about the actual research article at https://weightagnostic.github.io/ rather than the blog post):
Re: Exploring Weight Agnostic Neural Networks
#5how is this different from boring old evolutionary algorithms?
In my opinion the big breakthrough that enabled optimization and machine learning was the discovery of reverse mode automatic differentiation, since the space or family of all possible decision-functions is high dimensional, while the goal (survival, reproduction) is low dimensional. Unless I see a mathematical proof that evolutionary algorithms are as efficient as RM AD, I see little future in it, and apparently neither did biology since it decided to create brains.
It's not an ideological stance I take here (of nature vs nurture).
For simplicity, lets pretend humans are single-cellular organisms, what does natural selection exert pressure on? our DNA code: both the actual protein codes and the promotor regions. I claim that variation on the proteins are risky (a modification in a proteinn coding region could render a protein useless) while a variation on the promotor regions is much less risky: altering a nucleotide there would slightly affect the affinity modulating transcription, so the cell would behave essentially the same but with different treshold concentrations, think of continuous parameters that describe our body (assuming same nurture, food, etc) some people are a bit taller, some people a bit stronger, etc... so how many of these continuous parameters do we have? On the order of the same number as the total number of promotor regions in DNA in the fertilized egg: both on human DNA and in one mitochondria (assuming there isn't a chemical signals addressing and reading and writing scheme for say 10 mitochondria)...
EDIT: just adding that for a certain fixed environment, there are local (and a global) optimum of affinity values for each protein, so that near a local optimum the fitness is roughly shaped like -s(a-a_opt)^2 where s is spread and a_opt the local optimum affinity value. In other words, it is not so that "better affinity", means fitter, not at all, a collection of genomes from an identical environment will hover around an affinity sweet spot.
According to wikipedia [0] that would result in about
about 2x 20412 "floats" for just protein-coding genes
about 2x 34000 "floats" when also including the pseudo-genes
about 2x 62000 "floats" when also including long ncRNA, small ncRNA, miRNA, rRNA, snRNA, snoRNA
these "floats" are the variables that allow a species to modulate the reaction constants in the gene regulatory network, since natural selection can not directly modulate the laws of physics and chemistry, and modulating the protein directly instead of the promotor region affinities / reaction rates risks disfunctional proteins...
so my estimate of an upper limit of the number of "floats" in the genetic algorithm is ~120000 (and probably much less if not each of the above has a promotor region).
thats not a lot of information, if we think about the number of synaptic weights in the brain, and many of these are shared in utilization by the other cell types besides neurons.
I consider the possibility that: sperm cell, egg cell, or fertilized egg cell performs a kind of POST (power-on-self-test) that checks for some of the genes, although simply reaching the fertilized state may be enough of a selftest so no spontaneous abortion test may be needed (to save time and avoid resources spent on a probably malformed child).
[0] https://en.wikipedia.org/wiki/Human_genome#Molecular_organiz...
EDIT2: regarding:
>This makes WANNs particularly well positioned to exploit the Baldwin effect, the evolutionary pressure that rewards individuals predisposed to learn useful behaviors, without being trapped in the computationally expensive trap of ‘learning to learn’.
The computationally expensive trap of having to 'learn to learn' could end up being as mundane as a low number of hormones to which neurons in the brain globally or collectively respond, which enables learning by reward or punishment, and from then on anticipating reward or punishment, and our individual end goal stems from this anticipation, and anticipating the anticipation etc...
Re: Exploring Weight Agnostic Neural Networks
#6Unpopular quote from my image and video processing professor - “The only problem with machine learning is that the machine does the learning and you don’t.”
While I understand that is missing a lot of nuance, it has stuck with me over the past few years as I feel like I am missing out on the cool machine learning work going on out there.
There is a ton of learning about calculus, probability, and statistics when doing machine learning, but I can’t shake the fact that at the end of the day, the output is basically a black box. As you start toying with AI you realize that the only way to learn from your architecture and results is by tuning parameters and trial and error.
Of course there are many applications that only AI can solve, which is all good and well, but I’m curious to hear from some heavy machine learning practitioners - what is exciting to you about your work?
This is a serious inquiry because I want to know if it’s worth exploring again. In the past university AI classes I took, I just got bored writing tiny programs that leveraged AI libraries to classify images, do some simple predictions etc.
Re: Exploring Weight Agnostic Neural Networks
#7Unpopular quote from my image and video processing professor - “The only problem with machine learning is that the machine does the learning and you don’t.” While I understand that is missing a lot of nuance, it has stuck with me over the past few years as I feel like I am missing out on the cool machine learning work going on out there. There is a ton of learning about calculus, probability, and statistics when doin…
I’m not a heavy practitioner but as a robotics engineer being able to use existing algorithms to perform previously difficult tasks is exciting. I’m also hopeful that the coming decades will bring a lot more to robotics software as machine learning research continues.
One thing I’ve been learning is that the black box nature of machine learning algorithms is partially a myth. A lot of tools have been written to help explain models. However I’m a mere novice and student so that’s just something I’ve heard. Would love it if a skilled practitioner chimed in.
Re: Exploring Weight Agnostic Neural Networks
#8This seems like it has the potential for massive efficiency gains and maybe could help with better generalization if the much simpler networks could more easily be reused or recursed or something.
Re: Exploring Weight Agnostic Neural Networks
#9how is this different from boring old evolutionary algorithms? In my opinion the big breakthrough that enabled optimization and machine learning was the discovery of reverse mode automatic differentiation, since the space or family of all possible decision-functions is high dimensional, while the goal (survival, reproduction) is low dimensional. Unless I see a mathematical proof that evolutionary algorithms are as ef…
In the related work section of their paper they mention that WANNs are related to genetic programming [1], a subfield of evolutionary algorithms.
Genetic programming is quite a powerful tool. IIRC, a few years ago, a researcher evolved expressions to model the dynamics of a double pendulum based only on measured data. To his surprise he found that the expressions were the Lagrangian of the system.
Re: Exploring Weight Agnostic Neural Networks
#10how is this different from boring old evolutionary algorithms? In my opinion the big breakthrough that enabled optimization and machine learning was the discovery of reverse mode automatic differentiation, since the space or family of all possible decision-functions is high dimensional, while the goal (survival, reproduction) is low dimensional. Unless I see a mathematical proof that evolutionary algorithms are as ef…
>Unless I see a mathematical proof that evolutionary algorithms are as efficient as RM AD, I see little future in it, and apparently neither did biology since it decided to create brains.
In the biological context the complexity is shifted away from the actual selection algorithm and onto the "scoring function." Although the filter of reproduction is relatively simple [1], the reason why the organism was fit and could ultimately reproduce is very complex. The brain's "topology" is the result of selection pressure favoring adaptations that improve fitness by dynamically adapting to relevant patterns found in the environment. The brain attempts to accurately model important aspects of the complex environment it's challenged with to increase fitness.
[1] Nothing in biology is ever actually simple, even though individuals undergo fitness based scoring, the actual notion of individual is arbitrary. In the case of bees most of the individual bees in the colony are sterile, which is obviously bad for fitness of the "individual." However a few individuals reproduce, so all the sterile worker's fitness is shifted onto the queen/drones through the concept of inclusive fitness. This indirect selection also applies to humans, who undergo inclusive fitness through their siblings. This can even be generalized to your individual cells/organs which function as a colony supporting the gonads which actually undergo direct selective pressure.