Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Antifragility in complex dynamical systems

nature.com

Post reply on HN
Print view View original

71–80 of 115 posts

Re: Antifragility in complex dynamical systems

#71

Post by jcims »

bob1029 wrote: post #20
JackeJR wrote: post #12

Earlier quoted context omitted.

I am quite curious about your approach, Do you have an example code somewhere?

I'd start here: https://arxiv.org/abs/2406.12552 I will say that less is more and "The Bitter Lesson" applies here. Chasing biologically-inspired rabbits, such as STDP and LIF (see paper above/wikipedia), does seem to be a waste of time, especially when we have this thing entirely outside of biology that can arbitrarily replicate, serialize, mutate and simulate billions of instances of the same parent candidate in mi…

>Chasing biologically-inspired rabbits, such as STDP and LIF (see paper above/wikipedia), does seem to be a waste of time

Unless physiological compatibility with a biological brain is considered an interesting endpoint? If you think about Neuralink for example, wouldn’t it be interesting if our brains could directly engage the model? Not just a translation layer but perceive it directly and natively though some kind of synaptic modem that converts analog exchanges of neurotransmitters to the synthetic network in the digital domain.

Re: Antifragility in complex dynamical systems

#72

Post by sva_ »

bob1029 wrote: post #17
_yb2s wrote: post #13

Earlier quoted context omitted.

Very cool, but I'm surprised you're sharing this on here and not in a job interview with a deep learning startup and/or an arXiv paper.

> a job interview with a deep learning startup I am not aware of any DL startups that are interested in techniques which are incompatible with GPUs and back propagation. I think this is a solo journey through the dark forest for now.

> I think this is a solo journey through the dark forest for now.

In academia there's quite a bit of stuff on neuromorphic computing going on.

Re: Antifragility in complex dynamical systems

#73

Post by neom »

seanhunter wrote: post #68
bob1029 wrote: post #2

I've found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each generation) results in far more robust simulations that will meet fitness criteria with less likelihood of getting stuck somewhere. The tradeoff being that simulating will take longer, but this may be worth things like reducing the chances of your resource requi…

> I've found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each generation) results in far more robust simulations that will meet fitness criteria with less likelihood of getting stuck somewhere. If I understand you correctly this is similar to the philosophy behind simulated annealing isn't it? The idea being that adding…

Not op but what they described is indeed sim annealing coupled with relaxing feedback loops. This approach would in theory result in prevention of stagnation, forced adaptation and over time, creating some form of resiliency. Pretty cool stuff!

Re: Antifragility in complex dynamical systems

#74

Post by BriggyDwiggs42 »

hobs wrote: post #67
BriggyDwiggs42 wrote: post #38

Earlier quoted context omitted.

Isn’t it definitely not an organism? If death is the inevitable process from order to disorder, isn’t it deeply a-systemic at its core? If it were a system, it would also eventually have to end.

Arguably it does, the heat death of the universe option would spell the end of entropy, and the end of death (unless you believe in a conformal cyclic universe)

Actually yeah fair enough; I’ll grant that death may be a system.

Re: Antifragility in complex dynamical systems

#75

Post by bob1029 »

seanhunter wrote: post #68
bob1029 wrote: post #2

I've found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each generation) results in far more robust simulations that will meet fitness criteria with less likelihood of getting stuck somewhere. The tradeoff being that simulating will take longer, but this may be worth things like reducing the chances of your resource requi…

> I've found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each generation) results in far more robust simulations that will meet fitness criteria with less likelihood of getting stuck somewhere. If I understand you correctly this is similar to the philosophy behind simulated annealing isn't it? The idea being that adding…

I was actually thinking about simulated annealing as I got into this.

I've found that reducing this specific mode of perturbation over time has not been necessary. The simulation will oscillate its way right into a perfect fitness score assuming everything else is good.

Convergence seems to rely much more heavily on the quality of the fitness function and population management (i.e. size, selection criteria, mutation strategies). I apply random mutations to delays and weights, but using a spectrum of mutation each time (I.e. some clones are heavily mutated while others are barely touched). This spectra of mutation is constant all the way through the simulation.

Re: Antifragility in complex dynamical systems

#76

Post by bob1029 »

taylorius wrote: post #62
bob1029 wrote: post #2

I've found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each generation) results in far more robust simulations that will meet fitness criteria with less likelihood of getting stuck somewhere. The tradeoff being that simulating will take longer, but this may be worth things like reducing the chances of your resource requi…

Firstly, I'm coming at this from the pint of view of standard feed forward NNs, rather than spiking neural networks, so apologies in advance if I am way off-base. By global activation limit, I assume you mean the total sum of neuron activation during one forward pass? So halving it would strongly encourage sparsity? How do you enforce the reduced limit? Just put it in the loss function? Or does the spiking nature all…

> By global activation limit, I assume you mean the total sum of neuron activation during one forward pass?

Yes. For each candidate network I keep a counter of the total # of neuron activations. This figure is constrained by a global limit (which varies based on prior performance). We simply break out of the simulation loop once the allotted resources are exhauted. This means that more efficient networks are more likely to score higher.

Re: Antifragility in complex dynamical systems

#77

Post by bob1029 »

jcims wrote: post #71
bob1029 wrote: post #20

Earlier quoted context omitted.

I'd start here: https://arxiv.org/abs/2406.12552 I will say that less is more and "The Bitter Lesson" applies here. Chasing biologically-inspired rabbits, such as STDP and LIF (see paper above/wikipedia), does seem to be a waste of time, especially when we have this thing entirely outside of biology that can arbitrarily replicate, serialize, mutate and simulate billions of instances of the same parent candidate in mi…

>Chasing biologically-inspired rabbits, such as STDP and LIF (see paper above/wikipedia), does seem to be a waste of time Unless physiological compatibility with a biological brain is considered an interesting endpoint? If you think about Neuralink for example, wouldn’t it be interesting if our brains could directly engage the model? Not just a translation layer but perceive it directly and natively though some kind…

> Unless physiological compatibility with a biological brain is considered an interesting endpoint?

Excellent point. I am focused entirely on synthetic simulations without biological integration (for now).

I think that could be an interesting next step - determining a way to modify a SNN that was trained in a synthetic time domain such that it can process real time signals. Training these things online is not really an option. You have to run a large # of offline simulations before you can find the right networks. Learning rules like STDP could theoretically address the online learning problem, but I couldn't find any traction down that path yet.

Re: Antifragility in complex dynamical systems

#78

Post by lo0dot0 »

bob1029 wrote: post #2

I've found that intentionally causing abrupt (but reasonable) changes to hyperparameters in evolutionary spiking neural network simulations (I.e between each generation) results in far more robust simulations that will meet fitness criteria with less likelihood of getting stuck somewhere. The tradeoff being that simulating will take longer, but this may be worth things like reducing the chances of your resource requi…

Some of you may die... But that's a sacrifice i'm willing to make.

Re: Antifragility in complex dynamical systems

#79

Post by guerrilla »

soulofmischief wrote: post #59
fuzzfactor wrote: post #57

Earlier quoted context omitted.

When enduring successive hurricanes the most worthwhile effort might be to improve antifragility on more of a continuous basis than even recovery itself. You can definitely benefit by increasing your antifragility in the face of a second hurricane long before recovery from the first hurricane has been very well completed at all, no matter how focused the recovery effort. I never thought people needed a new term to be…

I think you mean resilient or robust, not antifragile. I know of no building which gets stronger under/after hurricane force winds.

No, but the community and relevant economy can.

Re: Antifragility in complex dynamical systems

#80

Post by seventytwo »

morning-coffee wrote: post #70
delichon wrote: post #4

Systems or organisms can be defined as antifragile if they derive benefit from systemic variability, volatility, randomness, or disorder. If that's a riddle, death fits. Those things are characteristic of the pestilence, famine, war, etc. that feed death. The decay surrounding death is complex and dynamical. And death is more sustainable than any living system.

What's the benefit derived from death? I'd suggest death is simply the terminal state that most systems find themselves in after a sufficiently long period of time has elapsed or they've experienced a set of inputs outside the range they were designed to handle and keep operating.

Entropy always wins
Post reply on HN