81–90 of 115 posts
Earlier quoted context omitted.
I think you mean resilient or robust, not antifragile. I know of no building which gets stronger under/after hurricane force winds.
No, but the community and relevant economy can.
Plus the individual may have to make progress without a community or economy to fall back on.
Whether you have a building or not.
Earlier quoted context omitted.
Mechanical engineering uses the term system resilience. I must admit I don't like antifragility, which sounds like a novlang and is poorly defined.
Resilience is not the same, a system is anti-fragile if it gets stronger under stress. Resilience is just "not getting weaker".
Earlier quoted context omitted.
Correct me if im wrong, this is just a question originating from my own curiosity, but could you not apply the same or similar techniques to classical NN architectures? I dont see exactly how the differentiability of the architecture matters? I.e. in between epochs randomly perturb weight values (similar effect I understand to introducing scheduled noise or non-isomorphic transforms on the features), implement a sche…
> I dont see exactly how the differentiability of the architecture matters? I don't know that classical architectures are infeasible to perturb as noted in other comments. I just think we have a really interesting knob to turn in this specific case that doesn't seem to exist in other architectures. How would you go about constraining the resource consumption (by up to half!) of a classical ANN while preserving realis…
See for example: https://arxiv.org/abs/2404.02258
They have a fixed compute budget which is lower than what the LLM need, and dynamically decide to allocate this compute budget to different part of the network.
So its not exactly what you propose since here the compute budget is fixed (that's the point of the paper: to make the network learn how to allocate the resources by itself) but its dynamic for each part of the network, so it shows that its possible.
Earlier quoted context omitted.
>Chasing biologically-inspired rabbits, such as STDP and LIF (see paper above/wikipedia), does seem to be a waste of time Unless physiological compatibility with a biological brain is considered an interesting endpoint? If you think about Neuralink for example, wouldn’t it be interesting if our brains could directly engage the model? Not just a translation layer but perceive it directly and natively though some kind…
> Unless physiological compatibility with a biological brain is considered an interesting endpoint? Excellent point. I am focused entirely on synthetic simulations without biological integration (for now). I think that could be an interesting next step - determining a way to modify a SNN that was trained in a synthetic time domain such that it can process real time signals. Training these things online is not really…
Thanks for sharing your work, would love to see a post on here down the road when you've found some light in the dark forest.
Earlier quoted context omitted.
I think you mean resilient or robust, not antifragile. I know of no building which gets stronger under/after hurricane force winds.
No, but the community and relevant economy can.
Antifragile is a bit odd, the idea is supposed to be things that get stronger because of the shocks. Think how your muscles grow specifically from micro tears. In that regard, they must have resistance to grow. Though, I think that also misses on some of the definition?
Really, the more I try to defend the idea, the more I question it. :(
Earlier quoted context omitted.
Resilience is not the same, a system is anti-fragile if it gets stronger under stress. Resilience is just "not getting weaker".
Pedantically-not pedantic, but isn’t resilience the ability to quickly recover?
Resistance is the ability of not requiring any adjustment to avoid a failure when the situation changes.
That antifragility term, that yeah, is the same as hormesis is the ability of increasing your fitness when the situation changes (in unpredictable ways).
A machine that has very thick parts is resistant. A machine with redundant parts that you can fix is resilient. A machine whose builder is always learning from failure and improving is antifragil.
Earlier quoted context omitted.
No, but the community and relevant economy can.
But you can do that with a simple growth model? Larger systems can already withstand larger shocks that would have been fatal when they were smaller. Antifragile is a bit odd, the idea is supposed to be things that get stronger because of the shocks. Think how your muscles grow specifically from micro tears. In that regard, they must have resistance to grow. Though, I think that also misses on some of the definition?…
I think the point is that the systems become stronger with negative stimuli. The community and relevant economy become stronger only because of the hurricane. They would not have done it otherwise.
Systems or organisms can be defined as antifragile if they derive benefit from systemic variability, volatility, randomness, or disorder. If that's a riddle, death fits. Those things are characteristic of the pestilence, famine, war, etc. that feed death. The decay surrounding death is complex and dynamical. And death is more sustainable than any living system.
Earlier quoted context omitted.
But you can do that with a simple growth model? Larger systems can already withstand larger shocks that would have been fatal when they were smaller. Antifragile is a bit odd, the idea is supposed to be things that get stronger because of the shocks. Think how your muscles grow specifically from micro tears. In that regard, they must have resistance to grow. Though, I think that also misses on some of the definition?…
I'm not sure why you're talking about the size of systems, maybe I missed something. I think the point is that the systems become stronger with negative stimuli. The community and relevant economy become stronger only because of the hurricane. They would not have done it otherwise.
At any rate, this is a hard analogy to stick with as it isn't the hurricane that makes things stronger. It is being ready for the hurricane. And, yeah, you can argue that getting hit by hurricanes is why the city was prepared, but that is kind of silly. Are non-coastal towns that don't have to evacuate for large storms somehow weaker than the ones that do? Not really, they are prepared for other shocks.
Again, muscle growth would kind of follow an antifragile model. In that you have to have some resistance for most growth. Maybe the old SR-71 was an even better example. Literally needed the resistance of flight so that it wouldn't leak. The saying for it was the faster it went, the faster it could go.
Earlier quoted context omitted.
Some of this reads like rediscovering control theory, which is all about stability and robustness within defined limits. It's more popularization than innovation. The person who first translated this idea into math was James Clerk Maxwell, in his paper, "On Governors", in 1868.[1] Maxwell was the first to get a mathematical handle on stability of feedback systems. He wrote: "If, by altering the adjustments of the mac…
I thought so too, and I thought "this is just robust control with different words", but after reading the abstract I don't think robust control actually describes deriving an improvement from perturbing an input. On the other hand, "persistence of excitation" (a phrase I've heard of but never applied) suggests this too is well trodden ground (Wikipedia suggests end of 70s & 80s)
Not really. But using a step input to get info to tune a control system is well known. Eurotherm's auto-tuning system for temperature control was one of the first commercial products to self-tune PID loops that way. It started by setting the control value to 0 and listening to the sensors for a while to get data on sensor noise. Then it set the control value to some large value and watched the sensed value ramp up, being careful not to overshoot the goal. This provided enough data to tune the controller. Simple, but practical.
The paper is much more general, but doesn't cover how to do much. One of the references, the one on traffic control, is more interesting.[1] It's about traffic signal control in the presence of disruption. They only did some simulated experiments on one road setup, though. It's hard to tell if that actually works, but it's a reasonable idea with some data behind it.