31–40 of 51 posts
So it sounds like these PNNs are essentially analog implementations of neural nets? Seems like an odd choice of naming to call them 'physical'.
ANN is taken.
Why this isn't called Hardware Neural Nets is beyond me.
My knowledge in this area is incredibly limited, but I figured the paper would mention NanoWire Networks (NWNs) as an emerging physical neural network[0]. Last year, researchers from the University of Sydney and UCLA used NWNs to demonstrate online learning of handwritten digits with an accuracy of 93%. [0] = https://www.nature.com/articles/s41467-023-42470-5
Last time I read about this the main practical difficulty was model transferability. The very thing that makes it so powerful and efficient is also the thing that make it uncopiable, because sensitivity to tiny physical differences in the devices inevitably gets encoded into the model during training. It seems intuitive this is an unavoidable, fundamental problem. Maybe that scares away big tech, but I quite like the…
Last time I read about this the main practical difficulty was model transferability. The very thing that makes it so powerful and efficient is also the thing that make it uncopiable, because sensitivity to tiny physical differences in the devices inevitably gets encoded into the model during training. It seems intuitive this is an unavoidable, fundamental problem. Maybe that scares away big tech, but I quite like the…
Couldn’t you still copy by training a new network on a new device to have same outputs for the same inputs as the original?
With ANN you can do it one time and then clone the result for negligible energy cost.
Maybe training a batch of PNNs in parallel could save some of the energy cost, but I don't know how feasible that is considering they could behave slightly differently during training causing divergence... Now that sarcastic comment at the bottom of this thread is starting to sound relevant "Schools".
Earlier quoted context omitted.
I mean LoRA was taken too before LoRA became a thing
I don't mean globally, LoRA are at least in different domains. Artificial Neural Networks and Physical Neural Networks are both machine learning, discussion referring to both is highly probable, and the former far more established so it calling it an Analog Neural Network would never last long.
Earlier quoted context omitted.
Couldn’t you still copy by training a new network on a new device to have same outputs for the same inputs as the original?
Yes, but training is the most expensive part of ML, for example GPT-3 is estimated to cost something like 1-4 million USD. With ANN you can do it one time and then clone the result for negligible energy cost. Maybe training a batch of PNNs in parallel could save some of the energy cost, but I don't know how feasible that is considering they could behave slightly differently during training causing divergence... Now t…
That entirely depends on how many inferences the model will perform during its lifecycle. You can find different estimates for the energy consumption of ChatGPT, but they range from something like 500-1000 MWh a day. Assuming an electricity price of $0.165 per kWh, that would put you at roughly $80,000 to a $160,000 a day.
Even at the lower end of $80,000 a day, you'll reach your $4 Million in just 50 days.
Earlier quoted context omitted.
Yes, but training is the most expensive part of ML, for example GPT-3 is estimated to cost something like 1-4 million USD. With ANN you can do it one time and then clone the result for negligible energy cost. Maybe training a batch of PNNs in parallel could save some of the energy cost, but I don't know how feasible that is considering they could behave slightly differently during training causing divergence... Now t…
> Yes, but training is the most expensive part of ML, for example GPT-3 is estimated to cost something like 1-4 million USD. That entirely depends on how many inferences the model will perform during its lifecycle. You can find different estimates for the energy consumption of ChatGPT, but they range from something like 500-1000 MWh a day. Assuming an electricity price of $0.165 per kWh, that would put you at roughly…
With PNN you would have to multiply n by 1-4 million, training cost explodes.
Earlier quoted context omitted.
Well, the brain is a physical neural network, and evolution seems to have figured out how to generate a (somewhat) copiable model. I bet we could learn a trick or two from biology here.
The way the brain does it is by giving users a largely untrained model that they themselves have to train over the next 20 years for it to be of any use.
Earlier quoted context omitted.
Well, the brain is a physical neural network, and evolution seems to have figured out how to generate a (somewhat) copiable model. I bet we could learn a trick or two from biology here.
The way the brain does it is by giving users a largely untrained model that they themselves have to train over the next 20 years for it to be of any use.
Earlier quoted context omitted.
Well, the brain is a physical neural network, and evolution seems to have figured out how to generate a (somewhat) copiable model. I bet we could learn a trick or two from biology here.
The way the brain does it is by giving users a largely untrained model that they themselves have to train over the next 20 years for it to be of any use.
A ton of that is probably encoded elsewhere, but no doubt the brain plays a huge part. And somehow, it's all reconstructed for each new "device".