51–59 of 59 posts
20 B200 hours for CIFAR-10 seems like a lot...
Earlier quoted context omitted.
If you want to understand the issue with analog computers, design a SHA-256 circuit for one of them and consider the consequences of trying to push a megabyte of data through it. While that is an extreme example I chose precisely to make the issues clear, much real computing has many of the same characteristics, just distributed a bit more widely in time and space. Or, to put it another way, you can make anything sou…
Is SHA-256 something that makes sense in analog realm, or is it something that only needs to exist due to digital constraints?
But if you want an even better exercise, sure, work out how to send, say, precise stock market transactions where a "$5.04" becoming a "$5.05" is a very big deal to some people who have lots of money, and work out a mechanism for verifying the integrity of a lot of such data efficiently.
Bear in mind "efficiently" in this case includes the idea that "$5.04" and "$5.05" are actually close together, not separated by quite a lot of signal bandwidth. It should be of a similar size to the current digital world where that is a single bit; if you're throwing more bandwidth at your representation you've already lost to digital. Or to put it in analog terms, that needs to be pretty close to the noise barrier already; it's not a solution to make it so that you end up with "$5.04 +/- 0.000001" and "$5.05 +/- 0.000001" as the two signals you send. That is, after all, what digital is in the first place: All signals are analog in the end, and we send 0s and 1s with enough separation that the receiver can then re-amplify them into a 0 or a 1 without loss. It's not really analog if you're not hard up against the noise floor, it's just digital wearing an analog wig.
If analog is supposedly "better" than digital... at least, for the sake of argument, I recognize you did not make that claim... that would include being able to do some of these things that we do in the digital world quite comfortably. If it's just a niche... well, that's exactly where we are now in the world anyhow.
Quite a lot of real-world data is quite digital in nature. This message I'm posting is intrinsically digital. Even if I were to write something by hand, we all know, and knew even before computers, that the essence of that message is captured by a stream of letters. When we read the Gettysburg Address it doesn't even occur to us to worry about the theoretically vast amount of information we lose by not having the handwritten original. While those can be of historical interest we all know the payload is in the digital stream of letters and words. You have probably never worried before about whether the nuances of Hacker News posts are lost because they're typed in a fixed-width font but displayed in a proportional one, because the digital text carries the vast majority of the content. Even in an "analog world" there is no escape from quite a lot of digital-characteristic data. And there is no escape from the many, many issues with truly analog solutions, such as the inability to copy data without loss. This text has undergone literally dozens of copies by the time it gets from my keyboard to you eyes, and that would drive the analog world insane. Either accept much more degradation than any of us are used to, or dedicated much much larger amounts of bandwidth to everything in a way we would find horribly inefficient in our real world.
Earlier quoted context omitted.
Is SHA-256 something that makes sense in analog realm, or is it something that only needs to exist due to digital constraints?
The use case of "I have a big pile of data and I need to ensure that it survived transmission intact" isn't going anywhere. Just being in an analog computing world would not make the way some data needs to be precise go away. Not everything is a TV signal being consumed by human eyeballs. But if you want an even better exercise, sure, work out how to send, say, precise stock market transactions where a "$5.04" becomi…
There’s also a lot of path dependency. SHA256 makes a lot of sense once you already have lots of digital data but you’d probably do something else entirely in a more analog world.
The required compute seems a bit high: "We trained all CIFAR-10 models on 1xB200 GPU, and all ImageNet 64×64 models on 8xB200 GPUs. The largest CIFAR-10 model uses 20 B200 hours to train, and the largest ImageNet 64×64 model uses 640 B200 hours" 20 B200 hours for CIFAR-10 seems like a lot...
Earlier quoted context omitted.
The use case of "I have a big pile of data and I need to ensure that it survived transmission intact" isn't going anywhere. Just being in an analog computing world would not make the way some data needs to be precise go away. Not everything is a TV signal being consumed by human eyeballs. But if you want an even better exercise, sure, work out how to send, say, precise stock market transactions where a "$5.04" becomi…
Some analog devices are "DSP" in a wig, but all digital signal processing is really just a convenience wrapper around analog current flow. A very handy one, admittedly, but one that might not always be worth it. There’s also a lot of path dependency. SHA256 makes a lot of sense once you already have lots of digital data but you’d probably do something else entirely in a more analog world.
Suppose you have some analog dataset, say 1000 readings between 0 and 100 V, and your sensor allows to resolve the values up to 0.1 V precision. Now you need to encrypt that data before, say, sending them over radio. How would you do it?
Earlier quoted context omitted.
Some analog devices are "DSP" in a wig, but all digital signal processing is really just a convenience wrapper around analog current flow. A very handy one, admittedly, but one that might not always be worth it. There’s also a lot of path dependency. SHA256 makes a lot of sense once you already have lots of digital data but you’d probably do something else entirely in a more analog world.
What would you do if you need to encrypt data using analog computer? Suppose you have some analog dataset, say 1000 readings between 0 and 100 V, and your sensor allows to resolve the values up to 0.1 V precision. Now you need to encrypt that data before, say, sending them over radio. How would you do it?
Earlier quoted context omitted.
It seems like total parameter count is more or less on par with conventional approaches so any gains won't be from there. We can implement coupled oscillators in hardware but are the couplings and frequencies programmable? If they're being streamed in I guess you'd still have a memory bandwidth bottleneck and associated energy usage. If not then the fair comparison is to a conventional model hardcoded in an ASIC whic…
Do the parameters in these harmonic systems compress better? Instead of needing to hold individual parameters for each oscillator, could groupings of oscillators be instead be described with its output over a given time and then just reverse that output to get the original parameters (I’m thinking the output is like an FFT of the oscillators which is a single value, then do an inverse FFT to get the original oscillat…
Earlier quoted context omitted.
Do the parameters in these harmonic systems compress better? Instead of needing to hold individual parameters for each oscillator, could groupings of oscillators be instead be described with its output over a given time and then just reverse that output to get the original parameters (I’m thinking the output is like an FFT of the oscillators which is a single value, then do an inverse FFT to get the original oscillat…
You mean describing the core of the system via its normal mode or similar or even performing normal mode analysis to recover it from a waveform? I have no idea how that performs as a lossy compression method versus other options such as quantization but regardless I'm fairly certain it would be computationally infeasible given the application.
… I still hold hope metaheuristics could at least do a good job of approximating. One day I’ll have time to play and go down the rabbit hole