91–100 of 160 posts
Earlier quoted context omitted.
So that's how savants do it ...
There is one savant that can do crazy math but is otherwise normal. Afgter an epileptic fit. His system is base 10000, he can add any two numbers below 5000 and come up with a single digit response in one loop. Then covert to base10 for the rest of us. Each digit in his base 10000 system has a different visual representation, like we have 0-9. It's integer accurate so I don't think it uses sine wave approximations. G…
Earlier quoted context omitted.
So that's how savants do it ...
There is one savant that can do crazy math but is otherwise normal. Afgter an epileptic fit. His system is base 10000, he can add any two numbers below 5000 and come up with a single digit response in one loop. Then covert to base10 for the rest of us. Each digit in his base 10000 system has a different visual representation, like we have 0-9. It's integer accurate so I don't think it uses sine wave approximations. G…
Earlier quoted context omitted.
One thing that the essay doesn't consider is the importance of efficiency in computation. Efficiency is important because in practice it is often the factor which most limits the scalability of a computational system. The human brain only consumes around 20W [1], but for numerical calculations it is massively outclassed by an ARM chip consuming a tenth of that. Conversely, digital models of neural networks need a hug…
Thanks for those numbers! They are frankly scary. Six millions years of power for a single brain is one year of power for six million brains. The knowledge that ChatGPT contains is far higher than the knowledge a random sample of six million people have produced in a year. With that said, I think we should apply ML and the results of the bitter lesson to things which are more intractable than searching the web or pla…
On the other hand, I don't think if it would be larger than that of six million people selected more carefully (though I suppose you'd have to include the costs of selection process in the tally). I also imagine it wouldn't be larger than six thousand people specifically raised and taught in coordinated fashion to fulfill this role.
On the other other hand, a human brain does a lot more than learning language, ideas and conversion between one and the other. It also, simultaneously, learns a lot of video and audio processing, not to mention smell, proprioception, touch (including pain and temperature), and... a bunch of other stuff (I was actually surprised by the size of the "human" part of the Wikipedia infobox here: https://en.wikipedia.org/wiki/Template:Sensation_and_percept...). So it's probably hard to compare to specialized NN models until we learn to better classify and isolate how biological brains learn and encode all the various things they do.
Earlier quoted context omitted.
Watts are actually a time independent measurement, note that the TWh has "hour" affixed to the end. This is 1 Tera Watt over the course of one hour, not one second. Your numbers are off by a factor of 3600. 1TWh / 20 Watt brain = 50,000,000,000 (50 Billion) Hours. 50 Billion Hours / (24h * 365.25) = 5,703,855.8 Years
thank you both.. was googling this and found some stackexchange type answers and still got confused. apparently it is quite common to see the "per time" part of the definition of a Watt and get it mixed up with Watt-hours. i think i got it now thank yoyu
Earlier quoted context omitted.
Watts are actually a time independent measurement, note that the TWh has "hour" affixed to the end. This is 1 Tera Watt over the course of one hour, not one second. Your numbers are off by a factor of 3600. 1TWh / 20 Watt brain = 50,000,000,000 (50 Billion) Hours. 50 Billion Hours / (24h * 365.25) = 5,703,855.8 Years
thank you both.. was googling this and found some stackexchange type answers and still got confused. apparently it is quite common to see the "per time" part of the definition of a Watt and get it mixed up with Watt-hours. i think i got it now thank yoyu
Earlier quoted context omitted.
Putting aside the idea that "analog" doesn't have overflow. (Any physical implementation of that analog signal would have limits in the real world. If nothing else, then how much current or voltage your PSU can supply, or how much current can go through your wire.) But that is not even the real issue. The real issue that it is not analog, it is just "analog" with quotes. The neural network is executed by a digital co…
Are there any true analog adder circuits? If so are they also less error prone like digital.
Sure. Here is a semantics with an op-amp: https://www.tutorialspoint.com/linear_integrated_circuits_ap...
Earlier quoted context omitted.
I’ve had all kinds of cheap electronics that came with unadvertised radio features for free.
It's sad that those pesky regulators from the FCC make it so hard to get devices with unintended radio functionality these days!
Earlier quoted context omitted.
Transformers are specialized to run on our hardware, whereas the article is suggesting architectures which are specialized for a specific task.
Are transformers not already very specialized to the task of learning from sequences of word vectors? I'm sure there is more that can be done with them other than making the input sequences really long, but my point was that LLMs are hardly lacking in design specialized to their purpose.
No, you can use transformers for vision, image generation, audio generation/recognition, etc.
They are 'specialized' in that they are for working with sequences of data, but almost everything can be nicely encoded as a sequence. In order to input images, for example, you typically split the image into blocks and then use a CNN to produce a token for each block. Then you concatenate the tokens and feed them into a transformer.
Earlier quoted context omitted.
Why it works here is that the "analog" representation is not influenced by noise, because it's simulated on digital hardware. On the other hand, why we use digital hardware precisely because it's robust against the noise present in the hardware analog circuits.
So in other words, this is analog computation on a digital hardware on analog substrate - the analog-to-digital step eliminates the noise of our physical reality, and the subsequent digital-to-analog reintroduces a certain flexibility of design thinking. I wonder, is there ever a case analog-on-digital is better to work with as an abstraction layer, or is it always easier to work with digital signals directly?
Of course, these are all just approximations of analog. Much like emulation, there are limitations, penalties, and inaccuracies that will inevitably kneecap applications when compared to a native implementation running on bare metal (though, it seems that humans lack a proper math coprocessor, so calculus could be considered no more "native" than traditional algebra).
We do sometimes see specialized hardware emerge (DSPs in the case of audio signal processing, FPUs in the case of floats, NPUs/TPUs in the case of neural networks), but these are almost always highly optimized for operations specific to analog-like data patterns (e.g.: fourier transforms) rather than true analog operations. This is probably because scalable/reliable/fast analog memory remains an unsolved problem (semiconductors are simply too useful).