Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Advances in semiconductors are feeding the AI boom

spectrum.ieee.org

Post reply on HN
Print view View original

81–90 of 124 posts

Re: Advances in semiconductors are feeding the AI boom

#81

Post by gamacodre »

passion__desire wrote: post #43
gamacodre wrote: post #39

Earlier quoted context omitted.

I think the problem here would be figuring out how much of the brain's power draw to attribute to the multiplication. A brain is more akin to a motherboard than a single CPU, with all kinds of I/O, internal regulation, and other ancillary stuff going on all the time.

Is the issue then we haven't discovered the magical algorithm run by our brain? If we discover it, then digital circuits will handsomely beat brain.

We can surely build more efficient and capable hardware than our current evolved wetware, since all of the details of how to build it are generally externalized. If the chips had to fab themselves, it would be a different story.

The software is a different story. Sure, the brain does all sorts of things that aren't necessary for $TASK, but we aren't necessarily going to be able to correctly identify which are which. Is your inner experience of your arm motion needed to fully parse the meaning in "raise a glass to toast the bride and groom", or respond meaningfully to someone who says that? Or perhaps it doesn't really matter - language is already a decent tool for bridging disjoint creature realities, maybe it'll stretch to synthetic consciousness too.

Re: Advances in semiconductors are feeding the AI boom

#82

Post by dragonwriter »

doug_durham wrote: post #76
barelyauser wrote: post #64

Earlier quoted context omitted.

Non-deterministic means random. AI or natural I is not random. Analog suffers immensely from noise and it is the reason the brain has such a large number of neurons, part to deal with noise and part to deal with losing some neurons along the way.

Non-deterministic doesn't mean random. Random means random. Non-deterministic means that specific inputs don't generate the same outputs. It says nothing about the distribution of the output values. A chaotic system isn't random, its just non-deterministic.

“Nondeternimism” gets used jn a variety of ways both from differing context and conflicting uses in the same context, but chaotic systems are fully deterministic in the most common relevant sense, but highly sensitive to inputs, resulting in even very small uncertainty in inputs to render them largely unpredictable.

Re: Advances in semiconductors are feeding the AI boom

#83

Post by HarHarVeryFunny »

taktoa wrote: post #79
HarHarVeryFunny wrote: post #73

Earlier quoted context omitted.

> The power efficiency difference may be explainable by the much slower frequency of brain computation (200 Hz vs. 2GHz). Partly, but also because the brain has an asynchronous data-flow design, while the GPU is synchronous, and as you say clocked at a very high frequency. In a clocked design the clock signal needs to be routed to every element on the chip which requires a lot of power, the more so the higher the fre…

> In a clocked design the clock signal needs to be routed to every element on the chip which requires a lot of power, the more so the higher the frequency is. Clock only needs to be distributed to sequential components like flip flops or SRAMs. The number of clock distribution wire-millimeters in typical chip is dwarfed by the number of data wire-millimeters, and if a neural network is well trained and quantized acti…

There's obvious potential savings in not wasting FLOPs recalculating things unnecessarily, but I'm not sure how much of that could be realized by just building a data-flow digital GPU. The only attempt at a data-flow digital processor I'm aware of was AMULET (by ARM designer Steve Furber), which was not very successful.

There's more promise in analog chip designs, such as here:

https://spectrum.ieee.org/low-power-ai-spiking-neural-net

Or otherwise smarter architectures (software only or S/W+H/W) that design out the unnecessary calculations.

It's interesting to note how extraordinarily wasteful transformer-based LLMs are too. The transformer was designed part inspired by linguistics and part based on the parallel hardware (GPU's etc) available to run it on. Language mostly has only local sentence structure dependencies, yet transformer's self-attention mechanism has every word in a sentence paying attention to every other word (to some learned degree)! Turns out it's better to be dumb and fast than smart, although I expect future architectures will be much more efficient.

Re: Advances in semiconductors are feeding the AI boom

#84

Post by cosmojg »

TRDRVR wrote: post #47
layer8 wrote: post #40

Earlier quoted context omitted.

Ignoring for the moment that transistors and synapses are very different in their function, the current in a CPU transistor is in the milliampere range, whereas in the ion channels of a synapse it is in the picoampere range. The voltage differs by roughly a factor of ten. So the wattage differs by a factor of 10^10. One important reason for the difference in current is that transistors need to reliably switch between…

Thank you for this - the name neural networks has made a whole generation of people forget that they have an endocrine system. We know things like sleep, hunger, fear, and stress all impact how we think, yet people want to still build this mental model that synapses are just dot products that either reach an activation threshold or don't.

Fortunately for academics looking for a new start in industry, this widespread misunderstanding has made it only far too easy to transition from a slow-paced career in computational neuroscience to an overwhelmingly lucrative one in machine learning!

Re: Advances in semiconductors are feeding the AI boom

#85

Post by whiplash451 »

layer8 wrote: post #40
highfrequency wrote: post #26

Wild that the human brain can squeeze in 100 trillion synapses ( very roughly analogous to model parameters / transistors) in a 3lb piece of meat that draws 20 Watts. The power efficiency difference may be explainable by the much slower frequency of brain computation (200 Hz vs. 2GHz). My impression is that the main obstacle to achieving a comparable volumetric density is that we haven't cracked 3d stacking of integr…

Ignoring for the moment that transistors and synapses are very different in their function, the current in a CPU transistor is in the milliampere range, whereas in the ion channels of a synapse it is in the picoampere range. The voltage differs by roughly a factor of ten. So the wattage differs by a factor of 10^10. One important reason for the difference in current is that transistors need to reliably switch between…

How do you get to 10^10? I might be missing a fundamental of physics here (asking genuinely).

Re: Advances in semiconductors are feeding the AI boom

#86

Post by gary_0 »

__loam wrote: post #70
TRDRVR wrote: post #47

Earlier quoted context omitted.

Thank you for this - the name neural networks has made a whole generation of people forget that they have an endocrine system. We know things like sleep, hunger, fear, and stress all impact how we think, yet people want to still build this mental model that synapses are just dot products that either reach an activation threshold or don't.

People will spout off about how machine learning is based on the brain while having no idea how the brain works.

It is based on the brain, but only in the loosest possible terms; ML is a cargo cult of biology. It's kind of surprising that it works at all.

Re: Advances in semiconductors are feeding the AI boom

#87

Post by CuriouslyC »

layer8 wrote: post #40
highfrequency wrote: post #26

Wild that the human brain can squeeze in 100 trillion synapses ( very roughly analogous to model parameters / transistors) in a 3lb piece of meat that draws 20 Watts. The power efficiency difference may be explainable by the much slower frequency of brain computation (200 Hz vs. 2GHz). My impression is that the main obstacle to achieving a comparable volumetric density is that we haven't cracked 3d stacking of integr…

Ignoring for the moment that transistors and synapses are very different in their function, the current in a CPU transistor is in the milliampere range, whereas in the ion channels of a synapse it is in the picoampere range. The voltage differs by roughly a factor of ten. So the wattage differs by a factor of 10^10. One important reason for the difference in current is that transistors need to reliably switch between…

The more of our logic we can implement with addition, the more can be offloaded to noisy analog systems with approximate computing. It would be funny if model temperature stopped being metaphorical.

Re: Advances in semiconductors are feeding the AI boom

#88

Post by passion__desire »

gamacodre wrote: post #81
passion__desire wrote: post #43

Earlier quoted context omitted.

Is the issue then we haven't discovered the magical algorithm run by our brain? If we discover it, then digital circuits will handsomely beat brain.

We can surely build more efficient and capable hardware than our current evolved wetware, since all of the details of how to build it are generally externalized. If the chips had to fab themselves, it would be a different story. The software is a different story. Sure, the brain does all sorts of things that aren't necessary for $TASK, but we aren't necessarily going to be able to correctly identify which are which.…

All of computation is realised by very few arithmetic operations. Then test energy efficiency of wetware and hardware on those operations. Then any difference can be attributed to algorithms.

Re: Advances in semiconductors are feeding the AI boom

#89

Post by ordu »

highfrequency wrote: post #26

Wild that the human brain can squeeze in 100 trillion synapses ( very roughly analogous to model parameters / transistors) in a 3lb piece of meat that draws 20 Watts. The power efficiency difference may be explainable by the much slower frequency of brain computation (200 Hz vs. 2GHz). My impression is that the main obstacle to achieving a comparable volumetric density is that we haven't cracked 3d stacking of integr…

> The power efficiency difference may be explainable by the much slower frequency of brain computation (200 Hz vs. 2GHz).

Or by a more static design. A GPU can't do a thing without all the weights and shaders. There are benefits of this, you can easily swap one model for another. Human mind from the other hand is not reprogrammable. It can learn new tricks, but you cannot extract a firmware from one person and to upload it to another person.

Just imagine if every logical neuron of AI was a real thing, with physical connections to other neurons as inputs. No more need to have a high throughput memory, no more need to have compute units with gigaherz frequency.

Re: Advances in semiconductors are feeding the AI boom

#90

Post by Borg3 »

gary_0 wrote: post #86
__loam wrote: post #70

Earlier quoted context omitted.

People will spout off about how machine learning is based on the brain while having no idea how the brain works.

It is based on the brain, but only in the loosest possible terms; ML is a cargo cult of biology. It's kind of surprising that it works at all.

It works because well, its actually pretty primitive at its core. Whole learning process is actually pretty brutal. Doing millions of interations w/ random (and semi-random) adjustments.
Post reply on HN