Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The von Neumann bottleneck is impeding AI computing?

research.ibm.com

Post reply on HN
Print view View original

21–30 of 34 posts

Re: The von Neumann bottleneck is impeding AI computing?

#21

Post by anyfoo »

bahmboo wrote: post #9
Animats wrote: post #4

Actual result: "This new process promises to increase the number of optical fibers that can be connected at the edge of a chip, a measure known as beachfront density, by six times." Faster interconnects are always nice, but this is more like routine improvement.

"In recent inference tests run on a 3-billion-parameter LLM developed from IBM’s Granite-8B-Code-Base model, NorthPole was 47 times faster than the next most energy-efficient GPU and was 73 times more energy efficient than the next lowest latency GPU." It's also fascinating that they are experimenting with analog memory because it pairs so well with model weights

Yeah, analog memory fits so incredibly well. Who cares if it's not "exact" and fuzzes around a bit if it's only used for weights and has massive efficiency advantages. Weights are never "exact" themselves, and it doesn't matter if they don't always read exactly the same. You basically just get some extra "temperature" for free!

A bit beautiful that we might end up partially going back to analog computers, which were quickly replaced by digital ones.

Re: The von Neumann bottleneck is impeding AI computing?

#22

Post by ajross »

nyrikki wrote: post #3

Nit, ARM processors primarily use a modified Harvard architecture, including the raspberry pi pico.

That's valid jargon but from the wrong layer of the stack. A Harvard bus is about the separation of the "instruction" memory from "data" memory so that (pipelined) instructions can fetch from both in parallel. And in practice it's implemented in the L1 (and sometimes L2) cache, where you have separate icache/dcache blocks in front of a conceptually unified[1] memory space.

The "Von Neumann architecture" is the more basic idea that all the computation state outside the processor exists as a linear range of memory addresses which can be accessed randomly.

And the (largely correct) argument in the linked article is that ML computation is a poor fit for Von Neumann machines, as all the work needed to present that unified picture of memory to all the individual devices is largely wasted since (1) very little computation is actually done on individual fetches and (2) the connections between all the neurons are highly structured in practice (specific tensor rows and columns always go to the same places), so a simpler architecture might be a better use of die space.

[1] Not actually unified, because there's a page translation, IO-MMUs, fabric mappings and security boundaries all over the place that prevents different pieces of hardware from actually seeing the same memory. But that's the idea anyway.

Re: The von Neumann bottleneck is impeding AI computing?

#23

Post by UltraSane »

Animats wrote: post #4

Actual result: "This new process promises to increase the number of optical fibers that can be connected at the edge of a chip, a measure known as beachfront density, by six times." Faster interconnects are always nice, but this is more like routine improvement.

In-Memory compute has nothing to do with connecting optical fibers to a chip.

Re: The von Neumann bottleneck is impeding AI computing?

#24

Post by nyrikki »

NooneAtAll3 wrote: post #5
nyrikki wrote: post #3

Nit, ARM processors primarily use a modified Harvard architecture, including the raspberry pi pico.

this isn't about Harvard/VonNeuman split/no-split between i-cache and d-cache I think this post is more about... compute in memory? if I got it right?

Here is John Backus' original paper[0], which is an easy read, but note what he calls functional programming_ has nothing to do with lambda calculus, Haskel etc... it is the APL family.

He is absolutely one of IBM's historical rockstars. IMHO they are invoking him to sell their NorthPole chips which have on-die memory distributed between the processing components and probably has value.

> In its simplest form a von Neumann computer has three parts: a central processing unit (or CPU), a store, and a connecting tube that can transmit a single word between the CPU and the store (and send an address to the store). I propose to call this tube the yon Neumann bottleneck. The task of a program is to change the contents of the store in some major way; when one considers that this task must be accomplished entirely by pumping single words back and forth through the von Neumann bottleneck, the reason for its name becomes clear.

IMHO IBM is invoking John Backus' work to sell what may be an absolutely great product but are really just ASICs and don't relate to his machine or programming language limits.

[0] https://dl.acm.org/doi/pdf/10.1145/359576.359579

Re: The von Neumann bottleneck is impeding AI computing?

#25

Post by magicalhippo »

anyfoo wrote: post #21
bahmboo wrote: post #9

Earlier quoted context omitted.

"In recent inference tests run on a 3-billion-parameter LLM developed from IBM’s Granite-8B-Code-Base model, NorthPole was 47 times faster than the next most energy-efficient GPU and was 73 times more energy efficient than the next lowest latency GPU." It's also fascinating that they are experimenting with analog memory because it pairs so well with model weights

Yeah, analog memory fits so incredibly well. Who cares if it's not "exact" and fuzzes around a bit if it's only used for weights and has massive efficiency advantages. Weights are never "exact" themselves, and it doesn't matter if they don't always read exactly the same. You basically just get some extra "temperature" for free! A bit beautiful that we might end up partially going back to analog computers, which were…

> A bit beautiful that we might end up partially going back to analog computers, which were quickly replaced by digital ones.

How long till we get a Ben Eater-style video about someone making a basic analog neural network using some DACs, analog multipliers[1] and bucket-brigade chips[2] for intermediate values?

[1]: https://www.analog.com/media/en/training-seminars/tutorials/...

[2]: https://en.wikipedia.org/wiki/Bucket-brigade_device

Re: The von Neumann bottleneck is impeding AI computing?

#26

Post by yellowcake0 »

stego-tech wrote: post #16

IBM initially leads with the more salient point (current architecture designs are hindering frontier computing concepts), then just kinda…relents into iterative improvement. Which is fine! I am all for iterative improvements, it’s how we got to where we are today. I just wish more folks would start openly admitting that our current architecture designs are broadly based off “low hanging fruit” of early electronics an…

Isn't returning to an era of chip architecture experimentation exactly what would be required to explore new and better alternatives?

Re: The von Neumann bottleneck is impeding AI computing?

#27

Post by rapjr9 »

About 20 years ago the CS community was getting excited about optical memory. It promised to be huge, must faster than static RAM, and hold it's state. Tied directly to the CPU as a very large cache+RAM replacement it would have revolutionized computing. There were other advantages besides speed. One was that you could just pause the CPU, put the computer to sleep, then wake it up later and everything was already in RAM and computation would continue where it left off. Instant boot. Running apps would be instant, they were already in RAM and could be run in place. Prototypes existed but optical memory never happened commercially. Not sure I remember why, maybe couldn't scale, or manufacturing problems. There was also the problem that code is never perfect, so what to do when something stored became corrupted? Without a boot phase there would be no integrity checks.

Re: The von Neumann bottleneck is impeding AI computing?

#29

Post by jesuswasrasta »

abrookewood wrote: post #28

Off topic, but does the sentence structure of STATEMENT-QUESTION MARK have a name? It's pretty annoying in my opinion. Why not write "IS the von Neumann bottleneck impeding AI computing?" instead?

As an Italian, it translates as we actually state questions, so it feels natural to me :)

But you're right, I think it's not even grammarly correct.

Anyway, I'd like always to remember this about headlines as a question: https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines

Re: The von Neumann bottleneck is impeding AI computing?

#30

Post by imtringued »

bahmboo wrote: post #9
Animats wrote: post #4

Actual result: "This new process promises to increase the number of optical fibers that can be connected at the edge of a chip, a measure known as beachfront density, by six times." Faster interconnects are always nice, but this is more like routine improvement.

"In recent inference tests run on a 3-billion-parameter LLM developed from IBM’s Granite-8B-Code-Base model, NorthPole was 47 times faster than the next most energy-efficient GPU and was 73 times more energy efficient than the next lowest latency GPU." It's also fascinating that they are experimenting with analog memory because it pairs so well with model weights

Their NorthPole chip doesn't look much different than the Groq LPU or Tenstorrent's hardware or even just AMD's NPU design. The tenstorrent cards have a pretty big amount of SRAM considering their price.
Post reply on HN