The von Neumann bottleneck is impeding AI computing? • Hacker News Board

Re: The von Neumann bottleneck is impeding AI computing?

#31

Post by stego-tech » Sat, Sep 27, 2025, 3:22 PM UTC

yellowcake0 wrote: post #26

stego-tech wrote: post #16

IBM initially leads with the more salient point (current architecture designs are hindering frontier computing concepts), then just kinda…relents into iterative improvement. Which is fine! I am all for iterative improvements, it’s how we got to where we are today. I just wish more folks would start openly admitting that our current architecture designs are broadly based off “low hanging fruit” of early electronics an…

Isn't returning to an era of chip architecture experimentation exactly what would be required to explore new and better alternatives?

Custom architecture, yes, but that's not what we're seeing. Companies aren't inventing new computing paradigms, just grabbing stuff off the shelf and shoe-horning desired accelerators onto the package for a spiffier product targeting their demographic.

Re: The von Neumann bottleneck is impeding AI computing?

#32

Post by bobmcnamara » Sat, Sep 27, 2025, 4:32 PM UTC

nyrikki wrote: post #18

bobmcnamara wrote: post #8

Earlier quoted context omitted.

Nit: RP2040 is a Von Neumann. There's only one AHB port on the m0. Edit: see also ARM7TDMI, Cortex-m0/0+/1, and probably a few others. All the big stuff is modified Harvard or very rarely pure Harvard.

You are correct I should have specified pico2 That said AVH-lite is called lite because it is a simplified form of the arm norm. The RP2350 can issue one fetch and one load/store per cycle, and that is that almost everything called a CPU and not a MCU will have ABH5 or better. The “von Neumann bottleneck” was (when I went to school) that the CPU cannot simultaneously fetch an instruction and read/write data from or t…

> That said AVH-lite is called lite because it is a simplified form of the arm norm.

> The RP2350 can issue one fetch and one load/store per cycle, and that is that almost everything called a CPU and not a MCU will have ABH5 or better.

I mean, yes, but I'm not sure I see your point. The Harvard vs Von Neumann architectural difference is more related to the number of AHB ports on the core.

> That doesn’t apply to smartphones, PCs or servers even in the intel world due to instruction caches etc…

I wouldn't confuse instruction caches with Harvard vs Von Neumann either - loads of Von Neumann machines have instruction or Flash caches too.

It's also not uncommon to run into Von Neumann cores in mobile and PC chips, just as peripheral co-processors.

It is just middle aged guy who did this stuff for years...

Re: The von Neumann bottleneck is impeding AI computing?

#33

Post by westurner » Sat, Sep 27, 2025, 6:57 PM UTC

imtringued wrote: post #30

bahmboo wrote: post #9

Earlier quoted context omitted.

"In recent inference tests run on a 3-billion-parameter LLM developed from IBM’s Granite-8B-Code-Base model, NorthPole was 47 times faster than the next most energy-efficient GPU and was 73 times more energy efficient than the next lowest latency GPU." It's also fascinating that they are experimenting with analog memory because it pairs so well with model weights

Their NorthPole chip doesn't look much different than the Groq LPU or Tenstorrent's hardware or even just AMD's NPU design. The tenstorrent cards have a pretty big amount of SRAM considering their price.

How does Cerebras WSE-3 with 44GB of 'L2' on-chip SRAM compare to Google's TPUs, Tesla's TPUs, NorthPole, Groq LPU, Tenstorrent's, and AMD's NPU designs?

Re: The von Neumann bottleneck is impeding AI computing?

#34

Post by bahmboo » Sat, Sep 27, 2025, 9:19 PM UTC

imtringued wrote: post #30

bahmboo wrote: post #9

Earlier quoted context omitted.

"In recent inference tests run on a 3-billion-parameter LLM developed from IBM’s Granite-8B-Code-Base model, NorthPole was 47 times faster than the next most energy-efficient GPU and was 73 times more energy efficient than the next lowest latency GPU." It's also fascinating that they are experimenting with analog memory because it pairs so well with model weights

Their NorthPole chip doesn't look much different than the Groq LPU or Tenstorrent's hardware or even just AMD's NPU design. The tenstorrent cards have a pretty big amount of SRAM considering their price.

I am not an expert on this but reading Groq's description of their hardware it still has a compute/memory split. They make the memory super fast so it can fully feed the CPU without latency (80 terabytes second!). In the end is it much different than moving the ALU into memory like IBM is doing? The goal for both is to eliminate the memory bottleneck so there can be a variety of valid approaches.