Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want a good parallel computer

raphlinus.github.io

Post reply on HN
Print view View original

41–50 of 209 posts

Re: I want a good parallel computer

#41

Post by convolvatron »

KerrAvon wrote: post #16
mikewarot wrote: post #14

Earlier quoted context omitted.

The key transformation required to make any parallel architecture work is going to be taking a program that humans can understand, and translating it into a directed acyclic graph of logical Boolean operations. This type of intermediate representation could then be broken up into little chunks for all those small CPUS. It could be executed very slowly using just a few logic gates and enough ram to hold the state, or…

Isn't that the Connection Machine architecture?

the CM architecture or programming model wasn't really a DAG. It was more like tensors of arbitrary rank with power of two sizes. Tensor operations themselves were serialized, but each of them ran in parallel. It was however much nicer than coding vectors today - it included Blelloch scans, generalizied scatter-gather, and systolic-esque nearest neighbor operations (shift this tensor in the positive direction along this axis). I would love to see a language like this that runs on modern GPUs, but its really not sufficiently general to get good performance there I think.

Re: I want a good parallel computer

#42

Post by dekhn »

There are many intertwined issues here. One of the reasons we can't have a good parallel computer is that you need to get a large number of people to adopt your device for development purposes, and they need to have a large community of people who can run their code. Great projects die all the time because a slightly worse, but more ubiquitous technology prevents flowering of new approaches. There are economies of scale that feed back into ever-improving iterations of existing systems.

Simply porting existing successful codes from CPU to GPU can be a major undertaking and if there aren't any experts who can write something that drive immediate sales, a project can die on the vine.

See for example https://en.wikipedia.org/wiki/Cray_MTA when I was first asked to try this machine, it was pitched as "run a million threads, the system will context switch between threads when they block on memory and run them when the memory is ready". It never really made it on its own as a supercomputer, but lots of the ideas made it to GPUs.

AMD and others have explored the idea of moving the GPU closer to the CPU by placing it directly onto the same memory crossbar. Instead of the GPU connecting to the PCI express controller, it gets dropped into a socket just like a CPU.

I've found the best strategy is to target my development for what the high end consumers are buying in 2 years - this is similar to many games, which launch with terrible performance on the fastest commericially available card, then runs great 2 years later when the next gen of cards arrives ("Can it run crysis?")

Re: I want a good parallel computer

#43

Post by morphle »

deviantbit wrote: post #36
morphle wrote: post #33

Earlier quoted context omitted.

If you fix the pages tables (partial tutorial online) you can have continuous unified address space on Apple Silicon.

Let’s be honest, saying “just fix the page tables” is like telling someone they can fly if they “just rewrite gravity.” Yes, on Apple Silicon, the hardware supports shared physical memory, and with enough “convincing”, you can rig up a contiguous virtual address space for both the CPU and GPU. Apple’s unified memory architecture makes that possible, but Apple’s APIs and memory managers don’t expose this easily or saf…

Not true. "just fix the page tables" took me 4 hours. And only 15 minutes with the Linux kernel on Apple Silicon.

Re: I want a good parallel computer

#44

Post by IshKebab »

turtletontine wrote: post #12
IshKebab wrote: post #3

Having worked for a company that made a "hundreds of small CPUs on a single chip", I can tell you now that they're all going to fail because the programming model is too weird, and nobody will write software for them. Whatever comes next will be a GPU with extra capabilities, not a totally new architecture. Probably an nVidia GPU.

Could you elaborate on this? How does many-small-CPUs make for a weirder programming model than a GPU? Im no expert, but I’ve done my fair share of parallel HPC stuff using MPI, and a little bit of Cuda. And to me the GPU programming model is far far “weirder” and harder to code for than the many-CPUs model. (Granted, I’m assuming you’re describing a different regime?)

I mean weird compared to what already exists.

Re: I want a good parallel computer

#45

Post by bigyabai »

andrewstuart wrote: post #22
bigyabai wrote: post #17

Earlier quoted context omitted.

> It’s faster at AI than an Nvidia RTX4090, because 96GB of the 128GB can be allocated to the GPU memory space I love AMD's Ryzen chips and will recommend their laptops over an Nvidia model all day. However, this is a pretty facetious comparison that falls apart when you normalize the memory. Any chip can be memory bottlenecked, and if we take away that arbitrary precondition the Strix Halo gets trounced in terms of…

“ AMD also claims its Strix Halo APUs can deliver 2.2x more tokens per second than the RTX 4090 when running the Llama 70B LLM (Large Language Model) at 1/6th the TDP (75W).” https://www.tomshardware.com/pc-components/cpus/amd-slides-c... You could argue it’s invalid claim because it’s from AMD not independent.

This is still a memory-constrained benchmark. The smallest Llama 70B model (gguf-q2) doesn't fit in-memory so is bottlenecked by your PCIe connector. It's a valid benchmark, but it's still guilty of being stacked in the exact way I described before.

A comparison of 7B/13B/32B model performance would actually test the compute performance of either card. AMD is appealing to the consumers that don't feel served by Nvidia's gaming lineup, which is fine but also doomed if Nvidia brings their DGX Spark lineup to the mobile form factor.

Re: I want a good parallel computer

#46

Post by CyberDildonics »

snovymgodym wrote: post #11
IshKebab wrote: post #3

Having worked for a company that made a "hundreds of small CPUs on a single chip", I can tell you now that they're all going to fail because the programming model is too weird, and nobody will write software for them. Whatever comes next will be a GPU with extra capabilities, not a totally new architecture. Probably an nVidia GPU.

Reminds me of Itanium

How is that at all like Itanium except for the superficial headline level where people say they are hard to program?

Re: I want a good parallel computer

#47

Post by bigyabai »

dist-epoch wrote: post #20
bigyabai wrote: post #17

Earlier quoted context omitted.

> It’s faster at AI than an Nvidia RTX4090, because 96GB of the 128GB can be allocated to the GPU memory space I love AMD's Ryzen chips and will recommend their laptops over an Nvidia model all day. However, this is a pretty facetious comparison that falls apart when you normalize the memory. Any chip can be memory bottlenecked, and if we take away that arbitrary precondition the Strix Halo gets trounced in terms of…

> However, this is a pretty facetious comparison that falls apart when you normalize the memory Why would you normalize though? You can't buy a 96 GB RTX4090. So it's fair to compare the whole deal, slowish APU with large RAM versus very fast GPU with limited RAM.

It is fair, it should just be contextualized with a comparison of 13B or 32B models as well. This is one of those Apple marketing moves where a very specific benchmark has been cherry-picked for a "2.2x improvement!" headline that people online misconstrue.

Re: I want a good parallel computer

#48

Post by smallmancontrov »

grg0 wrote: post #19

The issue is that programming a discrete GPU feels like programming a printer over a COM port, just with higher bandwidths. It's an entirely moronic programming model to be using in 2025. - You need to compile shader source/bytecode at runtime; you can't just "run" a program. - On NUMA/discrete, the GPU cannot just manipulate the data structures the CPU already has; gotta copy the whole thing over. And you better des…

You can have that today. Just go out and buy more CPUs until they have enough cores to equal the number of SMs in your GPU (or memory bandwidth, or whatever). The problem is that the overhead of being general purpose -- prefetch, speculative execution, permissions, complex shared cache hierarchies, etc -- comes at a cost. I wish it was free, too. Everyone does. But it just isn't. If you have a workload that can jettison or amortize these costs due to being embarrassingly parallel, the winning strategy is to do so, and those workloads are common enough that we have hardware for column A and hardware for column B.

Re: I want a good parallel computer

#49

Post by deviantbit »

"I believe there are two main things holding it back."

He really science’d the heck out of that one. I’m getting tired of seeing opinions dressed up as insight—especially when they’re this detached from how real systems actually work.

I worked on the Cell processor and I can tell you it was a nightmare. It demanded an unrealistic amount of micromanagement and gave developers rope to hang themselves with. There’s a reason it didn’t survive.

What amazes me more is the comment section—full of people waxing nostalgic for architectures they clearly never had to ship stable software on. They forget why we moved on. Modern systems are built with constraints like memory protection, isolation, and stability in mind. You can’t just “flatten address spaces” and ignore the consequences. That’s how you end up with security holes, random crashes, and broken multi-tasking. There's a whole generation of engineers that don't seem to realize why we architected things this way in the first place.

I will take how things are today over how things used to be in a heart beat. I really believe I need to spend 2-weeks requiring students write code on an Amiga, and the programs have to run at the same time. If anyone of them crashes, they all will fail my course. A new found appreciation may flourish.

Re: I want a good parallel computer

#50

Post by newpavlov »

grg0 wrote: post #19

The issue is that programming a discrete GPU feels like programming a printer over a COM port, just with higher bandwidths. It's an entirely moronic programming model to be using in 2025. - You need to compile shader source/bytecode at runtime; you can't just "run" a program. - On NUMA/discrete, the GPU cannot just manipulate the data structures the CPU already has; gotta copy the whole thing over. And you better des…

>What I would like is a CPU with a highly parallel array of "worker cores" all addressing the same memory and speaking the same goddamn language that the CPU does.

For "embarrassingly parallel" jobs vector extensions start to eat tiny bits of the GPU pie.

Unfortunately, just slapping thousands of cores works poorly in practice. You quickly get into the synchronization wall caused by unified memory. GPUs cleverly work around this issue by using numerous tricks often hidden behind extremely complex drivers (IIRC CUDA exposes some of this complexity).

The future may be in a more explicit NUMA, i.e. in the "network of cores". Such hardware would expose a lot of cores with their own private memory (explicit caches, if you will) and you would need to explicitly transact with the bigger global memory. But, unfortunately, programming for such hardware would be much harder (especially if code has to be universal enough to target different specs), so I don't have high hopes for such paradigm to become massively popular.

Post reply on HN