Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I want a good parallel computer

raphlinus.github.io

Post reply on HN
Print view View original

71–80 of 209 posts

Re: I want a good parallel computer

#71

Post by hollerith »

zozbot234 wrote: post #68
sitkack wrote: post #60

Earlier quoted context omitted.

Hey don't shit on my retro alternative timeline nostalgia. We were all writing Lisp programs on 64 CPU Transputer systems with FPGA coprocessors, dynamically reconfigured in realtime with APL.

/s/LISP/Prolog and you've basically described the old "Fifth Generation" research project. Unfortunately it turns out that trying to parallelize Prolog is quite a nightmare, the language is really , really not built for it. So the whole thing was a dead-end in practice. Arguably we didn't have a real "fifth-gen" programming language prior to Rust, given how it manages to uniquely combine ease of writing parallel+conc…

Also, the Fifth Generation computer project subscribed to the old AI paradigm, not based on statistics.

Re: I want a good parallel computer

#72

Post by tliltocatl »

grg0 wrote: post #19

The issue is that programming a discrete GPU feels like programming a printer over a COM port, just with higher bandwidths. It's an entirely moronic programming model to be using in 2025. - You need to compile shader source/bytecode at runtime; you can't just "run" a program. - On NUMA/discrete, the GPU cannot just manipulate the data structures the CPU already has; gotta copy the whole thing over. And you better des…

Larrabee was something like that, didn't took off.

IMHO, the real issue is cache coherence. GPUs are spared from doing a lot of extra work by relaxing coherence guarantees quite a bit.

Regarding the vendor situation - that's basically how most of computing hardware is, save for the PC platform. And this exception is due to Microsoft successfully commoditizing their complements (which caused quite some woe on the software side back then).

Re: I want a good parallel computer

#73

Post by sitkack »

zozbot234 wrote: post #68
sitkack wrote: post #60

Earlier quoted context omitted.

Hey don't shit on my retro alternative timeline nostalgia. We were all writing Lisp programs on 64 CPU Transputer systems with FPGA coprocessors, dynamically reconfigured in realtime with APL.

/s/LISP/Prolog and you've basically described the old "Fifth Generation" research project. Unfortunately it turns out that trying to parallelize Prolog is quite a nightmare, the language is really , really not built for it. So the whole thing was a dead-end in practice. Arguably we didn't have a real "fifth-gen" programming language prior to Rust, given how it manages to uniquely combine ease of writing parallel+conc…

You probably already know about https://github.com/mthom/scryer-prolog are you saying we should accelerate scryer-prolog with a Grayskull board?

Re: I want a good parallel computer

#74

Post by deviantbit »

morphle wrote: post #63
deviantbit wrote: post #55

Earlier quoted context omitted.

Inmos was a disaster. No application ever shipped on one. EVER. It used a serial bus to resolve the problems that should have never been problems. Clearly you never wrote code for one. Each oslink couldn't reach more than 3 feet. What a disaster that entire architecture was.

I shipped 5 applications on an 800 Inmos Transputer supercomputer. Sold my parallel C compilers, macro Assembler. Also an OS, Macintosh Nubus interface card, Transputer graphics cards, a full paper copier and laserprinter. I know of dozens of successful products.

Sure you did. What were they? The only successful transputer was the T414 and it never made it outside academia.

Re: I want a good parallel computer

#75

Post by brzozowski »

grg0 wrote: post #19

The issue is that programming a discrete GPU feels like programming a printer over a COM port, just with higher bandwidths. It's an entirely moronic programming model to be using in 2025. - You need to compile shader source/bytecode at runtime; you can't just "run" a program. - On NUMA/discrete, the GPU cannot just manipulate the data structures the CPU already has; gotta copy the whole thing over. And you better des…

> What I would like is a CPU with a highly parallel array of "worker cores" all addressing the same memory...

I too am very interested in this model. The Linux kernel supports up to 4,096 cores [1] on a single machine. In practice, you can rent a c7a.metal-48xl [2] instance on AWS EC2 with 192 vCPU cores. As for programming models, I personally find the Java Streams API [3] extremely versatile for many programming workloads. It effectively gives a linear speedup on serial streams for free (with some caveats). If you need something more sophisticated, you can look into OpenMP [4], an API for shared-memory parallelization.

I agree it is time for some new ideas in this space.

[1]: https://www.phoronix.com/news/Perf-Support-2048-To-4096-Core...

[2]: https://aws.amazon.com/ec2/instance-types/c7a/

[3]: https://docs.oracle.com/en/java/javase/24/docs/api/java.base...

[4]: https://docs.alliancecan.ca/wiki/OpenMP

Re: I want a good parallel computer

#76

Post by vgatherps »

Grosvenor wrote: post #30
grg0 wrote: post #19

The issue is that programming a discrete GPU feels like programming a printer over a COM port, just with higher bandwidths. It's an entirely moronic programming model to be using in 2025. - You need to compile shader source/bytecode at runtime; you can't just "run" a program. - On NUMA/discrete, the GPU cannot just manipulate the data structures the CPU already has; gotta copy the whole thing over. And you better des…

What I want is a Linear Algebra interface - As Gilbert Strang taught it. I'll "program" in LinAlg, and a JIT can compile it to whatever wonky way your HW requires. I'm not willing to even know about the HW at all, the higher level my code the more opportunities for the JIT to optimize my code. What I really want is something like Mathematica that can JIT to GPU. As another commenter mentioned all the API's assume you…

https://github.com/jax-ml/jax

Re: I want a good parallel computer

#77

Post by amelius »

sitkack wrote: post #61
amelius wrote: post #52

Isn't the ONNX standard already going into the direction of programming a GPU using a computation graph? Could it be made more general?

It lacks support for the serial portions of the execution graph, but yes. You should play around with ONNX, it can be used for a lot more than just ML stuff.

What do you mean by serial portions? Aren't operations automatically serialized if there are dependencies between them?

Re: I want a good parallel computer

#78

Post by winwang »

sitkack wrote: post #70

This essay needs more work. Are you arguing for a better software abstraction, a different hardware abstraction or both? Lots of esoteric machines are name dropped, but it isn't clear how that helps your argument. Why not link to Vello? https://github.com/linebender/vello I think a stronger essay would at the end give the reader a clear view of what Good means and how to decide if a machine is closer to Good than ano…

Personal opinion: it's the software (and software tooling).

The hardware is good enough (even if we're only talking 10x efficiency). Part of the issue seems slightly cultural, i.e. repetitively putting down the idea of traditional task parallelism (not-super-SIMD/data parallelism) on GPUs. Obviously, one would lose a lot of efficiency if we literally ran 1 thread per warp. But it could be useful for lightly-data-parallel tasks (like typical CPU vectorization), or maybe using warp-wide semantics to implement something like a "software" microcode engine. Dumb example: implementing division with long division using multiplications and shifts.

Other things a GPU gives: insanely high memory bandwidth, programmable cache (shared memory), and (relatively) great atomic operations.

Re: I want a good parallel computer

#79

Post by zozbot234 »

tliltocatl wrote: post #72
grg0 wrote: post #19

The issue is that programming a discrete GPU feels like programming a printer over a COM port, just with higher bandwidths. It's an entirely moronic programming model to be using in 2025. - You need to compile shader source/bytecode at runtime; you can't just "run" a program. - On NUMA/discrete, the GPU cannot just manipulate the data structures the CPU already has; gotta copy the whole thing over. And you better des…

Larrabee was something like that, didn't took off. IMHO, the real issue is cache coherence. GPUs are spared from doing a lot of extra work by relaxing coherence guarantees quite a bit. Regarding the vendor situation - that's basically how most of computing hardware is, save for the PC platform. And this exception is due to Microsoft successfully commoditizing their complements (which caused quite some woe on the soft…

Is cache coherence a real issue, absent cache contention? AIUI, cache coherence protocols are sophisticated enough that they should readily adapt to workloads where the same physical memory locations are mostly not accessed concurrently except in pure "read only" mode. So even with a single global address space, it should be possible to make this work well enough if the programs are written as if they were running on separate memories.

Re: I want a good parallel computer

#80

Post by 0xbadcafebee »

If we had distributed operating systems and SSI kernels, your computer could use the idle cycles of other computers [that aren't on battery power]. People talk about a grid of solar houses, but we could've had personal/professional grid computing like 15 years ago. Nobody wanted to invest in it, I guess because chips kept getting faster.
Post reply on HN