Probably old news, but forth cpu arrays are so radically fun .. I wonder why they're not interesting.
Every forth core can pass forth to its neighbors, forth is very expressive yet tiny since it's a stack language.
Anyway my 2cents
David Patterson Says It’s Time for New Computer Architectures and Languages
21–30 of 204 posts
Re: David Patterson Says It’s Time for New Computer Architectures and Languages
#22It's really hard. Remember the Itanium and the Cell. You can build it, but they may not come. GPUs, though. Those have turned out to be very successful, they can be parallelized as much as you're willing to pay for, and they're good for some non-graphics tasks. Much of machine learning is a simple repetitive computation running at low precision. Special purpose hardware can do that very well. So what's the next usefu…
Just "clean" what we already have is more than enough, IMHO.
Things can already target X64/Arm but with a minimal kernel and a foundation without C at the base.
---
Even with a complete new CPU/Arch (that weirdly, I dream about that!) you can solve compatibility with VM/Containers. And provide a better user-land.
Re: David Patterson Says It’s Time for New Computer Architectures and Languages
#23I'm a VC and I've seen lots of pitches lately for various tech related to new architectures, be it GPU, FPGA or RISC. I'd add cryptocurrency to list workloads with insatiable computing demands.
Massive parallel prefix and reduction operations in RAM. Start with just the MPI standard reduction ops. 1000x faster than Von Neuman bottleneck.
Then an ASIC for sparse matrix multiply. Valiant's reduction of CFG parsing. Massive speedup to parsing workflows, and boon to security by getting rid of hand rolled parsers.
Re: David Patterson Says It’s Time for New Computer Architectures and Languages
#24Are there languages that have first-class support for representing/optimizing memory hierarchy characteristics? Optimizing C compilers, for example, may have extensions to force specific alignments: https://software.intel.com/en-us/articles/coding-for-perform... But I'm not aware of languages where e.g. declaring alignments is part of the base language. Awareness of L1, L2, L3 cache characteristics, plus NUMA nodes,…
Sequoia is what I thought of at first (mentioned in sibling comment), but Scala has something related:
http://scala-miniboxing.org/ldl/
Basically you can write an algorithm that is independent of the exact data layout, and then the compiler has some flexibility to work with.
I haven't worked with it all, just read the paper awhile ago. But I'm pretty sure they did something with SoA - AoS transformation (structure of arrays/array of structures). At the very least, the compiler can make the data smaller, which fits better in cache. Not sure if it can do anything more advanced than that (i.e. scaling to multiple levels of memory hierarchy rather than compacting data).
My memory of Sequoia is that it's very very far semantically from "Python" (mentioned in the original article). Their example was matrix multiplication, and the language had much more of a functional flavor (perhaps unsurprisingly). I'll have to go back and check it out again.
You can also think of MapReduce as a programming model that is locality-aware, and there have been several papers about multicore (not distributed) MapReduce.
You partition your data into smaller sections, then a user-defined "mapper" runs on each segment. The shuffle is an efficient distributed sort which you don't have to write -- it can be specialized to different problem sizes. And then the user-defined reducer also operates on a (different) small partition of data.
I think the real problem is that the ISA doesn't really expose the memory hierarchy to the programming language, not necessarily that the programming language doesn't expose it to the programmer. GPGPU might change that but I'm not entirely familiar with it.
Re: David Patterson Says It’s Time for New Computer Architectures and Languages
#25It's really hard. Remember the Itanium and the Cell. You can build it, but they may not come. GPUs, though. Those have turned out to be very successful, they can be parallelized as much as you're willing to pay for, and they're good for some non-graphics tasks. Much of machine learning is a simple repetitive computation running at low precision. Special purpose hardware can do that very well. So what's the next usefu…
Even SIMD, a feature of processors for over 20 years, is not handled well by compilers for automatic optimization. It seems like all code taking advantage of SIMD either drops into ASM, or uses a library from ARM/Intel to give descriptive C functions names for the underlying ASM.
Re: David Patterson Says It’s Time for New Computer Architectures and Languages
#26Well, there are some interesting new languages out there already: https://julialang.org/ In fact Moore’s law must have been the reason why Python is what it is today: “Who cares if it’s slow, tomorrow’s computers will be faster!”
In the case of Python, who cares if it's slow, it's a thin wrapper around a C lib!
Re: David Patterson Says It’s Time for New Computer Architectures and Languages
#27Probably old news, but forth cpu arrays are so radically fun .. I wonder why they're not interesting. Every forth core can pass forth to its neighbors, forth is very expressive yet tiny since it's a stack language. Anyway my 2cents
> the so-called Adaptive Compute Acceleration Platform (ACAP) will deliver 20x and 4x performance increases on deep learning and 5G radio processing, respectively, Xilinx claimed. The first chip, called Everest, will tape out this year in a 7nm process.
> The centerpiece of the new architecture is word-based array of tiles made up of VLIW vector processors, each with local memory and interconnect.
> tiles communicate with each other to create data paths that best suit their application.
https://www.eetimes.com/document.asp?doc_id=1333632
So GA144 is going to mainstream finally.
Re: David Patterson Says It’s Time for New Computer Architectures and Languages
#28I have to say that most bottlenecks are from lazy designs, not wimpy hardware. As a thought experiment, suppose parallel processing scaled easily without worrying about Amdahl's Law and similar bottlenecks. So then we put 128 cores into our computers. Software companies would eat up every single core eventually if there's no penalty for inefficient coding. They don't pay our electric bill, we do. It's kind of like freeways: the more you build, the more the traffic increases to fill them up again.
It is "nice" to be able to slap components together like magic legos and/or a Dagwood Sandwich rather than smartly tune and coordinate everything, but doing such is often computationally inefficient. The speed of light will probably put a hard upper limit on this practice that no clever language or architecture can work around to a notable degree.
Re: David Patterson Says It’s Time for New Computer Architectures and Languages
#29It's really hard. Remember the Itanium and the Cell. You can build it, but they may not come. GPUs, though. Those have turned out to be very successful, they can be parallelized as much as you're willing to pay for, and they're good for some non-graphics tasks. Much of machine learning is a simple repetitive computation running at low precision. Special purpose hardware can do that very well. So what's the next usefu…
> Remember the Itanium Well, why not. > Q: I feel like deja vu - at Hot Chips, Intel introduced VLIW-concept Itanium that pushed complexity onto the compiler. I see traces of that here. What are you doing to avoid the Itanium traps? How will you avoid IP from Intel? > A: Itanium was in-order VLIW, hope people will build compiler to get perf. We came from opposite direction - we use dynamic scheduling. We are not VLIW…
That captures a lot of my take on Itanium at the time: despite being "VLIW" it took on what at the time was a vast amount of CPU controller complexity without an obvious design win or clear high-performance compilation model. As an outsider, it felt like an extremum of Intel's flavor of design hubris.
It's interesting to compare the Itanium's approach to a VLIW contemporary from TI: the C6x VLIW DSP ISA. The C6x ISA took on very little controller complexity. Instead, it exposed a very clear operational and optimization model for the VLIW instruction blocks. An optimizing compiler did a decent first-step job at generating code. That was complemented by clear documentation of the optimization workflow, from compiler-stage iteration down to good guidance on the nuts and bolts of hand-optimizing. By comparison, IA64 was just ... a big head-scratching moment all around. I never got to work with C6x code professionally, but dang it looked like a blast to code and optimize for.
Re: David Patterson Says It’s Time for New Computer Architectures and Languages
#30Are there languages that have first-class support for representing/optimizing memory hierarchy characteristics? Optimizing C compilers, for example, may have extensions to force specific alignments: https://software.intel.com/en-us/articles/coding-for-perform... But I'm not aware of languages where e.g. declaring alignments is part of the base language. Awareness of L1, L2, L3 cache characteristics, plus NUMA nodes,…
Welp, architectures don't really "expose" cache characteristics either in a first-order fashion; they also go along with the idea that it's all just "memory" like you say. Obviously you can query your cache characteristics but there's only so much you can do.
Some embedded systems had scratchpads and it is possible to 'wire down cache' on some architectures to similar effect. But it would be tricky in general at the language level.
I have a project half on the boil that's aiming to produce a language aimed at these kind of usages. I'm figuring that the supposed insult hurled at C ("structured assembly") isn't really an insult at all but that someone should actually do that. At the moment C, with its giant raft of Undefined Behavior, isn't really that language any more.