RAM is the new disk – and how to measure its performance (2015)
1–10 of 64 posts
Re: RAM is the new disk – and how to measure its performance (2015)
#2I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse.
A lot of the software architecture theory we learn is based on existing HW paradigms without much thought being given to how we can change HW paradigms. By nature HW is massively parallel but where physical distance from compute = latency (vs the ultimately serial execution nature of traditional CPUs that can process all data at blistering speed but only one at a time with some SIMD exceptions). There are real-world benefits to this kind of design - memory is cheap and simple to manufacture and abundantly available. The downside though is that the CPU is sitting doing nothing but waiting for memory most of the time, especially when processing large data sets.
Imagine how efficient a GC algorithm would be if it could compute a result in the background just doing a concurrent mark and sweep, perhaps as part of a DRAM refresh cycle so that you could even choose to stop refreshing that RAM because your application no longer needs that row.
The power and performance savings are pretty enticing.
Re: RAM is the new disk – and how to measure its performance (2015)
#3meanwhile, disk is potentially getting to be as fast as ram, throughput wise.
128 lanes of pcie 4.0 is 256GBps iirc. epyc's 8 channel ddr4-3200 otoh is good for 208GBps.
Let's Encrypt stopped a little short, using 24x nvme disks (it fits in a 2U though so that's nice)[1]. that could be up to 96 of 128 pcie links in use. with the right ssds, working on large data-objects, that'd be somewhere a bit under 192GBps versus the max 208GBps of their ram.
in truth, ram's random access capabilities are far better, there's much less overhead (although nvme is pretty good). and i'm not sure i've ever seen anyone try to confirm that those 128 lanes of pcie on epyc aren't oversubscribed, that devices really can push that much data around. note that this doesn't necessarily even have to mean using the cpu; pci p2p is where it's at for in-the-know folks doing nvme, network, and gpu data-pushing; epyc's io-die is acting like a data-packet switch in these conditions, rather than having the cpu process/crunch these peripheral's data.
Re: RAM is the new disk – and how to measure its performance (2015)
#4I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse. A lot of the software archite…
Oracle (Sun) latest CPUs support something called DAX (data analytics extension, not the same thing as the Intel DAX - direct access extensions). It's a coprocessor that allows offloading simpler operations closer to RAM, apparently:
"DAX is an integrated co-processor which provides a specialized set of instructions that can run very selective functionality – Scan, Extract, Select, Filter, and Translate – at fast speeds. The multiple DAX share the same memory interface with the processors cores so the DAX can take full advantage of the 140-160 GB/sec memory bandwidth of the SPARC M7 processor."
https://blogs.oracle.com/bestperf/accelerating-spark-sql-usi...
Re: RAM is the new disk – and how to measure its performance (2015)
#5meanwhile, disk is potentially getting to be as fast as ram, throughput wise. 128 lanes of pcie 4.0 is 256GBps iirc. epyc's 8 channel ddr4-3200 otoh is good for 208GBps. Let's Encrypt stopped a little short, using 24x nvme disks (it fits in a 2U though so that's nice)[1]. that could be up to 96 of 128 pcie links in use. with the right ssds, working on large data-objects, that'd be somewhere a bit under 192GBps versus…
Author (of the RAM article) here:
Indeed, you can go further, but got to plan for other bandwidth needed by other peripherals, data movement and inter-CPU bandwidth (NUMA) and intra-CPU-core bandwidth limitations too (AMD's infinity fabric is point to point between chiplets, but intel has some ring-bus architecture for moving bits between CPU cores).
I got my Lenovo ThinkStation P620 workstation (with AMD Zen-2 ThreadRipper Pro WX, 8-memory channels like EPYC) to scan 10 x PCIe 4.0 SSDs at 66 GB/s (I had to move SSD cards around so they'd use separate PCIe root complexes to avoid a PCIe CPU data transfer bottleneck. And even with doing I/O through 3 PCIe root complexes (out of 4 connected to that CPU), I seem to be hitting some inter-CPU-core bandwidth limitation. The throughput differs depending on which specific CPU cores happen to run the processes doing I/Os against different SSDs.
Planning to publish some blog entries about these I/O tests but a teaser tweet is here (11M IOPS with a single-socket ThreadRipper workstation - it's not even a NUMA server! :-)
Re: RAM is the new disk – and how to measure its performance (2015)
#6I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse. A lot of the software archite…
Interestingly enough, this is why Simultaneous multithreading [1] exists!
The revelation that "The CPU could be doing useful work while stalled waiting on a load" has lead to CPU designers "faking" the number of cores available to the OS to allow the CPU to do more useful work while different "threads" are paused waiting on memory to come back with the data they need.
[1] https://en.wikipedia.org/wiki/Simultaneous_multithreading
Re: RAM is the new disk – and how to measure its performance (2015)
#7I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse. A lot of the software archite…
Interestingly enough, this is why Simultaneous multithreading [1] exists! The revelation that "The CPU could be doing useful work while stalled waiting on a load" has lead to CPU designers "faking" the number of cores available to the OS to allow the CPU to do more useful work while different "threads" are paused waiting on memory to come back with the data they need. [1] https://en.wikipedia.org/wiki/Simultaneous_mu…
And all kinds of manual prefetching instructions built into the CPUs (for compilers to use) and prefetching algorithms & predictors built into the CPUs too!
Re: RAM is the new disk – and how to measure its performance (2015)
#8I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse. A lot of the software archite…
Even just "go to memory location X+Y, load the value there, set X = that value, repeat N times" would allow fast linked list traversal, as well as doing various high-level pointer chasing faster.
Re: RAM is the new disk – and how to measure its performance (2015)
#9I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse. A lot of the software archite…
There are some memories supporting basic in-memory operations. For example: https://mosys.com/products/blazar-family/be3rmw-bandwidth-en.... This supports operations like read-modify-write within the memory device itself. (I have no affiliation with this company.)
The barrier to adoption of this is not technical, it's economic. Memory industry has focused on making the highest capacity and lowest cost/bit products. This drives high manufacturing volume which drives economies of scale. Memory products with integrated functions are inherently niche, and therefore do not have anywhere near the market size and economy of scale. Designers have decided (historically) that it is cheaper at the system level to keep the logic operations within the CPU and use a "dumb" commodity memory, even though this necessitates more bandwidth usage. (It's a complex engineering trade-off.)
With logic performance continuing to scale faster than memory bandwidth, at some point an architecture that reduces the required memory bandwidth (such as computing in-memory) might start to make sense economically.
Re: RAM is the new disk – and how to measure its performance (2015)
#10I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse. A lot of the software archite…
Interestingly enough, this is why Simultaneous multithreading [1] exists! The revelation that "The CPU could be doing useful work while stalled waiting on a load" has lead to CPU designers "faking" the number of cores available to the OS to allow the CPU to do more useful work while different "threads" are paused waiting on memory to come back with the data they need. [1] https://en.wikipedia.org/wiki/Simultaneous_mu…
I think the point is that this still eats into CPU memory bandwidth. Offloading could let the CPU use that memory for better purposes, especially since the stall is from memory access anyways.