Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RAM is the new disk – and how to measure its performance (2015)

tanelpoder.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: RAM is the new disk – and how to measure its performance (2015)

#51

Post by vlovich123 »

55873445216111 wrote: post #9
vlovich123 wrote: post #2

I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse. A lot of the software archite…

There are some memories supporting basic in-memory operations. For example: https://mosys.com/products/blazar-family/be3rmw-bandwidth-en... . This supports operations like read-modify-write within the memory device itself. (I have no affiliation with this company.) The barrier to adoption of this is not technical, it's economic. Memory industry has focused on making the highest capacity and lowest cost/bit products.…

I think there are niche use-cases that would warrant the cost/complexity trade-off. Namely cloud infrastructure for databases where you might be processing large amounts and transferring all that data first to the CPU is pretty ridiculous.

I think the complexity problem is solvable. If you can build such a thing for memory, you can reuse the same general concept for storage like NVME/SSD/spinning disk. It’s entirely possible this is warranted even in consumer devices as quite a bit of OS operations deal with modifying/querying memory whereas doing offload can win you some serious wins (making the machine feel way more snappy/interactive + more powerful to execute things locally).

I don’t have hope for traditional CPU designers so the question is whether someone can both design a memory system with this power AND a computational model that makes this easy to adopt in major languages while offering a perf win (given that a memory fetch costs ~100ns for ~256 bytes). It’s challenging and unlikely to come from x86 land where back Hw compatibility is extremely important. The innovation may eventually come from mobile land (which is where Apple is coming from) where memory controllers are custom designed and part of the SoC anyway, changing with each revision, so the hard part remains again how you make this fast, efficient, able to handle multiple concurrent programs (or at least have the OS control coarsest which program’s memory accesses were being prioritized), and language integration (so you can hand off an algorithm and it would be executed without upending existing software design knowledge). The OS integration could be even more intelligent - if the process/thread is spending its time processing memory without returning results to the CPU, just put it to sleep until the result is available (separating memory access and CPU utilization, resulting in drastically better utilization of cycles rather than naively waiting for each memory stall one at a time).

So tldr I agree with you totally. This is not going to happen if memory manufacturers continue to go for the “cheap and large” memory route. I do see hope that such concepts may be explored if we get more innovation in the CPU space.

Re: RAM is the new disk – and how to measure its performance (2015)

#52

Post by vlovich123 »

WJW wrote: post #39
Arelius wrote: post #34

Earlier quoted context omitted.

> Just the other day I found a case someone had rolled their own priority queue with a sorted array instead of a heap. I feel like this story needs an ending? Did somebody re-implement it using a heap and found significant performance wins? Or was the sorted array used on purpose to take advantage of some specific cache constraints and actually end up being a huge win?

The story ends (so far) with me filing a GH issue and offering a PR, my own benchmarking getting an average 3x speedup over a wide range of inputs. I happen to know that the maintainer has been on vacation for the last week and only got back a few days ago so no further discussion has been had yet.

A serious HW perf improvement will, generally improve everything, including shitty algorithms. There’s a reason that through the 90s and 2000s serious developers routinely ignored spending significant time optimizing code because the hardware doubled the speed of ALL code every ~12-18 months. So even though there are low-hanging fruit for software optimization, such low-hanging fruit can be missing from already heavily optimized code (eg games, scientific computations etc) and is orders of magnitude more expensive than “run this on newer HW for faster perf”. If you can do this “transparently” (max some OS/compiler integration and you can use the new Hw for perf wins), then the cost economics are powerful. If it’s a special-purpose piece of HW then it gets trickier and will be relegated to accelerating certain memory-heavy programs (like GPUs accelerate certain kinds of mass-data algorithms).

Re: RAM is the new disk – and how to measure its performance (2015)

#53

Post by vlovich123 »

dragontamer wrote: post #38
freeone3000 wrote: post #33

Earlier quoted context omitted.

why is that? we need the CPU to handle page fault interrupts, in order to populate the RAM. But assuming the page is already in RAM, there's no reason any of the memory accesses actually need to go through the CPU. (hardware can already raise interrupts; if the MMU can raise a page fault indicator, then you might be able to bypass CPU entirely until a new page needs to be loaded) moreover, if we have support for mmap…

Okay, lets say page-faults are off the table for some reason. Lets think about what can happen even if everything is in RAM still. * MMAP is still on the table: different processes can share RAM at different addresses. (Process#1 thinks the data is at memory location 0x90000000, Process#2 thinks the data is at 0x70000000, but in both cases, the data is at physical location 0x42). * Physical location 0x42 is a far-rea…

Or the mapping info for your process is shared with that coprocessor (the OS just keeps this around in RAM anyway). Heck, you could have an OS-provided code that needs to be loaded so that it can properly resolve that mapping (which is what the TLB does by the way - it has a well-defined structure and when it’s missing from the cache if I recall correctly it’ll fetch some info directly from RAM until it gets to a point where it has to generate a page fault).

I don’t disagree that the memory model becomes more complex. For one CPU cache invalidation becomes really tricky. So do memory coherence rules.

I’m less clear how mmap matters here. That’s just a mechanism the OS uses to hand out views into the page cache to the process - if you’ve solve the virtual-physical mapping (which you have to do) then mmap is not relevant.

You’re spot on though that the particular design decisions are critical for this to be successful - pick the wrong point on the complexity/cost/perf curve and your solution will definitely be DOA.

Re: RAM is the new disk – and how to measure its performance (2015)

#54

Post by vlovich123 »

tanelpoder wrote: post #14
nomel wrote: post #10

Earlier quoted context omitted.

I think the point is that this still eats into CPU memory bandwidth. Offloading could let the CPU use that memory for better purposes, especially since the stall is from memory access anyways.

When doing pointer chasing, then it's gonna be more of a latency problem, there can be plenty of memory bandwidth available, but we don't know which memory line we want to go to next, before the previous line (where the pointer resides) has been loaded. So, the CPUs spend a lot of time in "stalled backend" mode due to the big difference in CPU cycle latency vs RAM access latency. Offloading some operations closer to…

I’m thinking more of something like a GC mark and sweep. You don’t care about how long the mark (or even sweep) operations take as long as they finish eventually. Those operations would be the easiest to write an algorithm you could offload to run directly inside a memory controller. These would avoid GC pauses (if using a concurrent GC) while also avoiding any CPU cycles doing anything but a trivial amount of small operations.

What stood out to me with this article is that the fancy “intelligent” columnar indices were actually performing worse than a naive brute traversal of the data. This is because there is a limit to your ability to predict whether a brute solution is faster or slower ahead of time than one using a fancy data structure. So a HW-offload is nice because in theory it should have less of an impact and you can run memory-intensive operations concurrently with CPU-intensive operations with no costs (+ the memory heavy operations might run faster due to living closer to the memory and thus having faster interconnects and lower latency).

Additionally, such HW can take advantage of the peculiarities of how RAM works for even added benefit. Your memory allocator could mark a block as unused (in the simplest v0 of what such Hw code do). The HW can then avoid refreshing those rows reducing power usage slightly, improving latency (since refresh is a stop-the-world operation) if you can get fine-grained enough and it’s not $ prohibitive. Garbage collection would certainly be a lot more attractive of a solution for this kind of design which would drastically change the SW economics (ie Rust/C/C++ suddenly lose a whole lot of the practical perf wins they hold over other languages).

Re: RAM is the new disk – and how to measure its performance (2015)

#55

Post by vlovich123 »

geocar wrote: post #12
nomel wrote: post #10

Earlier quoted context omitted.

I think the point is that this still eats into CPU memory bandwidth. Offloading could let the CPU use that memory for better purposes, especially since the stall is from memory access anyways.

Linked-list chasing (as in vlovich123's example) isn't bandwidth-limited as much as it is latency: SMT helps here because you're able to enqueue multiple wait-states "simultaneously".

I agree it’s not bandwidth limited. But the CPU is stuck waiting around for the results of that operation. So offloading it to dedicated Hw let’s you do an important background task without needing the OS and CPU to spend any wall-clock time on it. Same reason you have DMAs and other coprocessors/accelerators in the first place. A good such solution will increase effective memory bandwidth (offloading heavy background-worthy tasks for the coprocessor) but also latency-sensitive tasks (eg walking a large set of linked lists, you don’t need to pull in a 256 byte cache line at a time and can instead pull the 8 bytes for the pointer).

I don’t pretend that this is easy or an unbridled obvious idea. I recognize there are smarter people than me who have considered this (and from the posts clearly is being explored commercially). I am attracted to the idea on first principles that the CPU is served well by specialized coprocessors that have a performance profile better suited for some specific subset of operations. For example, memcpy is the easiest one. We still don’t have a simple (good) memcpy instruction/accelerator even though we know that an insanely large amount of work the CPU ends up doing is copying memory around in some way or another (yes Intel has one that isn’t useful/good performing).

Re: RAM is the new disk – and how to measure its performance (2015)

#56

Post by vlovich123 »

totalZero wrote: post #46
vlovich123 wrote: post #2

I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse. A lot of the software archite…

Perhaps one problem with that is the modularity with various processors, which would have to know when they are configured with memory to which certain operations can be farmed out, and when they are configured with traditional memory that cannot manage those in-place operations.

SoCs are a good vector of this. Once a technology like this makes it into a space it’s unlikely to ever get removed if it’s actually useful (ie good perf/cost/power wins) as there’s a competitive advantage to having a better one. The longer it’s around, the more second order-effects you have (eg if you need SW integration, tooling etc) entrenching the tech (the same forces that make it difficult to get off the ground in the first place). Apple is a great example of being a company that can pull this off as they have the vertical integration to get it off the ground AND the commitment to ensure a consistent application within a vertical. It may not be within Apple’s business interests if this is primarily useful for “big data”. That seems unlikely to me though so you’d need other applications they might see as strategically useful. SQLite is used heavily at Apple, so accelerating such workloads may be useful. A good chunk of what an OS does overall is largely manipulate memory, so something like the Os scheduler could be bifurcated to pick the next task and have that ready, including modifying the various lists, so that the CPU is doing even less (although unlikely to be a good example as I suspect this is a relatively short operation since it happens so frequently that there’s not really any room for a win).

Re: RAM is the new disk – and how to measure its performance (2015)

#57

Post by Tom4hawk »

spockz wrote: post #35
WJW wrote: post #31

Earlier quoted context omitted.

You'd hope that the only way to better performance was HW specialisation, but there are SO MANY algorithmic improvements still to be made. Just the other day I found a case someone had rolled their own priority queue with a sorted array instead of a heap. In a fairly popular open source library too. There's still loads of performance to be gained just by being better programmers.

How do you propose to fix this? Should languages include high(er?) performance data structures in their standard libraries? Or possibly even include some segmentation for small/medium/huge data sets?

I'm not a person you asked but in my opinion answer is very simple: YES

Example: https://www.youtube.com/watch?v=HgtRAbE1nBM&t=43m15s

Re: RAM is the new disk – and how to measure its performance (2015)

#58

Post by [deleted] »

tanelpoder wrote: post #16
MayeulC wrote: post #11

Earlier quoted context omitted.

That bandwidth limitation could be due to Infinity Fabric, which seems to be rated at 42GBps (x2 as full duplex, though)? https://en.wikichip.org/wiki/amd/infinity_fabric

Yes, that's what I'm suspecting too, although with higher clocked RAM, I should have somewhat more bandwidth. My DIMMs are 3200 MT, so should be running at 1600 MHz. But I saw a note (not sure where) that Infinity Fabric can run up to 2933 MT on my machine and it would run in sync with memory with DIMMs only up to 2933 MT. Unfortunately my BIOS doesn't allow to downgrade the RAM "clock" from 3200 MT to 2933, thus Ini…

[deleted]

Re: RAM is the new disk – and how to measure its performance (2015)

#59

Post by geocar »

vlovich123 wrote: post #55
geocar wrote: post #12

Earlier quoted context omitted.

Linked-list chasing (as in vlovich123's example) isn't bandwidth-limited as much as it is latency: SMT helps here because you're able to enqueue multiple wait-states "simultaneously".

I agree it’s not bandwidth limited. But the CPU is stuck waiting around for the results of that operation. So offloading it to dedicated Hw let’s you do an important background task without needing the OS and CPU to spend any wall-clock time on it. Same reason you have DMAs and other coprocessors/accelerators in the first place. A good such solution will increase effective memory bandwidth (offloading heavy backgroun…

> I am attracted to the idea on first principles that the CPU is served well by specialized coprocessors that have a performance profile better suited for some specific subset of operations.

It certainly sounds promising, that's for sure, e.g.

https://link.springer.com/content/pdf/10.1007%2Fs00778-019-0...

Re: RAM is the new disk – and how to measure its performance (2015)

#60

Post by 1996 »

tanelpoder wrote: post #26
1996 wrote: post #20

Earlier quoted context omitted.

> I had to move SSD cards around so they'd use separate PCIe root complexes to avoid a PCIe CPU data transfer bottleneck I am doing similar things. Have you considered looking at how to control by software the PCI lanes assignment? Intel HSIO seems to be software configurable - except that usually, it's all done just by the bios. But as PCI specs allow for both device-side and host-side negotiations, it should be doa…

As this is an AMD machine - and as it's a workstation, not server, perhaps this is why they've restricted it in BIOS. I'm not too much of an expert in PCI express - but if this workstation has 4 PCIe root complexes/host bridges, each capable of x32 PCIe 4.0 lanes - and there are no multi-root PCIe switches, wouldn't a lane physically have to communicate with just one PCIe root complex/CPU "port"?

In theory yes.

In practice many things can be different. Carefully check what's happening under the hood.

Hopefully, manual configuration of the PCIe will become more commonplace, as most bioses are badly broken each in their own unique way - and unfixable.

Post reply on HN