Earlier quoted context omitted.
Now? I had found some used epyc servers with 2TB ddr4 ram for around 5k usd yesteryear. Too bad I didn't purchase it.
He said GPU servers
He didn't say VRAM.
GPU servers are just servers with GPUs.
I/O is no longer the bottleneck? (2022)
61–70 of 133 posts
Re: I/O is no longer the bottleneck? (2022)
#62Not a new idea, but it's intriguing to think about an architecture that's just: CPU caches nonvolatile storage What if you could take it for granted that mmap()ing a file has the exact same performance characteristics as malloc(), aside from the data not going away when you free the address space? What if arbitrary program memory could be given a filename and casually handed off to the OS to make persistent? A lot of…
You can get something like this from Linux today. (And mmap is actually how you request memory from the kernel in almost all cases.) It's just that mmap is slower than using read/write, because the kernel knows less about your data access patterns and thus has to guess for how to populate caches etc.
Yes, I know mmap already sort of allows this (and has for well over a decade). To elaborate: when I want to, say, parse a megabytes-sized file, I don't muck about with mmap(), I just read() into a buffer; it's simple and it's fast enough even though I'm just wasting microseconds waiting for bytes on one fast chip to get copied into another slightly faster chip (and then copied into CPU cache). If I'm dealing with a larger amount of data, I'd be tempted to use database middleware to figure out all the platform-specific shuffling between RAM and disk (designed on the assumption that the disk is spinning rust, cough), but that pulls in yet another chunk of complexity.
Instead, imagine if I could just state in one line of system-agnostic code "give me a pointer to /home/user/abc" and it does the right thing--assuming there was some way around mmap's current set of caveats. Imagine if I could turn a memory buffer into a file in one line of code and it Just Worked. Imagine if the OS treated my M.2 SSD as just another chip on the bus instead of still having a good amount of code on the hot path that assumes I'm manually sending bytes to a mechanical drive.
Re: I/O is no longer the bottleneck? (2022)
#63still my bottleneck generally speaking, cloudvm/container filesys i/o sucks
Re: I/O is no longer the bottleneck? (2022)
#64Re: I/O is no longer the bottleneck? (2022)
#65Earlier quoted context omitted.
Have you benchmarked against Rust's rkyv, too?
No, I have not benchmarked yet against other languages. Rkyv is Rust only. One primary difference is that Rkyv does not support in-place mutation. So any modification of a message requires full reserialization, unlike Lite³.
Rkyv supports a limited amount of in-place mutation. But not in general like your approach does.
Re: I/O is no longer the bottleneck? (2022)
#66Earlier quoted context omitted.
The limit is the number of outstanding cache line requests to the memory controller. CPUs have a fixed number of slots for this, around 10-12 usually. Intel calls them LFBs (Line Fill Buffers) and AMD MSHRs (Miss Status Holding Registers). When the slots are filled, the CPU can issue no more requests and has to wait for them to complete. Apple M chips (probably) have more slots and the memory is physically packaged t…
I assume these must be really expensive? Otherwise it seems like a great way to improve throughput on low concurrency tasks.
At least in older CPUs the caches were SRAM (static RAM). It is complicated but requires no refreshing. DRAM is basically just a capacitor per bit and capacitors leak so you constantly have to refresh the entire memory space. When the CPU sends a request to RAM, the memory controller might be too busy refreshing the soon to decay parts to actually respond right away. And if I recall correctly when you read from DRAM you destroy what was there so the process is to read it, then write it back, then send the answer to the CPU which is just a lot of steps. But the price and die size difference is huge so we use GB or TB levels of DRAM and MB levels of SRAM.
Re: I/O is no longer the bottleneck? (2022)
#67Earlier quoted context omitted.
> 6 GB/s Samsung is selling NVMe SSDs claiming 14 GB/s sequential read speed.
Any code that's reading/writing to SSD needs to use multiple cores. The SSD is faster than a single CPU core.
That doesn’t sound right. A single core should more than fast enough to saturate IOPs (particularly with iouring) unless you’re doing something insane like a lot of small writes. A write of 16mib or 32mob should still be about 1 ssd iop - more CPUs shouldn’t help(and in fact should be slower if you have 2 16mib IOPs vs 1 32mib iop)
Re: I/O is no longer the bottleneck? (2022)
#68Re: I/O is no longer the bottleneck? (2022)
#69Earlier quoted context omitted.
Old IBM's term for RAM was "storage."
I wonder whether the current huge funding in AI will ever lead to a revolution in computer architecture. Modern PCIe/CXL is already starting to blur the difference between memory and I/O. Maybe the future is going to be that CPUs, RAM, storage devices, GPUs and other devices are going to directly address one another like a mesh network. Maybe the entire virtual memory model will change to include everything to be add…
Don't we already almost (but not quite) have that? PCIe devices can talk directly to each other (still centralized AFAIK though) and from the programmers perspective everything you mentioned is mapped into a single unified address space prior to use (admittedly piecemeal via mmap for peripherals).
Technically there's nothing stopping me from mmaping an entire multi-terabyte nvme at the block level except for the part where I don't want to reimplement a filesystem from scratch in addition to needing to share it between lots of different programs.
Re: I/O is no longer the bottleneck? (2022)
#70Earlier quoted context omitted.
The limit is the number of outstanding cache line requests to the memory controller. CPUs have a fixed number of slots for this, around 10-12 usually. Intel calls them LFBs (Line Fill Buffers) and AMD MSHRs (Miss Status Holding Registers). When the slots are filled, the CPU can issue no more requests and has to wait for them to complete. Apple M chips (probably) have more slots and the memory is physically packaged t…
I assume these must be really expensive? Otherwise it seems like a great way to improve throughput on low concurrency tasks.
bus wires. you can route only so many of them on a motherboard.
it's why GPUs have their memory chips in a circle around the GPU chip.