Earlier quoted context omitted.
Now many mini-PCs and desktops are able to read simultaneously from 1 PCIe 5.0 SSD and 1 PCIe 4.0 SSD. This can ensure a reading throughput around 20 GB/s, i.e. 20 times faster than on author's system. With only 1 PCIe 5.0 SSD, the reading throughput is still significantly more than 10 times faster than on author's system. So it is likely that inference speeds around 1 token/s are achievable on something like a NUC m…
I'm curious, where are you seeing M.2 2230 to M.2 2280 size NVME SSE that exceed 4.5 to 5GB/s sequential reads for large files such as a GGUF (likefrom an ordinary ext4fs file system with default options)? The PCI-E 4.0 or 5.0 bus they're attached to might be capable of greater speeds, but the bottleneck is the flash and the flash controller. The benchmarks I'm seeing for many of them don't really make me think that…
If there were such a slow SSD, it would not make sense to buy it instead of a cheaper PCIe 4.0 SSD.
For PCIe 4.0 SSDs, I have seen a very large number of benchmarks where the SSDs achieved read speeds close to the theoretical limit, i.e. around 7 GB/s.
Searching now randomly for recent SSD reviews, I find many reviews for "SanDisk WD_BLACK SN8100", which achieve between 13 GB/s and 15 GB/s reading speed, which is better than most other consumer PCIe 5.0 SSDs.
Of course, if you write a very simple program that invokes something like "fread" or "read" in a loop, you will not reach such speeds. Achieving a SSD throughput close to the limit requires a more complex program that can ensure that the SSD controller is permanently busy with pipelined read commands.