I've been advocating for SQLite+NVMe for a while now. For me it is a new kind of pattern you can apply to get much further into trouble than usual. In some cases, you might actually make it out to the other side without needing to scale horizontally. Latency is king in all performance matters. Especially in those where items must be processed serially. Running SQLite on NVMe provides a latency advantage that no other…
I still measure 1-2ms of latency with an NVMe disk on my Desktop computer, doing fsync() on a file on a ext4 filesystem. Update: about 800us on a more modern system.
Enterprise NVMe can do fsync much faster than consumer hardware. This is because they can cheat and report successful fsync() before data actually had been flushed to flash. They have backup capacitors which allow them to flush caches in case of power loss, so no data loss.
Here PM983 doing `fio --name=fsync_test --ioengine=sync --rw=randwrite --bs=4k --size=1G --numjobs=1 --runtime=10s --time_based --fsync=1`
Jobs: 1 (f=1): [w(1)][100.0%][w=183MiB/s][w=46.7k IOPS][eta 00m:00s]
fsync_test: (groupid=0, jobs=1): err= 0: pid=11905: Fri Mar 14 13:34:34 2025
write: IOPS=39.1k, BW=153MiB/s (160MB/s)(1527MiB/10001msec); 0 zone resets
clat (nsec): min=1052, max=223288, avg=1606.69, stdev=2345.64
lat (nsec): min=1082, max=223458, avg=1653.08, stdev=2346.58
clat percentiles (nsec):
| 1.00th=[ 1128], 5.00th=[ 1176], 10.00th=[ 1240], 20.00th=[ 1320],
| 30.00th=[ 1448], 40.00th=[ 1496], 50.00th=[ 1528], 60.00th=[ 1576],
| 70.00th=[ 1640], 80.00th=[ 1720], 90.00th=[ 1816], 95.00th=[ 1960],
| 99.00th=[ 2576], 99.50th=[ 3376], 99.90th=[ 10816], 99.95th=[ 32640],
| 99.99th=[124416]
bw ( KiB/s): min=123168, max=190568, per=99.00%, avg=154788.63, stdev=19610.50, samples=19
iops : min=30792, max=47642, avg=38697.16, stdev=4902.62, samples=19
lat (usec) : 2=95.61%, 4=4.10%, 10=0.19%, 20=0.04%, 50=0.03%
lat (usec) : 100=0.02%, 250=0.01%
fsync/fdatasync/sync_file_range:
sync (usec): min=13, max=1238, avg=23.08, stdev= 9.27
sync percentiles (usec):
| 1.00th=[ 15], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17],
| 30.00th=[ 18], 40.00th=[ 25], 50.00th=[ 26], 60.00th=[ 26],
| 70.00th=[ 26], 80.00th=[ 26], 90.00th=[ 26], 95.00th=[ 27],
| 99.00th=[ 34], 99.50th=[ 79], 99.90th=[ 101], 99.95th=[ 126],
| 99.99th=[ 347]
Jobs: 1 (f=1): [w(1)][100.0%][w=22.9MiB/s][w=5859 IOPS][eta 00m:00s]
fsync_test: (groupid=0, jobs=1): err= 0: pid=23328: Fri Mar 14 13:35:04 2025
write: IOPS=5742, BW=22.4MiB/s (23.5MB/s)(224MiB/10001msec); 0 zone resets
clat (nsec): min=400, max=110253, avg=797.80, stdev=1244.19
lat (nsec): min=430, max=110273, avg=826.49, stdev=1248.86
clat percentiles (nsec):
| 1.00th=[ 502], 5.00th=[ 540], 10.00th=[ 572], 20.00th=[ 612],
| 30.00th=[ 644], 40.00th=[ 668], 50.00th=[ 708], 60.00th=[ 748],
| 70.00th=[ 804], 80.00th=[ 868], 90.00th=[ 1032], 95.00th=[ 1176],
| 99.00th=[ 1560], 99.50th=[ 2224], 99.90th=[ 8384], 99.95th=[23424],
| 99.99th=[66048]
bw ( KiB/s): min=19800, max=24080, per=100.00%, avg=23004.21, stdev=1039.13, s amples=19
iops : min= 4950, max= 6020, avg=5751.05, stdev=259.78, samples=19
lat (nsec) : 500=0.80%, 750=58.72%, 1000=29.04%
lat (usec) : 2=10.89%, 4=0.28%, 10=0.18%, 20=0.04%, 50=0.04%
lat (usec) : 100=0.01%, 250=0.01%
fsync/fdatasync/sync_file_range:
sync (usec): min=136, max=28040, avg=172.88, stdev=195.00
sync percentiles (usec):
| 1.00th=[ 145], 5.00th=[ 149], 10.00th=[ 151], 20.00th=[ 151],
| 30.00th=[ 159], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159],
| 70.00th=[ 159], 80.00th=[ 161], 90.00th=[ 198], 95.00th=[ 202],
| 99.00th=[ 396], 99.50th=[ 416], 99.90th=[ 594], 99.95th=[ 1467],
| 99.99th=[ 5145]