Torus development has been stopped at CoreOS

Why not? Block storage is a weird beast, targeting legacy apps unable to run in multiple data centers. The same apps are also likely to be willing to lose some of the most recent data in case of a data center outage and trade this for performance, so the barrier is already low. The storage might only need consensus somewhere close by, where latency is very good and the network capacity is huge. Nodes in other data centers could receive data asynchronously (otherwise write latency is going to render the whole thing useless anyway). The question is whether there really are billions to be paid for something that ultimately cannot do well compared to proper distributed solutions.

I worked for several years in VFX/HPC. 30k+ cpus and 15pbs of storage.

Firstly with storage its very rare that people want actual block storage (unless you are hosting VMs, but thats so 2007.....) Yes, I know, openstack, but that's just fucking horrific, seriously just use netboot and be done with it. I've seen people do it inside new clustereing systems, but its really not fun to do, especially if you consumer is prone to disappearing without warning. (FSCK is a terrible mechanism for fast recovery)

Most apps, unless they have bought into the "shove everything over HTTP and pay the penalty", want a posix file system to store anything of importance. (yes, yes database, but where is that writing the data to?)

Now, there are three ways you can do this:

o use a clustered file system

o Use NFS (with or without a clustered filesystem underneath)

o Fuck about with iscsi/SAS/FC and dynamically map block dynamically.

Using a clustered filesystem spread over many clients is begging for trouble, mainly because one client can fuck it up for everyone. Some FSs are dynamic and sexy, but they have a habit of fucking up in new and interesting ways that even the authors can't figure out.

The common ground is having storage nodes attached directly to a pack of big fat disks(for streaming IO) or NVME/SSDs for random IO. They then serve out NFS traffic. Now, you can either have a clustered file system underneth, or not. (Having stand alone servers can be advantageous, if you can map your filesystem out hierarchically)

Now, unless you have a Storage area network, then the last option is just begging for shit performance. You really don't want IO traffic fighting with network traffic. However, if you want raw throughput, this is the way to go, but be warned, you won't get any friendly help if you accidentally disconnect a disk.

Basically, kubernetes/HPC and storage is a solved problem ducks no really, just map in NFS shares and be done with it. If its exotic, its probably going to fail hard, and in ungoogleable ways. More importantly only a few people are going to be able to help, and they may or may not still employed at your company.