Earlier quoted context omitted.
At the risk of topic drift, I'm curious where the problems are with being parallel and POSIX-compliant. I'm at a point where I think I need to consider a parallel file system and I don't have much experience with them, so I'm not aware of the issues.
Here’s something I wrote in this thread: https://news.ycombinator.com/item?id=15222470 At least on the real big systems, most of our work was with a small number of research groups basically doing the same workflow: start your job, read in some data, crunch, every 15 minutes or something slam the entire contents of system memory (100s of TB) out to spinning disk, crunch, slam, your job gets killed when your time slic…
You are of course very correct that if you have a massive number of processes using files for IPC, things will fail in very strange ways.
One of the very positive things that AWS/GCE provide is simple scalable primitives, with obvious metrics to measure limits. For example SQS is a brilliant primitive for many-many message based processing. It avoids the horror of running a HA kafka queue(or worse).