An A64FX CPU has 48 cores. 1 CPU/node means 96 cores on 2 nodes for $40k and ~6TFLOPS (3 per). Meanwhile the 128 thread Threadripper 3990X costs $3.6k, and benchs put it at 1.5TFLOPS in Linpack, which is less but not 10x less, and I don't know what the benchmark basis for the A64FX value is, so I suspect it's closer, especially since it says theoretical peak performance. ... Am I missing something?
Your 3990X system has dramatically less memory bandwidth with 4 64 bit memory channels good for a peak bandwidth of around 100GB/sec, which is 10% of the A64FX.
Your $3,500 price likely doesn't include an infiniband card, or a port on an IB leaf switch, or the port on spine IB switch, and you are getting quite a bit less memory bandwidth and floating point than the A64FX system. As a result you end up buying more threadripper nodes, paying for more power, more rack space, more cooling, more IB spine switches, more IB cables, more IB leaf switches, and potentially even a larger building ... just to hit the same performance.
Generally to get close to the memory bandwidth or flops with an x86-64 system you end up using an accelerator, like say the Nvidia A100. Problem is that the nvidia GPUs can only run CUDA aware programs, which rules out a significant chunk of HPC workloads.
The attraction of the A64FX is that it's efficient, has impressive flops per watt (which means cheaper cooling, racks, buildings), impressive memory bandwidth, and will run any python, perl, fortran, C, C++, go, java, code you throw at it.
For large clusters this isn't just a nice to have, it's a huge driver of real world performance / price. So much so that the top 4 supercomputers in the world are not x86-64. Even #5 uses a Intel Xeon to run the OS and do I/O and uses an accelerator (Matrix-2000) to do the heavy lifting. #6 and #7 are similar, but using a Nvidia cards. #8 is the first pure x86-64 cluster and is pretty small in comparison, 18 times smaller than #1.
Even #8 (Frontera) being 18 times smaller, makes it MUCH easier to scale codes that run across the entire cluster. Despite that huge advantage, Frontera scaled lipack to the entire cluster with 60% efficiency. The #1 Fugaku using the A64FX scaled at 80% efficiency, that's a pretty large real world advantage.
Keep in mind that Linpack is a pretty limited benchmark, heavily optimized, and not particularly memory or network intensive. Many real world codes will more heavily exercise the system and would likely show a larger performance differential than linpack.
Imagine you have 20,000 watts a rack and 100 racks. Sure you could use cheap nodes, but your total performance would be less... and that's before you throw in problems like scaling to 100 racks at 60% efficiency instead of 80%. When asking for your budget it will be that much harder if you only perform well on CUDA codes. Not that there's not room for cheaper node clusters out there, but non-x86-64 systems do seem to have a significant advantage on larger clusters.
Also general HPC devel/test systems often come with substantial support to enable users to tune their codes to a new platform. I wouldn't assume that if you want 100 racks of them that you'd pay $20k each, in fact the hardware you'd likely use isn't even the same hardware. The A64FX nodes used in clusters like Fugaku have higher density nodes that depend on water cooling, have more cores, and have a much better interconnect.