The numbers for GPU databases look “good” because you can get pretty high cross-sectional bandwidth to a reasonably large memory from 8 GPUs in one box, and advertise blazing speed from that. But it’s just a trick.
The only thing that matters for them here is the aggregate, cross-sectional bandwidth to your data’s working set in memory. For databases, especially for the approach that many GPU databases take (light on indexes since GPUs aren’t great at data-dependent memory movement or pointer chasing, just brute-force scan much of the data), the working set size is something that will only fit in main memory.
Instead of using 8 GPUs with a peak global, cross-sectional memory b/w of 8 * 320 = 2560 GB/sec and brute-force scans, you can parallelize across ~40 CPU nodes each with ~60 GB/sec b/w to main memory. The cross-sectional bandwidth will be about the same, and the cost to split and join the results of the query is likely small in comparison to actually doing the work, assuming the intermediate results are reasonably small. You can use a broadcast and reduction tree; the added latency of the broadcast and reduction tree's depth likely won't add much, since there isn’t much data to broadcast in a query, and the data returned by each machine for the reduction is hopefully (!) tiny in proportion to the actual data scanned.
If you want to consider indices on data, then maybe the heads of that can remain resident in a CPU’s cache, and will make the individual CPU scans even faster. The GPU caches are tiny and mainly serve to patch up strided loads and other bad uses of memory.
Whether or not it’s worth it one way or another depends upon how large your database size is, the relative cost of GPUs versus CPU nodes to get the memory you want and the cross-sectional b/w you need, perf/W and other issues.
You’re probably nowhere near close to arithmetic throughput bounds on GPUs or CPUs since these workloads have very low op / byte loaded ratios compared to typical HPC workloads, so that aspect of GPUs doesn’t matter. If you’re doing expensive pre- or post-processing on GPUs as well, then that may push the balance more towards GPUs.