A 1.6x single core performance difference won't negate a 6x core count advantage in peak multi core performance. The problem above is not that the cores are slower, it's that Geekbench will literally not utilize the additional cores in the first place. This compounds with what you're saying - the few cores that do get used have high clocks on the low core count optimized part but lower clocks on the high core count optimized part.
Compare this to a multithreaded benchmark that does scale to all of the cores and you'll find the higher core count CPUs are able to push significantly higher scores despite the single thread difference e.g. https://www.cpubenchmark.net/high_end_cpus.html has them at 62,711 vs 117,317 in the opposite ranking direction. That should feel about right, otherwise AMD would only make the 16 core high frequency CPUs instead of 128 core low frequency monsters.
That's not to say the Geekbench score is bad or useless. It represents a specific type of workload... just not "peak multi core performance". It's more indicative of "mixed workload performance", where the extra 2x cores on the Ultra are more apparently going to be irrelevant.