It's always the cache. Caches are evil! (security) No! Caches are amazing (speed). Fun fact: I never understood caches that well, until I learned about geo-caching and learned what caches were used for back in the day of yore!
Inclusive caches are a huge culprit security wise. If you can perform any variation on the clflush instruction by building eviction sets [1] then you are good to go to perform somekind of operation you're not allowed to do.
I feel that processor vendors are using their undocumented security by obscurity too much. Though I don't know if they do it intentionally or that these things happen merely via company processes and it's just tough to build a reliable safe processor.
Safe cache design simply ain't easy, not even in 2018. The hype is attacking inclusive caches in most cache attacks, but even when you build other things, hardware hackers will basically look at the computer schematics of everything again and they'll ask themselves how they're able to get that nice primitive back of inclusive caches [2].
[1] Algorithmically defined memory addresses that will kick your target cacheline out by flooding the same space.
[2] The primitive is: an inclusive cache by current design standards has a shared last level cache. Kick target cachelines out there and you're flushing a part of private caches from other processor cores as well.