Say what you will about Gentoo as a concept, it was a lot of fun when I was 17 learning more about software packaging, distributed compilation, and the ins and outs of compile-time optimization, not to mention Linux kernel optimization. And their community had pretty nice docs from what I recall. I think a few of my patches are still knocking around in some releases. I finally realized all the tweaking and optimizati…
Then AMD Opteron came out and got everyone on a new baseline: if you compiled for amd64 then that meant a specific CPU again, there were no newer instructions yet. Now AMD64 has several levels[1] which can start to matter for certain kinds of operations, such as SIMD. A more mundane one I often run into is that v2 added popcnt which is great for things like bitsets in isolation, but in overall program performance I measure almost no difference on any of my projects between v1 and v3.
When it comes to games, it's more than likely your games were binary-only and already compiled for the lowest common denominator and maybe used runtime feature selection, and even then, they were probably GPU-bound anyway.
[1] https://en.wikipedia.org/wiki/X86-64#Microarchitecture_level...