If you decide to go down this road just use Gentoo.
I haven't kept up with Gentoo. How long does it take these days to bootstrap a functional desktop from source? I remember when it was my main driver back in early '00s, running on a horribly underpowered PII at 266MHz. It would often be compiling 24/7 to keep up with the updates.
My first install was 2002 and it took my a good 24 hours to get X to boot.
It did catch a faulty memory DIM later on because the linking kept failing intermittently.