Earlier quoted context omitted.
Not sure where this is stored on Mac, but on Windows, clear the fontconfig directory in the AppData folder.
That's not really likely to help. The main issue appears to be that since fontconfig (a lib used by GIMP) is only "native" to Linux, for it to know about the TTF files, it has to scan them all and build it's cache at startup. For people with hundreds of fonts installed it can be a real PITA. I avoid it by being careful to not exit GIMP entirely (simply closing all the editing windows) until I am sure I won't need it…
Adding a few new fonts for a specific project does not cause long delays on any platform.
A compounding issue was that file locking issues prevented successful cache updates on the Windows platforms until recent versions of fontconfig, this lead to a long startup time on every start.