Higher sample rates are lower latency for the same block size and resampling is not "free" (pick 2: performance, aliasing, latency) so there can be advantages to working with audio archived at higher sample rates.
But all the advantages come down to professional or editing use cases. There's next to zero advantage to using it as a storage format for listening. Just like 24 bit audio (do you have an amp with 96dB SNR?).
Just personally, I have seen little evidence (personally, professionally, or academically) that there is any advantage for lossless audio for consumer applications. For professional applications there are plenty, and it's endlessly tiring to convince people that "no, actually I need 96kHz for my use case."
Where the audiophiles have _some_ argument here is the design of reconstruction filters which I've heard alleged can perform better in the audible frequency range if the stop band is outside of it. But I have never personally tested this, nor cared enough to. But the theory is sound.
Whether or not it's perceptible depends on what you're measuring, though. In theory, there should be perceptual differences in sound localization if your DAC's reconstruction filter is at 24kHz vs 48kHz since it will change the group delay in a critical frequency region, where you'll get sound at >~2kHz arriving later at the lower sample rate. I think it would be extremely hard to test this though, because humans are really shitty at sound localization to begin with, and practically speaking most recorded material is processed to shit in that frequency range to intentionally decorrelate the channels for the perception of "width."