The main question is "does it matter?" The answer depends case by case.
For example, if I am the PM for TurboTax Premier desktop software, I know that it's latent - but it probably doesn't matter in a way that deserves prioritization. Sure, my clients would like for the forms to load faster, but they care infinitely more about the scope of tax situations we cover, and the correctness of calculation. A person uses my product a few hours per year, so even if there's an accumulated minute of form refresh latency, it's just not that impactful. What that means is - if I have an "extra developer", I am going to direct them towards product scope and correctness, not latency.
On the other hand, something like VSCODE (an example the article mentions), speed is part of the value proposition. As a developer, I need to quickly change text, quickly look up a reference, quickly switch files, etc. If things are sluggish, it isn't just frustrating but limits my ability to do development work. So a user of a slow IDE would be very motivated to seek alternatives, since they basically live in the IDE. As a PM, I would absolutely invest in performance if it was making my product actually less useful.
It's like anything else, investment has a tradeoff. Chances are, whatever car you drive could have a faster 0-60 speed and in isolation, that would be great. But are you willing to pay 10x for the car? Are you willing to give up the seats and the trunk to make it happen? So your car may not have the best 0-60, but if it's an affordable family minivan that's probably the right call.