Here's the counter-argument that I've seen in cryptography circles:
Dual EC, a PRNG built on an asymmetric crypto template, was kind of a ham fisted and obvious NOBUS back door. The math behind it made such a backdoor entirely plausible.
That's less obvious in other cases.
Take the NIST ECC curves. If they're backdoored it means the NSA knows something about ECC we don't know and haven't discovered in the 20+ years since those curves were developed. It also means the NSA was able to search all ECC curves to find vulnerable curves using 1990s technology. Multiple cryptographers have argued that if this is true we should really consider leaving ECC altogether. It means a significant proportion of ECC curves may be problematic. It means for all we know Curve25519 is a vulnerable curve given the fact that this hypothetical vulnerability is based on math we don't understand.
The same argument could apply to Speck:
https://en.wikipedia.org/wiki/Speck_(cipher)
Speck is incredibly simple with very few places a "mystery constant" or other back door could be hidden. If Speck is backdoored it means the NSA knows something about ARX constructions that we don't know, and we have no idea whether this mystery math also applies to ChaCha or Blake or any of the other popular ARX construction gaining so much usage right now. That means if we (hypothetically) knew for a fact that Speck was backdoored but not how it's backdoored it might make sense to move away from ARX ciphers entirely. It might mean many or all of them are not as secure as we think.