You're right, though there are some things that are about as hard: compilers, language design, machine learning, databases with strong ACID guarantees, etc.
The problem I have with the "crypto should be a forbidden zone" line of reasoning is that the real world evidence shows that the old battle tested systems manifest flaws at least as often as competently designed newer systems do. Crypto, it turns out, is so hard that the probability of lurking issues with mature systems approaches or exceeds the probability of mistakes in new ones.
When I say competently designed, I mean a newer system that passes the sniff tests of experienced crypto engineers. An incomplete list: they're using a cipher that's been peer reviewed and is considered strong by modern standards, they're using that cipher correctly, they're authenticating before doing anything, they are using an IV (if needed), they are not sending anything secret in the clear, they're not branching on secret data, etc.
It's also important to refrain from criticizing people for claims they are not making. As far as I know, Weave is not claiming to implement the entire feature set of IPSec. They're just claiming to offer basic but strong crypto and authentication. If you want more, you are likely using other algorithms like SSL and SSH over the overlay network.
Yes, that comes with a performance penalty, but it's also defense in depth. It's better to trust multiple layers of crypto with independent implementations at each layer so that a compromise of one does not destroy your entire security posture.
It all comes down to the question of how paranoid you are. No encryption will give you the best performance, but no security. If you want maximums security you can run SSL over IPSec over Weave with different sets of keys and different ciphers at each level. Bonus points for generating those keys on different air-gapped hardware, etc.