I also first thought this was some smug condescending point of view, but the more I learned about it the more I realize this shit is hard.
And it's not hard because of the mathematical theory behind it. It's because there are so many subtleties in cryptography that in order to call yourself a "cryptographer", you not only need to have understood all the mathematical theory but also have thought about and understand all the subtle edge cases. And once you do get to that point, you become super careful about making certain claims.
Unlike engineering problems where a small error can be fixed somehow through monkey patching, a subtle loophole brings down the usefulness of the entire cryptographic algorithm. So no matter how powerful your next big cryptography algorithm is, if someone finds a very very small loop hole, it's fucked, and the entire algorithm is unusable.
This is so important that the WebCrypto API even named their main API endpoint as "crypto.subtle" (instead of just using the "crypto" prefix) to warn people.
So TLDR: it's not because people don't trust the expertise of a non-cryptographer, but because people trust that an actual "professional cryptographer" is humble enough to not make claims unless they can prove it.
My impression with IOTA is that it makes a lot of mind-bending claims which no one can easily prove or disprove. And also they don't use the proven-by-time cryptographic algorithm. This makes it impossible for anyone to make a trustworthy peer review. (It doesn't help that the founder acts like an asshole so nobody wants to waste their time doing a peer review on it anyway)