We are playing with a slippery slope! A backdoor is a backdoor. Honestly it is getting to the point where open source is the only way to go - imo. I'd like to be able to perform SAST scans and code review on all software that protects my enclaves.
Most open source crypto code just does what NIST and DJB say to do. There's no magic imparted by it being FOSS.
Plus it's also extremely difficult to make underhanded changes to magic numbers if the source code is public.