It is possible to analyze the behavior of a program without access to the source code that makes it easy to recompile and distribute modified builds. I routinely open binary files in disassemblers to read through their behavior. Yes, it is possible to obfuscate the hell out of parts of binary code, and
that would certainly throw up a red flag for me, but concentrating on the source code seems to miss the point that source code can also be obfuscated, if the source code looks "too weird" we probably aren't going to trust it either, and it is also possible to hide a bunch of behaviors across wide areas of "open" "clear" code (so nothing looks "too weird" at any given place in the code).
To be very explicit about this, as I think this is a very subtle problem that people tend to totally misunderstand: if I wanted to distribute a chat program and have it be "evil" I would not distribute a binary with hidden behavior (if nothing else, when you find this code in my binary I'm pretty damn well screwed ;P): I'd instead distribute an open source program that involved a threaded work queue for handling multiple socket connections to peers and which had a few very subtle use-after-free race conditions that would only come up under nearly impossible timing scenarios that I knew how to trigger and exploit, giving me complete control of your client whenever I wanted.
These are the kinds of bugs people use to attack open source "secure" web browsers like Chrome year after year at Pwn2Own because people are simply bad at concurrency. In this sense, I'd thereby trust a closed source web browser that had no threads or which was implemented in a type-safe garbage collected language (executed on a simply-engineered runtime from someone separate that I trusted, which could also be closed source for all I care) a lot more than I'd trust Chrome. I'd even probably have an easier time understanding what it is doing disassembling it than reading Chrome's code. (To be clear, such a browser doesn't exist: probably you should use Chrome.)