Downloading BINARY software safely is impossible indeed.
Downloading SOURCE software poses no security risk.
As long as only human read those sources, to check for this software behavior.
WHEN the sources have been audited, and cleared for any security or other bugs, you may consider compiling them.
That's where the real crux of the problem lies:
https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomp...
http://c2.com/cgi/wiki?TheKenThompsonHack
You cannot trust software more than you trust your compiler or your interpreter or in general, your processor.
Since big commercial OS providers are KNOWN to be collaborating with the NSA to but backdoors and spy on you in the first place, you cannot have any trust there anyways. Why worry about random binary software on the web, when your base OS is already compromized?
Otherwise, you may choose to use hardware that you trust, preferably, that you have built yourself. Alternatively, you may build a computer using triplicate parts from different sources (eg. put an Intel, an Elbrus-4C and a Godsoon, and compare bus traffic. As soon as a difference is detected, raise an exception and abort the process), similarly, put three different network controllers, made in three different countries, and compare output packets. As soon as there's a difference, drop the packets and signal the process. etc.
Once you've got a trustable hardware base, you can build a trustable software layer, using only sources and bootstrapping binaries by hand (with people you trust, preferably yourself), and as indicated in Countering Trusting Trust through Diverse Double-Compiling, http://www.acsa-admin.org/2005/abstracts/47.html
you may implement also at the software level, a similar kind of redundancy that allows you to increase the trust you may have in your bootstrapping chain.
But there is no way out, you have to start from the sources, read them and consider only source code when exchanging software.
Notice that debian is a binary distribution. Gentoo is a source distribution. Gentoo is more trustworthy (but you need to be careful how you bootstrap its installation, which is not easy).
This is why licenses such as the AGPL, and also works to make software higher level and shorter (understandable and therefore trustable by end users) such as Alan Kay's, are important.