Open-source hardware and software are analogous in this sense: pre-built hardware is like pre-built software. You can only truly trust it if you built it yourself. However, unlike pre-built software, often you can visually determine if high-order modifications have been made, e.g. additional components, re-routed traces, etc.
You can't really trust silicon, and this leads to a larger point about much of what has been called open-source hardware: many of the open-source boards out there which are really popular are little more than reference designs translated from a datasheet to a schematic capture program for a closed-source blob of silicon. However, this point doesn't escape many people in "the movement," the problem has just been that the technology that meets the users' expectation of performance and capability is out of their financial reach to duplicate. So, the users settle for "re-create" when they can, rather than "trust at all levels". In the majority of cases, absolute trust is not required, because a breech of that trust can result in practically nothing. ("Oh, this AVR chip is back-doored, allowing you to access its 32k of flash, but I haven't provided any means by which a 3rd party could access that via hardware.")