You're right that transparency alone is insufficient. It's
a dependency of trust, not
the dependency. However, it's still absolutely necessary.
Sure, you can create your own variety of transparency by reverse-engineering something to figure out how it works (or hiring someone you trust to do so on your behalf), but if you have to resort to that, it's because you already (rightfully) don't trust the thing.
"For most people 'seeing how it works' is utterly meaningless by itself"
Well yeah, most people aren't going to be able to make sense of even a perfectly-transparent system of sufficient complexity. That's why they hire auditors to do it for them. It ain't about the user actually doing it; it's about the user being able to do it should the user have the time and technical competence to do so.
Taken another way: if you don't even have the ability to inspect a transparent thing, then what makes you think you can meaningfully inspect an opaque thing?
"closed software can perfectly well be probed anyway (as this very software and every single vulnerability ever found in proprietary software demonstrates!)"
Correction/clarification: closed source software can occasionally - and with a bit of luck and a heck of a lot more skill and time and effort - be probed for specific reasons to be untrustworthy. You're unlikely to ever reach the point where you have full understanding of the system (and if you do reach that point, then it's pretty transparent - to you at least - and therefore possible to be trustworthy, at least until the next update). Without transparency, there's always the possibility of something nasty (a fatal bug, or an overreaching telemetry "feature", or somesuch) lurking in the places yet to be probed. Sure, it's possible for those to hide in transparent (e.g. FOSS) programs, too (though I'd argue they're not being particularly transparent in those cases), but it's much easier to find that nastiness in a transparent program than an opaque program.