That's a very good point. There's precedent for it too.
Humans themselves are black boxes too, and yet almost everyone grows to have a good handle on limitation of individuals, both in general and specific to contexts and individuals one encounters. It's not just peer-level understanding - parents quickly learn to understand the limitations and failure modes of newborns, and maintain that understanding as their children grow.
Now human-human understanding is cheating, since we all share the same hardware architecture. But we have no problem with getting a grasp on animal behavior and limits either. Really all it takes is enough personal observation and interaction. It's like we're wired for "getting a feel" for complex systems over time.