In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all…
Maybe, but one can find out. You can follow the chain of calls of any software, and they have access to everything, as well as the whole history. It's like when in an interview Buffets says it's very hard to know where all the money is in the financial system. Sure, it's a complex system, but it's more of a matter of incentive.
In the same way you can't enumerate the atoms or molecules in a human body. sure, you can come up with good estimates and summaries that are useful, but not the actual full and truthful state of affairs.
In addition, you are a self-referential participant in the system: so acting in an attempt to observe such likely immediately makes your information at least partially incorrect.
Additionally, if we can't get full/transparency or introspection into machine learning models or neural networks (and we can't because i don't believe we even possess the theoretical knowledge of how to do so), it should also be wilfully apparent that we can't actually know the true lineage or dependencies of systems that implement such in production.
Not to mention the difficulties as the inputs/outputs of such pass in and out of the notional entity or system being observed.