Earlier quoted context omitted.
I think mathematics and physics sets the bar higher then subjective relevance when formulating descriptions. If you did manage to create a description that would accurately predict every response of a person then you would have a hard time (if not impossible) discerning the difference between the two agents.
But one of them wouldn't be something anyone would call an 'agent.' One would be a description of an agent, or a recipe for producing one—but at the end of the day, one is meat and the other is a set of symbols.
You guys basically complemented each-other, and came back to the beauty of the question.
I'd say what's more pressing is, does the information match the implementation. That is, is there any glimpse of external forces (that implement our universe) inside of it.
See https://en.wikipedia.org/wiki/Brain_in_a_vat
If our whole reality is more of a rouge, rather than an accident, then it's possible our reality has almost nothing to do with the parent-verse. And well, everything we seem to value as good models for the true implementation, are well, garbage.