Earlier quoted context omitted.
I think this is a major challenge too, although it depends. Some models are easy to reason about, e.g. Bayesian graphical models, while black-box approaches like deep neural networks are not. One especially problematic issue is: if a model is too complex for humans to reason about, then a business could encode any kind of illicit behavior in the form of model parameters they like. Even if someone could prove that the…
> if a model is too complex for humans to reason about, then a business could encode any kind of illicit behavior in the form of model parameters they like If a model is too complex for humans to reason about, how would a business encode illicit behavior, unless the AI itself was running a significant portion of the business? Even in that case, someone or some group is responsible for setting the initial parameters o…
But that would be a large change in the way the legal system works. Fewer people than you'd expect understand that today, a large portion of crimes, if not most crimes, require both intentions and action for the legal system to judge guilt.
"A fundamental principle of Criminal Law is that a crime consists of both a mental and a physical element. Mens rea, a person's awareness of the fact that his or her conduct is criminal, is the mental element, and actus reus, the act itself, is the physical element."