What I don't understand is why the GDPR doesn't enforce the following:
1: Clear and concise information about the action being taken, and clear identification this was done using automation and did not involve any human oversight.
2: A process the user can invoke to request human intervention, and a confirmation email that a human will review the decision that was made within 30 days
3: Public statistics and transparency - any decision that was made that did not involve a human must be published, with stats on % of decisions made, number of cases flagged to human reviewers, and the success/failure rates (for example, number of cases resulting in an overturned AI decision by a human).
This could also be beneficial in other sectors too, like automated credit decisions and insurance policies, to publish statistics and data to afford transparency and identify possible biases. It should also be a requirement in law to preserve any code or algorithms should they need to be audited, including an AI system to be preserved "in time", so that the 2017 version can be audited in 2020 if an investigation is launched for example.
Right now, it's a complete free for all, too many edge cases and ways to game the system if you can figure its loopholes, and no requirement in law to provide a fair basis for users to appeal to a human without causing a PR shitstorm.
This early adoption of AI is quite bad, and I suspect we'll see such developments in the long term future as it matures.
[1]: https://ico.org.uk/for-organisations/guide-to-data-protectio...