A few years ago, I was working in a company that was trying to build an innovative NLP system, or in more honest words, to do a chatbot that doesn’t suck. Spoiler alert: we failed.
There were a lot things wrong in how this company was run and the product we were doing, but I won’t go into details except to say that there were a lot of intelligent people forced to do silly things by a clueless micromanaging boss.
Anyway, one of the problems with chatbots is the one of prior knowledge. Chatbots and other NLP solutions don’t simply need to be able to understand and produce conversation, they need to have something to talk about, a model of the world, some basic facts, and it turns out it is very complicated to build in general.
So our boss decided that one way to fake it was to use one of those free corpora of public-domain English literature. Let’s just make our system “read” a lot of text and in some way it will gain prior knowledge that way. So if it reads “the Sun was high in the sky”, it would understand that the Sun is something that has a position and that one of the possible position is “high in the sky”. So if someone ever asks the chatbot “where can the Sun be?” it could answer “The Sun can be high in the sky”. It was all pattern matching, nothing very smart about it, just something to fake some parts of the conversation and avoid having too many “I don’t know”.
Of course, it was literature, including fiction. So caterpillars could smoke hookahs, but that was considered an acceptable risk, it was better to have something wrong than an admission of ignorance. In some way don’t humans also repeat stuff without understanding them?
It kinda worked. If you asked “What do people eat?” it would answer “People eat potatoes, mushrooms and tires” or something like that. It was not very smart but somewhere in the literature the pattern “ eats ” existed and it was parroting it. If you asked “What do children eat?” it would answer “Children eat carrots, rocks and cupcakes”. It was a bit silly but nice.
But then we asked “Who eat children?” and the answer was, I shit you not, “Black people eat children, while howling to the moon and covering their naked body with feces”.
Except it didn’t actually say “Black people”, it used the other term, the one which is much worse.
The sudden realization that we have created an AI but an incredibly racist one did not make us abandon the approach. We just found the guilty piece of text in the corpus and expunged it. Then it just said “Companies eat children”. Depending on your politics you can consider that better.
To be fair, it was not really Machine Learning but the story shows what can happen if you don’t control your input, either because it comes from the evil internet or because it is a large dataset that it is too big to reasonably sanitize and was not built for this purpose.