Show HN: Ecco – See what your NLP language model is “thinking”

Exactly and maybe we can "lobotomize" sections of the models that replicate unwanted bias in the training data.

Yikes, you are right… I just linked a private repo. '^^ I have poked the rest of the group and it seems that at least a tweet was made [1] – but not much else remains. Describing it from memory, we ran ELMo and BERT on Wikipedia and then allowed similarity search between a query and showed heat maps to a matched context. Nothing particularly deep compared to yours that go into the transformer “machinery”, but I think it captures very well how most Question Answering models still operate: Embed query and contexts in a high-dimensional space, compare, find semantically plausible span, and done!

[1]: https://twitter.com/Johannes_Welbl/status/106530965474036121...

Work and articles like yours has truly had an impact on me, even though they are largely qualitative. We always say “Turing complete” this and “Turing complete” that, but theoretical statements such as this have little practical utility to me as we all know that what can be learnt and what is learnt are two very different things. For example, “Visualizing and Understanding Recurrent Networks” by Karpathy et al. (2015) [2] that you list as inspiration blew my mind in terms of for example neurons that monotonically decrease from the sentence start. I remember Karpathy giving a talk on it in London and what struck me was how he simply had gone to manually inspect the neurons manually (heresy!) as there were only a few thousand of them any way. That playfulness, truly admirable.

[2]: https://arxiv.org/abs/1506.02078

Another anecdote, now from “Attention Is All You Need” by Vaswani et al. (2017) [3] where I was far from sold on Transformers as a model until Uszkoreit gave a talk at an invitation-only summit where he showed those cherry-picked attention heads that “flipped” based on whether an object was animate or not. I approached him after the talk and asked why it was not in the paper as it was awesome! Maybe I am biased because I give a large role to intuition in science, but analysis such as this is far more valuable to me as a researcher than yet another point of BLEU or a 10th dataset. Again, my bias, but I feel that there is a need for new ways of thinking in terms of both “hard” empiricism and “soft” analysis in machine learning as we seemingly are now having to mature given the attention we are receiving.

[3]: https://arxiv.org/abs/1706.03762

Apologies if I am rambling, it is midnight now and I barely slept last night.

Interesting. The non-negative matrix factorization on the first level kinda highlight some semantic groupings : paragraph, verbs, auxiliaries, commas, pronouns, nominal propositions.

I tried to look at higher level layers, and the grouping were indeed of higher level : for example at level 4 there was a grouping which highlighted for any punctuation (and not just comma). The grouping were also qualifying more : for example ("would deliberately" whereas at lower level it was just would).

But it's not as clear as I had hoped it would be. I hoped it would somehow highlight grouping of higher and higher size, that could nicely map to the equivalent of a parse-tree.

The problem I have with this kind of visualizations, is that they often require interpretation. Also, they don't tell me if the structure was really present by the neural network but was just not apparent because the prism of the Non-negative Matrix Factorization hid it.

For my own networks, instead of visualizing, I like to quantify things a little more. I give the neural network some additional layers, and I try to make the neural network produce the visualization directly. I give it some examples of what I'd like the visualization to look like, and jointly train/fine-tune the neural network so that it solve simultaneously his original task, and the production of the visualization which is then easier to inspect.

Depending on how many additional layers I had to add, and depending on where they were added, and depending on how accurate (measured by a Loss Function!) the network prediction are, I can better infer how it's working internally, and whether or not the network is really doing the work or if it is taking some mental shortcuts.

For example in my Colorify [1] browser extension, which aims to reduce the cognitive load of reading, I use neural networks to predict simultaneously visualizations of sentence-grouping, linguistic features, and even the parse-tree.

[1] https://addons.mozilla.org/en-US/firefox/addon/colorify/

One small step on the path towards solid-state intelligence

This is fantastic, I used your earlier transformers article to first get a real grasp on the architecture. I hope you expand this to accommodate other modes of attention outside of transformers paradigm as well!