Towards a conversational agent that can chat about anything
ai.googleblog.com
Towards a conversational agent that can chat about anything
1–10 of 108 posts
Re: Towards a conversational agent that can chat about anything
#2One key point is that they are using a metric called perplexity, "the uncertainty of predicting the next token."
From the post:
> Surprisingly, in our work, we discover that perplexity, an automatic metric that is readily available to any neural seq2seq model, exhibits a strong correlation with human evaluation, such as the SSA value. Perplexity measures the uncertainty of a language model. The lower the perplexity, the more confident the model is in generating the next token (character, subword, or word).
Re: Towards a conversational agent that can chat about anything
#3One key point is that they are using a metric called perplexity, "the uncertainty of predicting the next token." From the post: > Surprisingly, in our work, we discover that perplexity, an automatic metric that is readily available to any neural seq2seq model, exhibits a strong correlation with human evaluation, such as the SSA value. Perplexity measures the uncertainty of a language model. The lower the perplexity,…
Is perplexity related to the Shannon entropy?
Seems like this might make sense, with the exception of social niceties the point of human communication is to transfer information that was hitherto unknown.
Re: Towards a conversational agent that can chat about anything
#4[deleted]
Re: Towards a conversational agent that can chat about anything
#5Cool, how do I talk to it?
I see these big announcements with nothing behind them but a white paper and no way to reproduce. It's not a fact if there is no falsifiability.
Re: Towards a conversational agent that can chat about anything
#6One key point is that they are using a metric called perplexity, "the uncertainty of predicting the next token." From the post: > Surprisingly, in our work, we discover that perplexity, an automatic metric that is readily available to any neural seq2seq model, exhibits a strong correlation with human evaluation, such as the SSA value. Perplexity measures the uncertainty of a language model. The lower the perplexity,…
Is perplexity related to the Shannon entropy? Seems like this might make sense, with the exception of social niceties the point of human communication is to transfer information that was hitherto unknown.
Yea, perplexity is ~exp(entropy)
Re: Towards a conversational agent that can chat about anything
#7I wonder what they'd say now about Terry Winograd works on SHRDLU and the accepted conclusion...
Re: Towards a conversational agent that can chat about anything
#8One key point is that they are using a metric called perplexity, "the uncertainty of predicting the next token." From the post: > Surprisingly, in our work, we discover that perplexity, an automatic metric that is readily available to any neural seq2seq model, exhibits a strong correlation with human evaluation, such as the SSA value. Perplexity measures the uncertainty of a language model. The lower the perplexity,…
Actually, the key point is that they aren't using perplexity. They built a chatbot that aims to maximize "SSA" (Sensibleness and Specificity Average), and then found that SSA is correlated with perplexity.
Re: Towards a conversational agent that can chat about anything
#9One key point is that they are using a metric called perplexity, "the uncertainty of predicting the next token." From the post: > Surprisingly, in our work, we discover that perplexity, an automatic metric that is readily available to any neural seq2seq model, exhibits a strong correlation with human evaluation, such as the SSA value. Perplexity measures the uncertainty of a language model. The lower the perplexity,…
Actually, the key point is that they aren't using perplexity. They built a chatbot that aims to maximize "SSA" (Sensibleness and Specificity Average), and then found that SSA is correlated with perplexity.
"The training objective is to minimize perplexity, the uncertainty of predicting the next toke""
SSA is used as an evaluation metric. It is generated by human raters, and as such is not differentiable, making I hard to directly optimize for with current methods.
Re: Towards a conversational agent that can chat about anything
#10In 2015 Google claimed to have created a chat bot that can do common-sense reasoning and do tech support just by reading a sufficiently large dataset: