Making better decisions with the Brier score
datarecipes.io
Making better decisions with the Brier score
1–10 of 19 posts
Re: Making better decisions with the Brier score
#2[deleted]
Re: Making better decisions with the Brier score
#3In my experience I’ve never found an instance where you would use Brier scores over cross entropy/Bernoulli/Binomial log likelihoods. Does anybody know a concrete example when you would prefer Brier??
Re: Making better decisions with the Brier score
#4What's the difference with MSE loss?
Re: Making better decisions with the Brier score
#5For folks who want to try the kind of forecasting being discussed here, Metaculus is a pretty great community: https://www.metaculus.com/
Their FAQ has a great explanation of how they 'score' user forecasts --- including a summary of Brier scores for binary yes/no questions, and the log score used for both binary and continuous questions: https://www.metaculus.com/help/faq/#howscore
Re: Making better decisions with the Brier score
#6In my experience I’ve never found an instance where you would use Brier scores over cross entropy/Bernoulli/Binomial log likelihoods. Does anybody know a concrete example when you would prefer Brier??
The article mentions Brier score is just mean squared error, so it's connected to binomial through that (e.g. where correct prediction is 1, incorrect is 0, it is the mean of the binomial).
Re: Making better decisions with the Brier score
#7What's the difference with MSE loss?
Brier score is MSE applied to predicted probabilities. One is a loss function for "regression" type problems, the other is a proper scoring rule.
Re: Making better decisions with the Brier score
#8What's the difference with MSE loss?
No difference this is exactly same as brier score. MSE is the KL divergence between ground truth and true prediction, assuming a gaussian error distribution. We use MSE as loss because we try to minimize KL divergence (again assuming gaussian error distribution). The article is very shallow, I am surprise it comes on HN front page.
Re: Making better decisions with the Brier score
#9Anyone else reading Bier score?
Re: Making better decisions with the Brier score
#10In my experience I’ve never found an instance where you would use Brier scores over cross entropy/Bernoulli/Binomial log likelihoods. Does anybody know a concrete example when you would prefer Brier??
It's appropriate when the cost of false-positive and false-negative errors are the same. Which isn't common in the real world.