Bayesian Neural Networks
cs.toronto.edu
Bayesian Neural Networks
1–10 of 60 posts
Re: Bayesian Neural Networks
#2I like Bayesian inference for few-parameter models where I have solid grounds for choosing my priors. For neural networks, I like to ask people "what's your prior for ReLU versus LeakyReLU versus sigmoid?" and I've never gotten a convincing answer.
Re: Bayesian Neural Networks
#3I like Bayes, but I thought the "surprising" result is that double descent is supposed to prevent nns from overfitting?
Re: Bayesian Neural Networks
#4I like Bayesian inference for few-parameter models where I have solid grounds for choosing my priors. For neural networks, I like to ask people "what's your prior for ReLU versus LeakyReLU versus sigmoid?" and I've never gotten a convincing answer.
I'm sure there is a way of interpreting a relu as a sparsity prior on the layer.
Re: Bayesian Neural Networks
#5Bayesian Neural Networks just seem like a failed approach, unfortunately.
For one, Bayesian inference and UQ fundamentally depends on the choice of the prior, but this is rarely discussed in the Bayesian NN literature and practice, and is further compounded by how fundamentally hard to interpret and choose these priors are (what is the intuition behind a NN's parameters?). Add to that the fact that the Bayesian inference is very much approximate, and you should see the trouble.
If you want UQ, 'frequentist nonparametric' approaches like Conformal Prediction and Calibration/Multi-Calibration methods seem to work quite well (especilly when combined with the standard ML machinery of taking a log-likelihood as your loss), and do not suffer from any of the issues above while also giving you formal guarantees of correctness. They are a strict improvement over Bayesian NNs, IMO.
Re: Bayesian Neural Networks
#6I like Bayesian inference for few-parameter models where I have solid grounds for choosing my priors. For neural networks, I like to ask people "what's your prior for ReLU versus LeakyReLU versus sigmoid?" and I've never gotten a convincing answer.
Kolmogorov Arnold nets might have an answer for you!
Re: Bayesian Neural Networks
#7I like Bayesian inference for few-parameter models where I have solid grounds for choosing my priors. For neural networks, I like to ask people "what's your prior for ReLU versus LeakyReLU versus sigmoid?" and I've never gotten a convincing answer.
Kolmogorov Arnold nets might have an answer for you!
Could you say a bit more about how so?
Re: Bayesian Neural Networks
#8Earlier quoted context omitted.
Kolmogorov Arnold nets might have an answer for you!
Could you say a bit more about how so?
KANs have learnable activations based on splines parameterized on few variables. You can specify a prior over those variables, effectively establishing a prior over your activation function.
Re: Bayesian Neural Networks
#9[flagged]
Re: Bayesian Neural Networks
#10Bayesian Neural Networks just seem like a failed approach, unfortunately. For one, Bayesian inference and UQ fundamentally depends on the choice of the prior, but this is rarely discussed in the Bayesian NN literature and practice, and is further compounded by how fundamentally hard to interpret and choose these priors are (what is the intuition behind a NN's parameters?). Add to that the fact that the Bayesian infer…
Conformal learning is relatively new to me. Tell me if I'm getting any of this wrong: Conformal learning is a frequentist approach that uses a calibration set to determine how unusual a prediction is.
It seems like the main time they aren't a strict improvement over bayesian methods is when it is difficult to define your calibration set? I know this scenario isn't so commonplace, but I'm working in a scenario where I quickly looked at conformal learning and wasn't sure if it is applicable.