Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Group Lasso Regularization

leimao.github.io

Post reply on HN
Print view View original

1–10 of 11 posts

Re: Group Lasso Regularization

#3

Post by abhgh »

Not very commonly discussed, but great topic to be aware of. One particularly powerful use of this is in learning models with interaction terms: more powerful than a linear model but still interpretable, [1].

[1] R package https://cran.r-project.org/web/packages/glinternet/index.htm... (paper linked therein)

Re: Group Lasso Regularization

#6

Post by currymj »

In Bayesian terms, ridge regression is equivalent to putting a Gaussian prior on the weights, which gets wider the weaker the penalty term.

The lasso is the same, but with a Laplace distribution as the prior.

The "elastic net", which combines the l1 and l2 penalties, even has a Bayesian interpretation, with a fairly weird prior [1, 2].

Anyone know if there's an equivalent Bayesian interpretation of group lasso? Maybe it's just a Gaussian prior but with block-wise correlation between the variables?

[1]: https://stats.stackexchange.com/questions/283238/is-there-a-...

[2]: https://www.tandfonline.com/doi/abs/10.1198/jasa.2011.tm0924...

Re: Group Lasso Regularization

#8

Post by laGrenouille »

currymj wrote: post #6

In Bayesian terms, ridge regression is equivalent to putting a Gaussian prior on the weights, which gets wider the weaker the penalty term. The lasso is the same, but with a Laplace distribution as the prior. The "elastic net", which combines the l1 and l2 penalties, even has a Bayesian interpretation, with a fairly weird prior [1, 2]. Anyone know if there's an equivalent Bayesian interpretation of group lasso? Maybe…

From a Bayesian perspective, it’s not entirely accurate to say that there is the same relationship between ridge/Gaussian-prior and lasso/Laplacian-prior. In the first case, ridge regression yields a "proper" Bayesian estimator (e.g., mean of the posterior); the latter is only the MAP estimator, which is not really recommended from a purely Bayesian perspective. Ridge is also a MAP estimator, but that’s in addition to the stronger condition of minimizing the Bayes risk.

There is a way to compute a "true" Bayesian Lasso, but it doesn’t yield a sparse model [0].

[0] http://archived.stat.ufl.edu/casella/Papers/Lasso.pdf

Re: Group Lasso Regularization

#9

Post by unemphysbro »

currymj wrote: post #6

In Bayesian terms, ridge regression is equivalent to putting a Gaussian prior on the weights, which gets wider the weaker the penalty term. The lasso is the same, but with a Laplace distribution as the prior. The "elastic net", which combines the l1 and l2 penalties, even has a Bayesian interpretation, with a fairly weird prior [1, 2]. Anyone know if there's an equivalent Bayesian interpretation of group lasso? Maybe…

It's similar to lasso where the Laplace distribution is decomposed using a gaussian mixture scaled with an exponential/gamma distribution. In group lasso, there is one variance term per group which is generated by the exponential/gamma prior.

See https://www.cs.ubc.ca/~murphyk/MLbook/

for the mathematical decomp. It should be noted that obtained using that prior in a MAP estimation per the other comment.

Post reply on HN