Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fun times with energy-based models

mpmisko.github.io

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Fun times with energy-based models

#11

Post by cshimmin »

blt wrote: post #9

If the author is reading: In the proof, at the end of Step 6, it's confusing that the "uv" term of the integration by parts is suddenly given a range from -∞ to ∞, as if we had previously assumed x ∈ ℝ. But elsewhere in the article, including the examples, we have higher-dimensional x's. I suggest to either 1) include the full multidimensional version from the paper, or 2) explicitly mention that this is the simple 1…

In the multidimensional case of integration by parts, the limit of integration is understood to be the (d-1 dimensional) boundary _at infinity _, so writing +/-inf is a reasonable shorthand. In almost all cases, it’s used when the term uv can be assumed to be zero (or at least constant) at this boundary.

Re: Fun times with energy-based models

#13

Post by uoaei »

I wish I had more opportunity to use EBMs. Joint distributions seem to be more relevant to our epistemology (vis a vis data and what we can say about it) than conditional ones. But the optimization steps for fitting parameters is kind of a dealbreaker because of how many steps it can take, and also because most ML frameworks are aggressively feedforward.

Re: Fun times with energy-based models

#14

Post by uoaei »

mpmisko wrote: post #4
esafak wrote: post #2

I think the rationale for using tricks like score matching and contrastive divergence deserves a mention: the partition function is computationally expensive. Since we're on the subject, what are EBMs good for today?

EBMs show up all over the place, apparently even your classifier is an EBM :) ( https://arxiv.org/abs/1912.03263 ).

You can take many equivalent perspectives on learning systems, but mostly it reduces to "messing with denominators in Bayes' rule". This is no different.

EBMs today aren't used because first you have to fit the joint model, then you have to fix some inputs, then fit the other inputs in a second optimization step. That's just too much compute for today's workloads compared to feedforward NNs.

Re: Fun times with energy-based models

#15

Post by uoaei »

adamnemecek wrote: post #10

We are working on a startup that is revisiting the math that underlies EBMs. If you want to work with us or invest, check out these links http://traceoid.ai https://x.com/adamnemecek1/status/1822727041399328839

Is there a chance that you would ever consider passionate non-PhD candidates with corporate, startup, and FAANG experience?

Re: Fun times with energy-based models

#16

Post by adamnemecek »

uoaei wrote: post #15
adamnemecek wrote: post #10

We are working on a startup that is revisiting the math that underlies EBMs. If you want to work with us or invest, check out these links http://traceoid.ai https://x.com/adamnemecek1/status/1822727041399328839

Is there a chance that you would ever consider passionate non-PhD candidates with corporate, startup, and FAANG experience?

Definitely. DM me on discord (see http://traceoid.ai for invite).
Post reply on HN