Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Tiny Boltzmann Machine

eoinmurray.info

Post reply on HN
Print view View original

51–59 of 59 posts

Re: A Tiny Boltzmann Machine

#51

Post by sfifs »

macintux wrote: post #41
AstroJetson wrote: post #32

Earlier quoted context omitted.

The key takeaways are that there are lots of people involved with making these breakthroughs. The value of grad students is often overlooked, they contribute so much and then later on advance the research even more. Why does America look on research as a waste, when it has move everything so far?

It's more accurate to say that businesspeople consider research a waste in our quarter-by-quarter investment climate, since it generally doesn't lead to immediate gains. And our current leadership considers research a threat, since science rarely supports conspiracy theorists or historical revisionism.

More charitably, America's current government has an unusually large concentration of business people. Interestingly, they were elected as a vote for change by a population of non business people who were tired of the economic marginalization they suffered when their government consisted largely of non business people not once but twice. It will be interesting to see how this plays out.

Re: A Tiny Boltzmann Machine

#52

Post by richk449 »

AstroJetson wrote: post #32
sitkack wrote: post #22

Fun article on David Ackley https://news.unm.edu/news/24-nobel-prize-in-physics-cited-gr... Do check out his T2 Tile Project.

The key takeaways are that there are lots of people involved with making these breakthroughs. The value of grad students is often overlooked, they contribute so much and then later on advance the research even more. Why does America look on research as a waste, when it has move everything so far?

Why do you say that America looks in research as a waste? We spend higher percentage of gdp on R&D than just about any other country in the world:

https://en.wikipedia.org/wiki/List_of_sovereign_states_by_re...

Re: A Tiny Boltzmann Machine

#54

Post by alimw »

oac wrote: post #44
itissid wrote: post #23

IIUC, we need gibbs sampling(to compute the weight updates) instead of using the gradient based forward and backward passes with today's NNetworks that we are used to. Any one understand why that is so?

I might be mistaken, but I think this is partly because of the undirected structure of RBMs, so you can't build a computational graph in the same way as with feed-forward networks.

By "undirected structure" I assume you refer to the presence of cycles in the graph? I was taught to call such networks "recurrent" but it seems that that term has evolved to mean something slightly different. Anyway yeah, because of the cycles Gibbs sampling is key to the network's operation. One still employs gradient descent during training, but the procedure to calculate the gradient itself involves Gibbs sampling.

Edit: Actually was talking about the General Boltzmann Machine. For the Restricted Boltzmann Machine an approximation has been assumed which obviates the need for full Gibbs sampling during training. Then (quoting the article, emphasis mine) "after training, it can sample new data from the learned distribution using Gibbs sampling."

Re: A Tiny Boltzmann Machine

#55

Post by anomancer »

anomancer wrote: post #53

Author here! Thanks for all the comments, didn't expect this to hit the front page. Cleaning up the abundance of typos, margin, and scroll issues now, thanks for pointing them out.

Have cleaned up the typos, and it should look much better on mobile now

Re: A Tiny Boltzmann Machine

#56

Post by nonrandomstring »

gopalv wrote: post #49
nonrandomstring wrote: post #6

This takes me back. 1990, building Boltzman machines and Perceptrons from arrays of void pointers to "neurons" in plain C. What did we use "AI" for back then? To guess the next note in a MIDI melody, and to recognise the shape of a scored note, minim, crotchet, quaver on a 5 x 9 dot grid. 85% accuracy was "good enough" then.

> recognise the shape of a scored note, minim, crotchet, quaver on a 5 x 9 dot grid Reading music off a lined page sounds like a fun project, particularly to do it from scratch like 3Blue1Brown's number NN example[1]. Mix with something like Chuck[2] and you can write a completely clientside application with today's tech. [1] - https://www.3blue1brown.com/lessons/neural-networks [2] - https://chuck.stanford.edu/

Thanks for these links. You're right, I think computer-vision "sight reading" is now a fairly done deal. Very impressive progress in the past 30 years.

Re: A Tiny Boltzmann Machine

#57

Post by btickell »

itissid wrote: post #23

IIUC, we need gibbs sampling(to compute the weight updates) instead of using the gradient based forward and backward passes with today's NNetworks that we are used to. Any one understand why that is so?

Thought I'd weigh in here as well, I believe Gibbs sampling is being used as a way to approximate the expectation over the model distribution. This value is required to compute the gradient of the log likelihood but integrating the distribution is intractable.

This is done in a similar way as you may use MCMC to draw a representative sample from a VAE. In the deep learning formulation of a neural network the gradient is estimated over batches of the dataset rather than over an explicitly modeled probability distribution.

Re: A Tiny Boltzmann Machine

#58

Post by anomancer »

croemer wrote: post #19
munchler wrote: post #15

Earlier quoted context omitted.

Another typo (or thinko) in the very first sentence: "Here we introduce introduction to Boltzmann machines"

More typos (LLMs are really good at finding these): "Press the "Run Simulation" button to start traininng the RBM." ("traininng" -> "training") "...we want to derivce the contrastive divergence algorithm..." ("derivce" -> "derive") "A visisble layer..." ("visisble" -> "visible")

Author here, there was so many typos in first draft that hit hn, all fixed now
Post reply on HN