Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Probabilistic programming does in 50 lines of code what used to take thousands

newsoffice.mit.edu

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Probabilistic programming does in 50 lines of code what used to take thousands

#71

Post by FreeFull »

baldfat wrote: post #50
duaneb wrote: post #42

Earlier quoted context omitted.

You do not need 850 lines of code to output "hello world" unless you actually include a header. You could just stub printf instead of including and get it done with like 10 lines of code.

Assembly: Microsoft Macro Assembly "Hello World" .MODEL TINY .CODE CODE SEGMENT BYTE PUBLIC 'CODE' ASSUME CS:CODE,DS:CODE ORG 0100H DB 'HELLO WORLD$', 0 INC DH MOV AH,9 INT 21H RET

And then the interrupt calls into DOS code, which then calls into BIOS code, which then has the hardware do the work. The hardware itself will do stuff like translating instructions into microcode..

Re: Probabilistic programming does in 50 lines of code what used to take thousands

#72

Post by murbard2 »

BenoitEssiambre wrote: post #52
murbard2 wrote: post #40

Earlier quoted context omitted.

It's a very powerful statistical technique, yes, but I doubt it will be enough for AI. The problem is that you need to sample efficiently from the posterior distribution, and for anything AI related, MCMC isn't going to cut it. Let's take a step back and consider a logical problem. Can you put N socks in N-1 boxes such that no box contains more than one socks? Obviously not, it's the pigeonhole principle. Convert tha…

I have been out of the machine learning field for years and I haven't look into deep learning methods even though they are intriguing (just not sufficiently bayesian to pique my curiosity enough to spend my limited free time on it). I do believe that automatically building a hierarchical structure (as I assume happens in deep learning) is the way to abstract away complexity but I think this is achievable within a gen…

Using a probabilistic program to generate and interpret a program is an interesting idea, but I think you're missing the main problem.

How do you sample from the set of programs that produce the correct (or approximately correct) output?

You could use rejection sampling, but that would take very long as only a tiny fraction of all possible programs produce the desired output.

You could use a MCMC method, but the problem here is designing a proposal density. There is no reason to expect programs which are almost correct to be syntactically close to programs which aren't. Changing a single bit in a program tends to radically alter its behavior, in a very non linear way. Program space does not map smoothly to output space.

You mention hyperparameters, but what would those be? If you're building a probabilistic grammar, they might be things like emissions probabilities... but tweaking the proportion of "if" statements you produce, as opposed to, say, function calls, is hardly going to make you better at sampling from the set of correct programs.

In general, the idea of meta-learning is a powerful one, but without a way to guide the search, I don't see it as feasible.

Re: Probabilistic programming does in 50 lines of code what used to take thousands

#73

Post by eli_gottlieb »

dxbydt wrote: post #45
eob wrote: post #20

Earlier quoted context omitted.

It's a fair question, but from my heydays doing ML research even good libraries can be pretty revolutionary. The engineering-side of ML is still in relative infancy. For a great example look at all the papers Andrew McCallum's group has been able to publish by building on top of FACTORIE: they get to focus their time on the problem at hand rather than all the math required to solve it. Basically they can write code t…

> ML is getting closer and closer to being an everyday tool for engineers rather than something that requires you to be a full-time math person I so wish! As a Sr Data Scientist, I interview potential candidates quite often, many of these are 10x engineers. Me: (2,3,4) is a vector. Eng: Ok. Me: Gimme a unit vector in the same direction. Eng 1: ??? Eng 2: "It can be done. I don't know how, but with Spark it can be don…

Hmmm... I don't think I remember how to compute the eigenvectors of a 2x2 matrix, other than constructing a system of linear equations representing the outcome of the matrix multiplication and solving it.

But come on, everyone knows how to use the definition of the standard (l2) vector norm to normalize a vector! Don't joke!

Re: Probabilistic programming does in 50 lines of code what used to take thousands

#75

Post by eli_gottlieb »

murbard2 wrote: post #40
BenoitEssiambre wrote: post #34

Earlier quoted context omitted.

I've played with it a bit and, in my opinion, the principles behind it at least, the streamlined and optimized simulation of bayesian generative models is the best chance we have to solve artificial general intelligence. Reading probmods.org and dippl.org made me go from being very pessimistic I would see it in my lifetime to a solid maybe.

It's a very powerful statistical technique, yes, but I doubt it will be enough for AI. The problem is that you need to sample efficiently from the posterior distribution, and for anything AI related, MCMC isn't going to cut it. Let's take a step back and consider a logical problem. Can you put N socks in N-1 boxes such that no box contains more than one socks? Obviously not, it's the pigeonhole principle. Convert tha…

For the record, proving the pigeonhole principle in Coq is sufficiently doable to be a mere four-star textbook exercise (http://www.cis.upenn.edu/~bcpierce/sf/current/MoreLogic.html). No exponential explosion of terms required!

>The problem is that you need to sample efficiently from the posterior distribution, and for anything AI related, MCMC isn't going to cut it.

Which is precisely why so much of the actual research work in probabilistic programming is to add more and faster inference strategies. One paper already got a 600x speedup just by using some techniques from the compilers/VM world related to caching whole and partial computation traces.

Re: Probabilistic programming does in 50 lines of code what used to take thousands

#76

Post by eli_gottlieb »

murbard2 wrote: post #72
BenoitEssiambre wrote: post #52

Earlier quoted context omitted.

I have been out of the machine learning field for years and I haven't look into deep learning methods even though they are intriguing (just not sufficiently bayesian to pique my curiosity enough to spend my limited free time on it). I do believe that automatically building a hierarchical structure (as I assume happens in deep learning) is the way to abstract away complexity but I think this is achievable within a gen…

Using a probabilistic program to generate and interpret a program is an interesting idea, but I think you're missing the main problem. How do you sample from the set of programs that produce the correct (or approximately correct) output? You could use rejection sampling, but that would take very long as only a tiny fraction of all possible programs produce the desired output. You could use a MCMC method, but the prob…

>You could use a MCMC method, but the problem here is designing a proposal density. There is no reason to expect programs which are almost correct to be syntactically close to programs which aren't. Changing a single bit in a program tends to radically alter its behavior, in a very non linear way. Program space does not map smoothly to output space.

Ahaha, "this sounds like a job for topology!"

Re: Probabilistic programming does in 50 lines of code what used to take thousands

#77

Post by oh_sigh »

eximius wrote: post #5
newpattern wrote: post #2

Anybody here on HN have experience with probabilistic-programming ? This looks quite disruptive if it works.

Really? "Disruptive"?

Sure. If someone here has nothing more than high school math under their belt, they may not understand that this isn't a huge deal.

For example, I refer you to the paper wherein some biologists "discovered" integrals in 1993. http://care.diabetesjournals.org/content/17/2/152.abstract

Re: Probabilistic programming does in 50 lines of code what used to take thousands

#78

Post by BenoitEssiambre »

murbard2 wrote: post #72
BenoitEssiambre wrote: post #52

Earlier quoted context omitted.

I have been out of the machine learning field for years and I haven't look into deep learning methods even though they are intriguing (just not sufficiently bayesian to pique my curiosity enough to spend my limited free time on it). I do believe that automatically building a hierarchical structure (as I assume happens in deep learning) is the way to abstract away complexity but I think this is achievable within a gen…

Using a probabilistic program to generate and interpret a program is an interesting idea, but I think you're missing the main problem. How do you sample from the set of programs that produce the correct (or approximately correct) output? You could use rejection sampling, but that would take very long as only a tiny fraction of all possible programs produce the desired output. You could use a MCMC method, but the prob…

" There is no reason to expect programs which are almost correct to be syntactically close to programs which aren't. Changing a single bit in a program tends to radically alter its behavior, in a very non linear way. Program space does not map smoothly to output space."

You correctly identify the main hurdle. I admit I am not sure it will work.

However, I think it might be possible to design the programming language such that changing a few bits doesn't usually radically alter the output.

For example, let's say you are trying to learn a grammar to draw a blue irregular polygon. If there are primitives with fewer parameters that can get you an approximate output, say a circle, this makes possible a learning path that looks like "circle"->"blue circle"->"simple blue polygon"->"complex blue polygon". In addition to that, if the grammar rules that generate similar output can be clustered in the space of grammar rules, small jumps in this space may give you small output changes. Using bayesian priors will naturally use the simpler, fewer parameter shapes first and pivot to more complex ones as enough information is learned while, I think, creating these more complex rules close to the simple ones in the space of grammar rules. That is my hope anyways. I got it working as expected-ish with a simple vision example like I just described.

Re: Probabilistic programming does in 50 lines of code what used to take thousands

#79

Post by Retra »

vidarh wrote: post #70
Retra wrote: post #59

Earlier quoted context omitted.

Isn't functional programming a standard part of any computer science curriculum? Why would you expect programmers not to know it?

Even the places where it is a part of the curriculum, it is often such a small part that unless people specifically take courses related to functional programming you can't expect them to be able to actually use it. Or remember much of it for that matter. Heck, I spent months on a binge reading functional programming research papers, and it still doesn't mean I know any functional languages other than very superficia…

That's true of any language though. If you know Java, you don't automatically know C#, but the core concepts will still mostly transfer. You'll still know Object-Oriented Programming, even if you've never used C# and can't write HelloWorld without looking something up.

Re: Probabilistic programming does in 50 lines of code what used to take thousands

#80

Post by stevenspasbo »

x0054 wrote: post #63
baldfat wrote: post #19

Earlier quoted context omitted.

What do we say about languages built on C? Is it 100 lines of code but there are hundreds of thousands of lines of code for that higher level language you just coded? I don't think libraries count in terms of code. We all use code to program. Standing on the shoulder that preceded us. Using a library and a function should just count for the most part.

I think stevenspasbo did not mean to criticize the progress in this area, but rather just point out the misleading title of the article. Unless you program everything in pure ASM, Bios and up, you always stand on the shoulders of others, that's how since works. But the title of the article, "Probabilistic programming does in 50 lines of code what used to take thousands," makes it sound as if there was a solution to t…

Thank you, that's exactly what I meant.
Post reply on HN