Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BayesDB: A probabilistic programming platform

probcomp.csail.mit.edu

Post reply on HN
Print view View original

11–16 of 16 posts

Re: BayesDB: A probabilistic programming platform

#11

Post by sanxiyn »

mozartoz wrote: post #8
indescions_2017 wrote: post #6

Glad to see this is out as well! Using probabilistic frameworks has the potential to eliminate a lot of the human error which can easily enter a large simulation. It's fair to say in the future probabilistic modules will become part of every standard library in every programming language, and distribution sampling functions will be as common as trig functions in a math library. I am curious though how I would build u…

Some probabilisitic systems can cope with big volumes of data. E.g. http://i.stanford.edu/hazy/

Note that Apple recently acquired this technology. Lattice Data is the commercialization of DeepDive.

Re: BayesDB: A probabilistic programming platform

#12

Post by carterschonwald »

3pt14159 wrote: post #4

Welcome to the project I've been waiting for years to get out of alpha. It's frustrating. If I had a hundred million dollars I'd burn a couple million getting this funded. It seems like it will be useful to humanity.

... they are funded. (I know of >=1 of the investors) https://empirical.com/

Re: BayesDB: A probabilistic programming platform

#14

Post by nightski »

afpx wrote: post #7

Anyone doing probabilistic programming with big data? I started experimenting with probabilistic programming frameworks several years ago, but couldn't get it to scale to the level of data I'm working with (~6 dimensions ~100 trillion vectors), or even a small fraction of that. But, I'm sure it's being done in scientific circles somewhere. Are there communities to collaborate on probabilistic programming? It seems li…

I have not, so take my words with a grain of salt. However, with only 6 dimensions 100 trillion vectors is probably extremely overkill (unless you are talking time series data?). Probabilistic programming, from what I understand, thrives on situations where you do not have a lot of data (instead of an overabundance).

You could probably achieve very interesting results by taking a much smaller sample from your large data set.

Re: BayesDB: A probabilistic programming platform

#15

Post by afpx »

nightski wrote: post #14
afpx wrote: post #7

Anyone doing probabilistic programming with big data? I started experimenting with probabilistic programming frameworks several years ago, but couldn't get it to scale to the level of data I'm working with (~6 dimensions ~100 trillion vectors), or even a small fraction of that. But, I'm sure it's being done in scientific circles somewhere. Are there communities to collaborate on probabilistic programming? It seems li…

I have not, so take my words with a grain of salt. However, with only 6 dimensions 100 trillion vectors is probably extremely overkill (unless you are talking time series data?). Probabilistic programming, from what I understand, thrives on situations where you do not have a lot of data (instead of an overabundance). You could probably achieve very interesting results by taking a much smaller sample from your large d…

Could be. I'm just a software engineer with a high pain tolerance and a lot of tenacity who is trying to make conclusions from lots of error-ridden data. It seems difficult to find experts in this space. I've interviewed several statistics Ph.d. from reliable schools, but no luck yet. By the way, sampling doesn't seem to help because all of the data is relational. Or, maybe I don't know how to sample it effectively.

Anyway, in my case, I have data feeds, but none of them are 100% reliable. There is error, and I can guess the error. I want to infer things from the data, but I know that the conclusions are unreliable. So, I want to know how unreliable my conclusions are, if that makes sense.

Anyway, I'm an amateur here. But, My independent research led me to things like MCMC and probabilistic programming which allows me to model things better.

Re: BayesDB: A probabilistic programming platform

#16

Post by 2309kdujj »

currymj wrote: post #10
afpx wrote: post #7

Anyone doing probabilistic programming with big data? I started experimenting with probabilistic programming frameworks several years ago, but couldn't get it to scale to the level of data I'm working with (~6 dimensions ~100 trillion vectors), or even a small fraction of that. But, I'm sure it's being done in scientific circles somewhere. Are there communities to collaborate on probabilistic programming? It seems li…

Things have gotten much more mature in the past few years. Check out edwardlib.org which is adapting Tensorflow to support probabilistic modeling (with a heavy focus on variational inference, less so on MCMC methods). If you’ve got trillions of observations you can use stochastic VI. And tensorflow now can do distributed computation graphs, or you could just go data parallel and then average your parameters at the en…

Funny you mention those.

I've been really excited about Edward but when I tried it for a project last year I could never get it to come together in the right way. I got the sense that it wasn't quite ready for prime time yet, although very promising. My memory of it was that a lot of claimed flexibility in how to specify models wasn't really implemented fully. The experience also turned me off of TensorFlow a bit. But that was a year ago, so maybe it's improved?

I ended up doing it in Stan in part because I was more familiar with that, and it worked out fairly well.

Just a personal anecdote.

Post reply on HN