Many people learn about the philosophy of the Bayesian estimation and fall in love with it, or at least that happened to me.
I thought that only in the Bayesian formulation of statistics the estimated parameters (mean, standard deviation, kurtosis, percentiles, whatever) remain uncertain after the estimation, and therefore they have a distribution; I didn't know that in the frequentist interpretation what you calculate are actually estimators, and they are random variables and therefore have uncertainty in them. Silly me.
One day, reading about estimators on stackexchange, I was led to a quote from the "Elements of Statistical Learning" ([1], p. 272): "we might think of the bootstrap distribution as a poor man's Bayesian posterior".
So, I put off learning all the details of the Bayesian estimation, and I started using the bootstrap to deal with my problems. I recommend everyone to give the bootstrap method a try before they go all in for Bayesian estimation.
Once you get familiar with the bootstrap, you might feel that actually Bayesian estimation might be overkill.
But you shouldn't stop there. With a little more contemplation, you might start doubting the whole Bayesian edifice. Let me tell you why.
Here's a quote from a book [2] "Bayesian Risk Management" that, unsurprisingly, given the title, extols the virtues of the Bayesian framework: "If the data are consistent with our prior estimates, the location of the parameters will be little changed and the variance of the posterior distribution will shrink. If the data are surprising given our prior estimates, the variance will increase and the location will migrate" (p. 11). This is the common view of the Bayesian estimation, and it's wrong. To give you an example, if you do Bayesian estimation for the mean of a random sample assumed to come from a normal distribution with a given standard deviation, then the variance of that mean keeps going down with each observation regardless of the value of the observation (check first entry in [3])
But that shouldn't shatter your hope for a better world. The Bayesian estimation is bound to produce the wrong result if you assume the wrong model, regardless of what prior you use. But now you discover that the Bayesian estimation doesn't absolve you of the responsibility of choosing a good model (which is another commonly held believe). But then, if you do need to carefully choose a model, why do you need Bayesian after all.
I am not sure. Let me kick Bayesian while it's down a bit more, before I start defending it.
If you ever get curious about Kalman filter estimation, a good book to use is Durbin and Koopman [4]. In the first 20 pages you will learn that in the simplest setting (local level model), the Kalman filter gives exactly the same results in the frequentist and bayesian interpretation. Food for thought.
Here's a quote from Efron&Hastie "Computer Age Statistical Inference": "Computer-age statistical inference at its most successful combines elements of the two philosophies, as for instance the empirical Bayes methods in Chapter 6, and the lasso in Chapter 16. There are two arrows in the statistician's philosophical quiver, and faces, say, with 1000 parameters and 1,000,000 data points, there's no need to go hunting armed with just one of them."
Now I did say I'll come to the defense of Bayesian. To my knowledge the Markov Chain Monte Carlo method was developed only in the Bayesian setting. I do believe a frequentist interpretation is entirely possible, but so far nobody offered it. But until someone needs to use MCMC, I don't really see a need to go Bayesian, when bootstrap works perfectly fine.
[1] http://statweb.stanford.edu/~tibs/ElemStatLearn/
[2] http://www.wiley.com/WileyCDA/WileyTitle/productCd-111870860...
[3] https://en.wikipedia.org/wiki/Conjugate_prior#Continuous_dis...
[4] https://books.google.com/books/about/Time_Series_Analysis_by...
[5] https://web.stanford.edu/~hastie/CASI/