I rather like the derivation of the CLT using MGFs.
https://courses.cs.washington.edu/courses/cse312/20su/files/...
Probability-generating functions
31–40 of 54 posts
Re: Probability-generating functions
#32”If we want to encode the vector [6,2,8,4] in a single expression we can create a function containing those numbers: f(x) = 6 + 2x² + 8x³ + 4x⁴ …or if you flip the vector and use x=10: 6284
Yes, but the polynomial form generalizes to coefficients of an arbitrary field, not just naturals. If your vector were, say, [1.3, 2.197656, pi, -1/2, 3*2i] then there wouldn’t be a reasonable base you could pick for a place-value representation.
Re: Probability-generating functions
#33Probably worth noting that as we know know, polynomials (over a field) are a vector space, not just convertible to one. The set of formal variables { x^0, x^1, x^2, … } is an orthogonal basis.
Re: Probability-generating functions
#34Re: Probability-generating functions
#35For those interested in looking slightly more into the characteristic function, it may be worth pointing out that the characteristic function is equal to the Fourier-transform (with the sign of the argument being reversed) of the probability distribution in question. In my own experience teaching teaching probability theory to physicists and engineers, establishing this connection is often a good way of helping peopl…
I feel like it's almost criminal of textbook writers not to mention this when introducing the characteristic function... At least as an aside or a footnote, for readers already familiar with Fourier transforms.
Re: Probability-generating functions
#36I've always wondered why the hell generating functions existed, and I think this line sums it up:
> When de Moivre invented much of modern probability in the mid-1700s, he didn’t have vectors! Vectors are an 1800s invention.
Doesn't explain why we still teach them 300 years later though. Thats what the second half of the article covers.
Re: Probability-generating functions
#37Earlier quoted context omitted.
One can also think of probability generating functions as (flipped) Z transforms, moment generating functions as (flipped Laplace transforms), and characteristic functions as Fourier transforms of the respective PMF/PDF. Lot of their properties then follow from simple properties of Signals and Systems.
Do you have a reference that explains this in more detail? I'd be curious to know.
Don't have a reference on the top of my head, but the main idea is as follows:
The definition of MGF of a random variable with PDF f(x) is
E[e^{sX}] = int_{-inf}^{inf} f(x) e^{sx} dx
The definition of Laplace Transform of a signal f(t) is
F(s) = int _{-inf}^{inf} f(t) e^{-st} dt
Hence MGF is 'flipped' Laplace transform
Now for we know that the MGF of sum independent RVs is the product of their MGFs. So if we take the inverse Laplace transform, the density of the sum is convolution of the individual densities.
Similarly, if we take derivative in frequency domain, that is same as multiplying in time domain: So M'_X(s) is the 'flipped Laplace transform' of x f(x) and its value at s=0 is the 'DC-gain' of the signal.
And so on... the properties are all immediate consequence of the definition of MGF and since the definition is essentially the same as that of a Laplace transform , there is an equivalent property in signals and systems as well.
Re: Probability-generating functions
#38Is there a relationship between algebraic polynomial encoding of sequences, and https://en.wikipedia.org/wiki/G%C3%B6del_numbering_for_seque... ?
Does an encoding of a sequence in a given Gödel numbering, also somehow "retrievably" encode the probability space of the sequence's terms?
Re: Probability-generating functions
#39For those interested in looking slightly more into the characteristic function, it may be worth pointing out that the characteristic function is equal to the Fourier-transform (with the sign of the argument being reversed) of the probability distribution in question. In my own experience teaching teaching probability theory to physicists and engineers, establishing this connection is often a good way of helping peopl…
Yes, this provides good intuition about why it is useful: the PDF of the sum of two random variables is the convolution of the original PDFs. A convolution is awkward to work with, but by the convolution theorem it is a multiplication in the Fourier domain. This immediately suggests that the Fourier transform of a PDF would be a useful thing to work with. If you don't say that this is what you are doing then it all s…
> the PDF of the sum of two random variables is the convolution of the original PDFs
(Probably obvious to everyone reading, but the variables should be independent.)
Re: Probability-generating functions
#40"I have long struggled with understanding what probability-generating functions are and how to intuit them. There were two pieces of the puzzle missing for me, and we’ll go through both in this article." Great article. For more, I really recommend Analytic Combinatorics: https://ac.cs.princeton.edu/home/
Second this. This class is a classical example of conceptual blockbuster. Once one learns it, the complexity analysis of algorithms will never be the same again. In general, if a techie wants to spend their spare time learning new stuff, they will be better off focusing more on such conceptual stuff, as the return will compound over the years.