Karhunen-Loève expansions: A primer
Somehow I’ve managed to have nothing to do with Karhunen-Loève expansions of random processes at work in lab. A couple of weeks ago, though, I started working with stochastic collocation for partial differential equations and there they were, Karhunen-Loève expansions. That name, Karhunen-Loève, is long enough to promise good entertainment, and as probability theory remains in the list of topics that give me an insta-headache everytime I have to deal with them, it seemed to me, well, not a recipe for a good time. To my surprise, though, after a couple of days of hands-on I can say this is actually a lot of fun (Well, at least as fun as probability theory can be in your book). The topic allows going from a state of almost-no-idea-whatsoever to oh-now-I-kinda-get-what’s-going-on in a not painfully long time. Let’s summarize then some of the stuff I’ve learned about K-L expansions and in the meantime practice some .
Let’s say we have a second order centered random process , with (that is, with zero mean, or ). We’ll assume that is continuous in quadratic mean (or q.m.), so that we can employ the calculus in q.m. developed by Loève (1977) (for more information on that, and expositions on the topic with actual technical rigour, check that reference and Potthoff, 2010). This process can be represented as an infinite linear combination of orthonormal functions whose coefficients are zero-mean random variables , that is,
As they are orthonormal, the functions satisfy . The coefficients are then given by
as is customary with orthogonal expansions. K-L expansions require the random variables to be mutually uncorrelated, i.e., , with the variance of . What does this say about the random variables and the orthonormal functions? To investigate this, we’ll follow the procedure outlined in this article, starting with using the expression above for the to rewrite the covariance of the random variables:
where is the covariance of (which exists, as is second order). This last expression can be rewritten as
which can be easily satisfied by setting the stuff in the curly brackets equal to . If we let we obtain the following relation:
This result, a Friedholm equation, states the relationship satisfied by each of the and the variance of its corresponding random variable. It is not difficult to see that and are the set of eigenfunctions and eigenvalues of the linear operator
that is, .
The usual form of the K-L expansion is obtained by writing , where the are zero-mean, unit variance random variables. The final result is then
where , satisfy (1) and the random variables are given by
with . It can be proved that the series (2) converges uniformly in . For this, we’ll follow the procedure of Loève (1977) and start introducing Mercer’s theorem: A nonnegative-definite type function continuous in can be expanded as
where , are again the solutions of (1). This series converges absolutely and uniformly of . Covariance functions are positive-definite, so good times. Now, equation (3) implies ; also, we need to define
With these elements, it’s easy to see that
In virtue of Mercer’s theorem, the last term on the RHS converges to as and the LHS converges to zero uniformly on .
Well, that’s gonna be pretty much it for the moment. I just want to add that this entry was brought to you with the help of LaTeX2WP, this super-neat Python script that converts LaTeX file to discernible HTML ready to copy-paste to the WordPress editor. Awesome! Now go and do something fun.
Adler, R.J. (1990). An Introduction to Continuity, Extrema and Related Topics for General Gaussian Processes. Institute of Mathematical Statistics.
Johnson, D. (n.d.). Karhunen-Loève Expansions. In Connexions. Retrieved March 17, 2011 from http://cnx.org/content/m11259/latest/
Loève, M. (1978). Probability theory. Vol. II, 4th ed. New York: Springer-Verlag.
Potthoff, J. (2010). Sample properties of random fields — III: Differentiability. Commun. Stoch. Anal., 4, 335-353.
And obviously, the Wikipedia article:
Karhunen-Loève theorem. (n.d.). In Wikipedia. Retrieved March 17, 2011 from http://en.wikipedia.org/wiki/Karhunen-Loève_theorem