Isserlis' Theorem

Isserlis' theorem rewrites higher-order moments of zero-mean Gaussian random variables as a sum of $2$$2$nd-order moments of pairings. For zero-mean Gaussian random variables $x_1,x_2,\cdots ,x_d$$x_1, x_2, \cdots, x_d$, Isserlis' theorem tells us for even $d$$d$

where $p$$p$ denotes a pair of indices in $\{1,2,\cdots ,d\}$$\{1,2, \cdots , d\}$. There are $(d-1)!!$$(d-1)!!$ terms in the sum.

For odd $d$$d$​, $\mathrm{E}[x_1{x}_2\cdots x_d]=0$${\mathrm E} [x_1 x_2 \cdots x_d] =0$. Note that the $2$$2$nd-order moments are also covariances for zero-mean random variables: $\mathrm{E}[x_i{x}_j]=\mathrm{Cov}[x_i{x}_j]$${\mathrm E} [x_i x_j] = {\mathrm {Cov}} [x_i x_j]$.

As an example, Isserlis' theorem applied to the $4$$4$​-th order moments goes like this

Proof from Scratch

We have the random variable $\mathbf{x}\in {\mathbb{R}}^d$$\mathbf x \in \mathbb R^d$, which follows a zero-mean Gaussian distribution $\mathbf{x}\sim \mathcal{N}(\mathbf{0},\mathbf{\Sigma })$$\mathbf x \sim {\mathcal N}(\mathbf 0, \mathbf \Sigma)$.

We use a quadratic identity equation

Because the probability density function integrate to $1$$1$, we have

Then we make the genius move of differentiating Equation $(4)$$(4)$ with respect to $v_1,v_2,\cdots ,v_d$$v_1, v_2, \cdots, v_d$ and evaluating at $v_1,v_2,\cdots ,v_d=0$$v_1, v_2, \cdots, v_d=0$. Differentiating the right-hand side, we get exactly the moments we want

Differentiating the left-hand side, we get

This is getting really cumbersome to write. I am lazy; so let's just assume $d=4$$d=4$ as an example of the even $d$$d$ case. We continue the derivation for $d=4$$d=4$

Note that only the terms with $v_1{v}_2{v}_3{v}_4$$v_1 v_2 v_3 v_4$ in the sum have nonzero derivatives with respect to $v_1,v_2,v_3,v_4$$v_1, v_2, v_3, v_4$. There are $4!=24$$4!=24$ arrangements for $\{v_1,v_2,v_3,v_4\}$$\{v_1, v_2, v_3, v_4\}$. There are $C_4^2/2=3$$C_4^2/2=3$ permutations for ${\mathbf{\Sigma }}_{ij}{\mathbf{\Sigma }}_{kl}$$\mathbf \Sigma_{ij} \mathbf \Sigma_{kl}$, namely ${\mathbf{\Sigma }}_{12}{\mathbf{\Sigma }}_{34},{\mathbf{\Sigma }}_{13}{\mathbf{\Sigma }}_{24},{\mathbf{\Sigma }}_{14}{\mathbf{\Sigma }}_{23}$$\mathbf \Sigma_{12} \mathbf \Sigma_{34}, \mathbf \Sigma_{13} \mathbf \Sigma_{24}, \mathbf \Sigma_{14} \mathbf \Sigma_{23}$. Each permutation of ${\mathbf{\Sigma }}_{ij}{\mathbf{\Sigma }}_{kl}$$\mathbf \Sigma_{ij} \mathbf \Sigma_{kl}$ appears $24/3=8$$24/3=8$ times. We thus write out Equation $(8)$$(8)$ as

By the definition of the covariance matrix, we finish the proof

Although I only completed the derivation for $d=4$$d=4$, we can generalize the result to arbitrary even $d$$d$ by induction. For instance, for $d=6$$d=6$, we can first break things down to $2$$2$ and $4$$4$ and then break down the $4$$4$. Namely, we first have

The $4$$4$-th order moment in each term can be further decomposed to three terms that only contain $2$$2$nd moments using Equation $(2)$$(2)$. So we eventually break the $6$$6$-th order moment into $5\times 3=15$$5\times 3=15$ terms.

Proof via Stein's Lemma

If one already knows Stein's Lemma, Isserlis' theorem is quite easy to prove. Stein's lemma states that for zero-mean Gaussian random variables $x_1,x_2,\cdots ,x_d$$x_1, x_2, \cdots, x_d$, we have

Applying this lemma to $\mathrm{E}[x_1{x}_2{x}_3{x}_4]$${\mathrm E} [x_1 x_2 x_3 x_4]$ immediately gives us Equation $(2)$$(2)$

Comments

Isserlis' theorem is specific to zero-mean Gaussian random variables. It does not extend to other distributions.

Isserlis' theorem allows the random variables to have correlations. If the random variables are independent $x_1\perp x_2$$x_1 \perp x_2$, we have $\mathrm{E}[x_1{x}_2]=\mathrm{E}[x_1]\mathrm{E}[x_2]$${\mathrm E}[x_1 x_2] = {\mathrm E}[x_1] {\mathrm E}[x_2]$ no matter what distributions $x_1$$x_1$ and $x_2$$x_2$​ follow.