1.4 - Method of Moments

In short, the method of moments involves equating sample moments with theoretical moments. So, let's start by making sure we recall the definitions of theoretical moments, as well as learn the definitions of sample moments.

Definitions.

  1. \(E(X^k)\) is the \(k^{th}\) (theoretical) moment of the distribution (about the origin), for \(k=1, 2, \ldots\)
  2. \(E\left[(X-\mu)^k\right]\) is the \(k^{th}\) (theoretical) moment of the distribution (about the mean), for \(k=1, 2, \ldots\)
  3. \(M_k=\dfrac{1}{n}\sum\limits_{i=1}^n X_i^k\) is the \(k^{th}\) sample moment, for \(k=1, 2, \ldots\)
  4. \(M_k^\ast =\dfrac{1}{n}\sum\limits_{i=1}^n (X_i-\bar{X})^k\) is the \(k^{th}\) sample moment about the mean, for \(k=1, 2, \ldots\)

One Form of the Method Section

The basic idea behind this form of the method is to:

  1. Equate the first sample moment about the origin \(M_1=\dfrac{1}{n}\sum\limits_{i=1}^n X_i=\bar{X}\) to the first theoretical moment \(E(X)\).
  2. Equate the second sample moment about the origin \(M_2=\dfrac{1}{n}\sum\limits_{i=1}^n X_i^2\) to the second theoretical moment \(E(X^2)\).
  3. Continue equating sample moments about the origin, \(M_k\), with the corresponding theoretical moments \(E(X^k), \; k=3, 4, \ldots\) until you have as many equations as you have parameters.
  4. Solve for the parameters.

The resulting values are called method of moments estimators. It seems reasonable that this method would provide good estimates, since the empirical distribution converges in some sense to the probability distribution. Therefore, the corresponding moments should be about equal.

Example 1-7 Section

Let \(X_1, X_2, \ldots, X_n\) be Bernoulli random variables with parameter \(p\). What is the method of moments estimator of \(p\)?

Answer

Here, the first theoretical moment about the origin is:

\(E(X_i)=p\)

We have just one parameter for which we are trying to derive the method of moments estimator. Therefore, we need just one equation. Equating the first theoretical moment about the origin with the corresponding sample moment, we get:

\(p=\dfrac{1}{n}\sum\limits_{i=1}^n X_i\)

Now, we just have to solve for \(p\). Whoops! In this case, the equation is already solved for \(p\). Our work is done! We just need to put a hat (^) on the parameter to make it clear that it is an estimator. We can also subscript the estimator with an "MM" to indicate that the estimator is the method of moments estimator:

\(\hat{p}_{MM}=\dfrac{1}{n}\sum\limits_{i=1}^n X_i\)

So, in this case, the method of moments estimator is the same as the maximum likelihood estimator, namely, the sample proportion.

Example 1-8 Section

Let \(X_1, X_2, \ldots, X_n\) be normal random variables with mean \(\mu\) and variance \(\sigma^2\). What are the method of moments estimators of the mean \(\mu\) and variance \(\sigma^2\)?

Answer

The first and second theoretical moments about the origin are:

\(E(X_i)=\mu\qquad E(X_i^2)=\sigma^2+\mu^2\)

(Incidentally, in case it's not obvious, that second moment can be derived from manipulating the shortcut formula for the variance.) In this case, we have two parameters for which we are trying to derive method of moments estimators. Therefore, we need two equations here. Equating the first theoretical moment about the origin with the corresponding sample moment, we get:

\(E(X)=\mu=\dfrac{1}{n}\sum\limits_{i=1}^n X_i\)

And, equating the second theoretical moment about the origin with the corresponding sample moment, we get:

\(E(X^2)=\sigma^2+\mu^2=\dfrac{1}{n}\sum\limits_{i=1}^n X_i^2\)

Now, the first equation tells us that the method of moments estimator for the mean \(\mu\) is the sample mean:

\(\hat{\mu}_{MM}=\dfrac{1}{n}\sum\limits_{i=1}^n X_i=\bar{X}\)

And, substituting the sample mean in for \(\mu\) in the second equation and solving for \(\sigma^2\), we get that the method of moments estimator for the variance \(\sigma^2\) is:

\(\hat{\sigma}^2_{MM}=\dfrac{1}{n}\sum\limits_{i=1}^n X_i^2-\mu^2=\dfrac{1}{n}\sum\limits_{i=1}^n X_i^2-\bar{X}^2\)

which can be rewritten as:

\(\hat{\sigma}^2_{MM}=\dfrac{1}{n}\sum\limits_{i=1}^n( X_i-\bar{X})^2\)

Again, for this example, the method of moments estimators are the same as the maximum likelihood estimators.

In some cases, rather than using the sample moments about the origin, it is easier to use the sample moments about the mean. Doing so provides us with an alternative form of the method of moments.

Another Form of the Method Section

The basic idea behind this form of the method is to:

  1. Equate the first sample moment about the origin \(M_1=\dfrac{1}{n}\sum\limits_{i=1}^n X_i=\bar{X}\) to the first theoretical moment \(E(X)\).
  2. Equate the second sample moment about the mean \(M_2^\ast=\dfrac{1}{n}\sum\limits_{i=1}^n (X_i-\bar{X})^2\) to the second theoretical moment about the mean \(E[(X-\mu)^2]\).
  3. Continue equating sample moments about the mean \(M^\ast_k\) with the corresponding theoretical moments about the mean \(E[(X-\mu)^k]\), \(k=3, 4, \ldots\) until you have as many equations as you have parameters.
  4. Solve for the parameters.

Again, the resulting values are called method of moments estimators.

Example 1-9 Section

Let \(X_1, X_2, \dots, X_n\) be gamma random variables with parameters \(\alpha\) and \(\theta\), so that the probability density function is:

\(f(x_i)=\dfrac{1}{\Gamma(\alpha) \theta^\alpha}x^{\alpha-1}e^{-x/\theta}\)

for \(x>0\). Therefore, the likelihood function:

\(L(\alpha,\theta)=\left(\dfrac{1}{\Gamma(\alpha) \theta^\alpha}\right)^n (x_1x_2\ldots x_n)^{\alpha-1}\text{exp}\left[-\dfrac{1}{\theta}\sum x_i\right]\)

is difficult to differentiate because of the gamma function \(\Gamma(\alpha)\). So, rather than finding the maximum likelihood estimators, what are the method of moments estimators of \(\alpha\) and \(\theta\)?

Answer

The first theoretical moment about the origin is:

\(E(X_i)=\alpha\theta\)

And the second theoretical moment about the mean is:

\(\text{Var}(X_i)=E\left[(X_i-\mu)^2\right]=\alpha\theta^2\)

Again, since we have two parameters for which we are trying to derive method of moments estimators, we need two equations. Equating the first theoretical moment about the origin with the corresponding sample moment, we get:

\(E(X)=\alpha\theta=\dfrac{1}{n}\sum\limits_{i=1}^n X_i=\bar{X}\)

And, equating the second theoretical moment about the mean with the corresponding sample moment, we get:

\(Var(X)=\alpha\theta^2=\dfrac{1}{n}\sum\limits_{i=1}^n (X_i-\bar{X})^2\)

Now, we just have to solve for the two parameters \(\alpha\) and \(\theta\). Let'sstart by solving for \(\alpha\) in the first equation \((E(X))\). Doing so, we get:

\(\alpha=\dfrac{\bar{X}}{\theta}\)

Now, substituting \(\alpha=\dfrac{\bar{X}}{\theta}\) into the second equation (\(\text{Var}(X)\)), we get:

\(\alpha\theta^2=\left(\dfrac{\bar{X}}{\theta}\right)\theta^2=\bar{X}\theta=\dfrac{1}{n}\sum\limits_{i=1}^n (X_i-\bar{X})^2\)

Now, solving for \(\theta\)in that last equation, and putting on its hat, we get that the method of moment estimator for \(\theta\) is:

\(\hat{\theta}_{MM}=\dfrac{1}{n\bar{X}}\sum\limits_{i=1}^n (X_i-\bar{X})^2\)

And, substituting that value of \(\theta\)back into the equation we have for \(\alpha\), and putting on its hat, we get that the method of moment estimator for \(\alpha\) is:

\(\hat{\alpha}_{MM}=\dfrac{\bar{X}}{\hat{\theta}_{MM}}=\dfrac{\bar{X}}{(1/n\bar{X})\sum\limits_{i=1}^n (X_i-\bar{X})^2}=\dfrac{n\bar{X}^2}{\sum\limits_{i=1}^n (X_i-\bar{X})^2}\)

Example 1-10 Section

Let's return to the example in which \(X_1, X_2, \ldots, X_n\) are normal random variables with mean \(\mu\) and variance \(\sigma^2\). What are the method of moments estimators of the mean \(\mu\) and variance \(\sigma^2\)?

Answer

The first theoretical moment about the origin is:

\(E(X_i)=\mu\)

And, the second theoretical moment about the mean is:

\(\text{Var}(X_i)=E\left[(X_i-\mu)^2\right]=\sigma^2\)

Again, since we have two parameters for which we are trying to derive method of moments estimators, we need two equations. Equating the first theoretical moment about the origin with the corresponding sample moment, we get:

\(E(X)=\mu=\dfrac{1}{n}\sum\limits_{i=1}^n X_i\)

And, equating the second theoretical moment about the mean with the corresponding sample moment, we get:

\(\sigma^2=\dfrac{1}{n}\sum\limits_{i=1}^n (X_i-\bar{X})^2\)

Now, we just have to solve for the two parameters. Oh! Well, in this case, the equations are already solved for \(\mu\)and \(\sigma^2\). Our work is done! We just need to put a hat (^) on the parameters to make it clear that they are estimators. Doing so, we get that the method of moments estimator of \(\mu\)is:

\(\hat{\mu}_{MM}=\bar{X}\)

(which we know, from our previous work, is unbiased). The method of moments estimator of \(\sigma^2\)is:

\(\hat{\sigma}^2_{MM}=\dfrac{1}{n}\sum\limits_{i=1}^n (X_i-\bar{X})^2\)

(which we know, from our previous work, is biased). This example, in conjunction with the second example, illustrates how the two different forms of the method can require varying amounts of work depending on the situation.