Here, we present and prove four key properties of an exponential random variable.

In the previous lesson, we learned that in an approximate Poisson process with mean \(\lambda\), the waiting time \(X\) until the first event occurs follows an exponential distribution with mean \(\theta=\frac{1}{\lambda}\). We now let \(W\) denote the waiting time until the \(\alpha^{th}\) event occurs and find the distribution of \(W\). We could represent the situation as follows:

Just as we did in our work with deriving the exponential distribution, our strategy here is going to be to first find the cumulative distribution function \(F(w)\) and then differentiate it to get the probability density function \(f(w)\). Now, for \(w>0\) and \(\lambda>0\), the definition of the cumulative distribution function gives us:

\(F(w)=P(W\le w)\)

The rule of complementary events tells us then that:

\(F(w)=1-P(W> w)\)

Now, the waiting time \(W\) is greater than some value \(w\) only if there are fewer than \(\alpha\) events in the interval \([0,w]\). That is:

\(F(w)=1-P(\text{fewer than }\alpha\text{ events in } [0,w]) \)

A more specific way of writing that is:

\(F(w)=1-P(\text{0 events or 1 event or ... or }(\alpha-1)\text{ events in } [0,w]) \)

Those mutually exclusive "ors" mean that we need to add up the probabilities of having 0 events occurring in the interval \([0,w]\), 1 event occurring in the interval \([0,w]\), ..., up to \((\alpha-1)\) events in \([0,w]\). Well, that just involves using the probability mass function of a Poisson random variable with mean \(\lambda w\). That is:

\(F(w)=1-\sum\limits_{k=0}^{\alpha-1} \dfrac{(\lambda w)^k e^{-\lambda w}}{k!}\)

Now, we could leave \(F(w)\) well enough alone and begin the process of differentiating it, but it turns out that the differentiation goes much smoother if we rewrite \(F(w)\) as follows:

\(F(w)=1-e^{-\lambda w}-\sum\limits_{k=1}^{\alpha-1} \dfrac{1}{k!} \left[(\lambda w)^k e^{-\lambda w}\right]\)

As you can see, we merely pulled the \(k=0\) out of the summation and rewrote the probability mass function so that it would be easier to administer the product rule for differentiation.

Now, let's do that differentiation! We need to differentiate \(F(w)\) with respect to \(w\) to get the probability density function \(f(w)\). Using the product rule, and what we know about the derivative of \(e^{\lambda w}\) and \((\lambda w)^k\), we get:

\(f(w)=F'(w)=\lambda e^{-\lambda w} -\sum\limits_{k=1}^{\alpha-1} \dfrac{1}{k!} \left[(\lambda w)^k \cdot (-\lambda e^{-\lambda w})+ e^{-\lambda w} \cdot k(\lambda w)^{k-1} \cdot \lambda \right]\)

Pulling \(\lambda e^{-\lambda w}\) out of the summation, and dividing \(k\) by \(k!\) (to get \( \frac{1}{(k-1)!}\)) in the second term in the summation, we get that \(f(w)\) equals:

\(=\lambda e^{-\lambda w}+\lambda e^{-\lambda w}\left[\sum\limits_{k=1}^{\alpha-1} \left\{ \dfrac{(\lambda w)^k}{k!}-\dfrac{(\lambda w)^{k-1}}{(k-1)!} \right\}\right]\)

Evaluating the terms in the summation at \(k=1, k=2\), up to \(k=\alpha-1\), we get that \(f(w)\) equals:

\(=\lambda e^{-\lambda w}+\lambda e^{-\lambda w}\left[(\lambda w-1)+\left(\dfrac{(\lambda w)^2}{2!}-\lambda w\right)+\cdots+\left(\dfrac{(\lambda w)^{\alpha-1}}{(\alpha-1)!}-\dfrac{(\lambda w)^{\alpha-2}}{(\alpha-2)!}\right)\right]\)

Do some (lots of!) crossing out (\(\lambda w -\lambda w =0\), for example), and a bit more simplifying to get that \(f(w)\) equals:

\(=\lambda e^{-\lambda w}+\lambda e^{-\lambda w}\left[-1+\dfrac{(\lambda w)^{\alpha-1}}{(\alpha-1)!}\right]=\lambda e^{-\lambda w}-\lambda e^{-\lambda w}+\dfrac{\lambda e^{-\lambda w} (\lambda w)^{\alpha-1}}{(\alpha-1)!}\)

And since \(\lambda e^{-\lambda w}=\lambda e^{-\lambda w}=0\), we get that \(f(w)\) equals:

\(=\dfrac{\lambda e^{-\lambda w} (\lambda w)^{\alpha-1}}{(\alpha-1)!}\)

Are we there yet? Almost! We just need to reparameterize (if \(\theta=\frac{1}{\lambda}\), then \(\lambda=\frac{1}{\theta}\)). Doing so, we get that the probability density function of \(W\), the waiting time until the \(\alpha^{th}\) event occurs, is:

\(f(w)=\dfrac{1}{(\alpha-1)! \theta^\alpha} e^{-w/\theta} w^{\alpha-1}\)

for \(w>0, \theta>0\), and \(\alpha>0\).

Recall that \(\theta\) is the mean waiting time until the first event, and \(\alpha\) is the number of events for which you are waiting to occur. It makes sense then that for fixed \(\alpha\), as \(\theta\) increases, the probability "moves to the right," as illustrated here with \(\alpha\)fixed at 3, and \(\theta\) increasing from 1 to 2 to 3:

The plots illustrate, for example, that if we are waiting for \(\alpha=3\) events to occur, we have a greater probability of our waiting time \(X\) being large if our mean waiting time until the first event is large (\(\theta=3\), say) than if it is small (\(\theta=1\), say).

It also makes sense that for fixed \(\theta\), as \(\alpha\) increases, the probability "moves to the right," as illustrated here with \(\theta\)fixed at 3, and \(\alpha\) increasing from 1 to 2 to 3:

The plots illustrate, for example, that if the mean waiting time until the first event is \(\theta=3\), then we have a greater probability of our waiting time \(X\) being large if we are waiting for more events to occur (\(\alpha=3\), say) than fewer (\(\alpha=1\), say).

We'll primarily use the definition in order to help us prove the two theorems that follow.

Here, after formally defining the gamma distribution (we haven't done that yet?!), we present and prove (well, sort of!) three key properties of the gamma distribution.

Before we get to the three theorems and proofs, two notes:

1. We consider \(\alpha>0\) a positive integer if the derivation of the p.d.f. is motivated by waiting times until α events. But the p.d.f. is actually a valid p.d.f. for any \(\alpha>0\) (since \(\Gamma(\alpha)\) is defined for all positive \(\alpha\)).

2. The gamma p.d.f. reaffirms that the exponential distribution is just a special case of the gamma distribution. That is, when you put \(\alpha=1\) into the gamma p.d.f., you get the exponential p.d.f.

Chi-squared distributions are very important distributions in the field of statistics. As such, if you go on to take the sequel course, Stat 415, you will encounter the chi-squared distributions quite regularly. In this course, we'll focus just on introducing the basics of the distributions to you. In Stat 415, you'll see its many applications.

As it turns out, the chi-square distribution is just a special case of the gamma distribution! Let's take a look.

As the following theorems illustrate, the moment generating function, mean and variance of the chi-square distributions are just straightforward extensions of those for the gamma distributions.

One of the primary ways that you will find yourself interacting with the chi-square distribution, primarily later in Stat 415, is by needing to know either a chi-square value or a chi-square probability in order to complete a statistical analysis. For that reason, we'll now explore how to use a typical chi-square table to look up chi-square values and/or chi-square probabilities. Let's start with two definitions.

Definition. Let \(\alpha\) be some probability between 0 and 1 (most often, a small probability less than 0.10). The upper \(100\alpha^{th}\) percentile of a chi-square distribution with \(r\) degrees of freedom is the value \(\chi^2_\alpha (r)\) such that the area under the curve and to the right of \(\chi^2_\alpha (r)\) is \(\alpha\):

The above definition is used, as is the one that follows, in Table IV, the chi-square distribution table in the back of your textbook.

Definition. Let \(\alpha\) be some probability between 0 and 1 (most often, a small probability less than 0.10). The \(100\alpha^{th}\) percentile of a chi-square distribution with \(r\) degrees of freedom is the value \(\chi^2_{1-\alpha} (r)\) such that the area under the curve and to the right of \(\chi^2_{1-\alpha} (r)\) is \(1-\alpha\):

With these definitions behind us, let's now take a look at the chi-square table in the back of your textbook.

In summary, here are the steps you should use in using the chi-square table to find a chi-square value:

1. Find the row that corresponds to the relevant degrees of freedom, \(r\) .
2. Find the column headed by the probability of interest... whether it's 0.01, 0.025, 0.05, 0.10, 0.90, 0.95, 0.975, or 0.99.
3. Determine the chi-square value where the \(r\) row and the probability column intersect.

Now, at least theoretically, you could also use the chi-square table to find the probability associated with a particular chi-square value. But, as you can see, the table is pretty limited in that direction. For example, if you have a chi-square random variable with 5 degrees of freedom, you could only find the probabilities associated with the chi-square values of 0.554, 0.831, 1.145, 1.610, 9.236, 11.07, 12.83, and 15.09:

What would you do if you wanted to find the probability that a chi-square random variable with 5 degrees of freedom was less than 6.2, say? Well, the answer is, of course... statistical software, such as SAS or Minitab! For what we'll be doing in Stat 414 and 415, the chi-square table will (mostly) serve our purpose. Let's get a bit more practice now using the chi-square table.

Sometimes taking the integral is not an easy task. We do have some tools, however, to help avoid some of them. Let's take a look at an example.

Suppose we have a random variable, \(X\), that has a Gamma distribution and we want to find the Moment Generating function of \(X\), \(M_X(t)\). There is an example of how to compute this in the notes but let's try it another way.

\(\begin{align*}  M_X(t)&=\int_0^\infty \frac{1}{\Gamma(\alpha)\beta^\alpha} x^{\alpha-1}e^{-x/\beta}e^{tx}dx\\ & = \int_0^\infty \frac{1}{\Gamma(\alpha)\beta^\alpha} x^{\alpha-1}e^{-x\left(\frac{1}{\beta}-t\right)}dx \end{align*}\)

Let's rewrite this integral, taking out the constants for now and rewriting the exponent term.

\(\begin{align*}  M_X(t)&= \frac{1}{\Gamma(\alpha)\beta^\alpha}\int_0^\infty x^{\alpha-1}e^{-x\left(\frac{1}{\beta}-t\right)}dx\\ & =\frac{1}{\Gamma(\alpha)\beta^\alpha}\int_0^\infty x^{\alpha-1}e^{-x\left(\frac{1-\beta t}{\beta}\right)}dx\\ & = \frac{1}{\Gamma(\alpha)\beta^\alpha}\int_0^\infty x^{\alpha-1}e^{-x/\left(\frac{\beta}{1-\beta t}\right)}dx\\ & = \frac{1}{\Gamma(\alpha)\beta^\alpha}\int_0^\infty g(x)dx \end{align*}\)

When we rewrite it this way, the term under the integral, \(g(x)\), looks almost like a Gamma density function with parameters \(\alpha\) and \(\beta^*=\dfrac{\beta}{1-\beta t}\). The only issues that we need to take care of are the constants in front of the integral and the constants to make \(g(x)\) a gamma density function.

To make \(g(x)\) a Gamma density function, we need the constants. Therefore, we need a \(\dfrac{1}{\Gamma(\alpha)}\) term and a \(\dfrac{1}{(\beta^*)^\alpha}\) term. We already have the first term, so lets rewrite the function.

\(M_X(t)=\frac{1}{\beta^\alpha}\int_0^\infty \frac{1}{\Gamma(\alpha)}x^{\alpha-1}e^{-x/\left(\frac{\beta}{1-\beta t}\right)}dx\)

All that is left is the \(\dfrac{1}{(\beta^*)^\alpha}\).

\(\frac{1}{(\beta^*)^\alpha}=\frac{1}{\left(\frac{\beta}{1-\beta t}\right)^\alpha}=\frac{\left(1-\beta t\right)^\alpha}{\beta^\alpha}\)

We have the denominator term already. Lets rewrite just for clarity.

\(M_X(t)=\int_0^\infty \frac{1}{\Gamma(\alpha)\beta^\alpha}x^{\alpha-1}e^{-x/\left(\frac{\beta}{1-\beta t}\right)}dx\)

Since \(\dfrac{1}{(\beta^*)^\alpha}=\dfrac{\left(1-\beta t\right)^\alpha}{\beta^\alpha}\), we need only to include the \((1-\beta t) ^\alpha\) term. If we include the term in the integral, we have to multiply by one. Therefore,

\( \begin{align*} & M_X(t)=\left(\frac{(1-\beta t)^\alpha}{(1-\beta t)^\alpha}\right)\int_0^\infty \frac{1}{\Gamma(\alpha)\beta^\alpha}x^{\alpha-1}e^{-x/\left(\frac{\beta}{1-\beta t}\right)}dx\\ & = \left(\frac{1}{(1-\beta t)^\alpha}\right)\int_0^\infty \frac{(1-\beta t)^\alpha}{\Gamma(\alpha)\beta^\alpha}x^{\alpha-1}e^{-x/\left(\frac{\beta}{1-\beta t}\right)}dx\\ & \left(\frac{1}{(1-\beta t)^\alpha}\right)\int_0^\infty \frac{1}{\Gamma(\alpha)\left(\frac{\beta}{1-\beta t}\right)^\alpha}x^{\alpha-1}e^{-x/\left(\frac{\beta}{1-\beta t}\right)}dx\\ & \left(\frac{1}{(1-\beta t)^\alpha}\right)\int_0^\infty h(x) dx \end{align*}\)

Therefore, \(h(x)\) is now a Gamma density function with parameters \(\alpha\) and \(\beta^*=\dfrac{\beta}{1-\beta t}\). And, since \(h(x)\) is a pdf and we are integrating over the whole space, \(x\ge 0\), then \(\int_0^\infty h(x)dx=1\). If the integral is equal to 1 based on the definition of a pdf, we are left with:

\(M_X(t)=\dfrac{1}{(1-\beta t)^\alpha}\left(1\right)=\dfrac{1}{(1-\beta t)^\alpha}\)

From the notes and the text, you can see that the moment generating function calculated above is exactly what we were supposed to get.

Just to summarize what we did here. We did not actually calculate the integral. We used algebra to manipulate the function to use the definition of a pdf. I find that after practice, this method is a lot quicker for me than doing the integrals.