Loading [MathJax]/jax/element/mml/optable/MathOperators.js


5.2.3 Conditioning and Independence

Here, we will discuss conditioning for continuous random variables. In particular, we will discuss the conditional PDF, conditional CDF, and conditional expectation. We have discussed conditional probability for discrete random variables before. The ideas behind conditional probability for continuous random variables are very similar to the discrete case. The difference lies in the fact that we need to work with probability density in the case of continuous random variables. Nevertheless, we would like to emphasize again that there is only one main formula regarding conditional probability which is P(A|B)=P(AB)P(B), when P(B)>0. Any other formula regarding conditional probability can be derived from the above formula. In fact, for some problems we only need to apply the above formula. You have already used this in Example 5.17. As another example, if you have two random variables X and Y, you can write P(XC|YD)=P(XC,YD)P(YD), where C,DR. However, sometimes we need to use the concepts of conditional PDFs and CDFs. The formulas for conditional PDFs and CDFs of continuous random variables are very similar to those of discrete random variables. Since there are no new fundamental ideas in this section, we usually provide the main formulas and guidelines, and then work on examples. Specifically, we do not spend much time deriving formulas. Nevertheless, to give you the basic idea of how to derive these formulas, we start by deriving a formula for the conditional CDF and PDF of a random variable X given that XI=[a,b]. Consider a continuous random variable X. Suppose that we know that the event XI=[a,b] has occurred. Call this event A. The conditional CDF of X given A, denoted by FX|A(x) or FX|aXb(x), is FX|A(x)=P(Xx|A)=P(Xx|aXb)=P(Xx,aXb)P(A). Now if x<a, then FX|A(x)=0. On the other hand, if axb, we have FX|A(x)=P(Xx,aXb)P(A)=P(aXx)P(A)=FX(x)FX(a)FX(b)FX(a). Finally, if x>b, then FX|A(x)=1. Thus, we obtain FX|A(x)={1x>bFX(x)FX(a)FX(b)FX(a)ax<b0otherwise Note that since X is a continuous random variable, we do not need to be careful about end points, i.e., changing x>b to xb does not make a difference in the above formula. To obtain the conditional PDF of X, denoted by fX|A(x), we can differentiate FX|A(x). We obtain fX|A(x)={fX(x)P(A)ax<b0otherwise It is insightful if we derive the above formula for fX|A(x) directly from the definition of the PDF for continuous random variables. Recall that the PDF of X can be defined as fX(x)=limΔ0+P(x<Xx+Δ)Δ. Now, the conditional PDF of X given A, denoted by fX|A(x), is fX|A(x)=limΔ0+P(x<Xx+Δ|A)Δ=limΔ0+P(x<Xx+Δ,A)ΔP(A)=limΔ0+P(x<Xx+Δ,aXb)ΔP(A). Now consider two cases. If ax<b, then fX|A(x)=limΔ0+P(x<Xx+Δ,aXb)ΔP(A)=1P(A)limΔ0+P(x<Xx+Δ)Δ=fX(x)P(A). On the other hand, if x<a or xb, then fX|A(x)=limΔ0+P(x<Xx+Δ,aXb)ΔP(A)=0.
If X is a continuous random variable, and A is the event that a<X<b (where possibly b= or a=), then FX|A(x)={1x>bFX(x)FX(a)FX(b)FX(a)ax<b0x<a fX|A(x)={fX(x)P(A)ax<b0otherwise

The conditional expectation and variance are defined by replacing the PDF by conditional PDF in the definitions of expectation and variance. In general, for a random variable X and an event A, we have the following:
E[X|A]=xfX|A(x)dx,E[g(X)|A]=g(x)fX|A(x)dx,Var(X|A)=E[X2|A](E[X|A])2

Example
Let XExponential(1).
  1. Find the conditional PDF and CDF of X given X>1.
  2. Find E[X|X>1].
  3. Find Var(X|X>1).
  • Solution
      1. Let A be the event that X>1. Then P(A)=1exdx=1e. Thus, fX|X>1(x)={ex+1x>10otherwise For x>1, we have FX|A(A)=FX(x)FX(1)P(A)=1ex+1. Thus, FX|A(x)={1ex+1x>10otherwise
      2. We have E[X|X>1]=1xfX|X>1(x)dx=1xex+1dx=e1xexdx=e[exxex]1=e2e=2.
      3. We have E[X2|X>1]=1x2fX|X>1(x)dx=1x2ex+1dx=e1x2exdx=e[2ex2xexx2ex]1=e5e=5. Thus, Var(X|X>1)=E[X2|X>1](E[X|X>1])2=54=1.


Conditioning by Another Random Variable:

If X and Y are two jointly continuous random variables, and we obtain some information regarding Y, we should update the PDF and CDF of X based on the new information. In particular, if we get to observe the value of the random variable Y, then how do we need to update the PDF and CDF of X? Remember for the discrete case, the conditional PMF of X given Y=y is given by

PX|Y(xi|yj)=PXY(xi,yj)PY(yj). Now, if X and Y are jointly continuous, the conditional PDF of X given Y is given by fX|Y(x|y)=fXY(x,y)fY(y). This means that if we get to observe Y=y, then we need to use the above conditional density for the random variable X. To get an intuition about the formula, note that by definition, for small Δx and Δy we should have fX|Y(x|y)P(xXx+Δx|yYy+Δy)Δx(definition of PDF)=P(xXx+Δx,yYy+Δy)P(yYy+Δy)ΔxfXY(x,y)ΔxΔyfY(y)ΔyΔx=fXY(x,y)fY(y). Similarly, we can write the conditional PDF of Y, given X=x, as fY|X(y|x)=fXY(x,y)fX(x).
For two jointly continuous random variables X and Y, we can define the following conditional concepts:
  1. The conditional PDF of X given Y=y: fX|Y(x|y)=fXY(x,y)fY(y)
  2. The conditional probability that XA given Y=y: P(XA|Y=y)=AfX|Y(x|y)dx
  3. The conditional CDF of X given Y=y: FX|Y(x|y)=P(Xx|Y=y)=xfX|Y(x|y)dx

Example Let X and Y be two jointly continuous random variables with joint PDF fXY(x,y)={x24+y24+xy60x1,0y20otherwise For 0y2, find
  1. the conditional PDF of X given Y=y;
  2. P(X<12|Y=y).
  • Solution
      1. Let us first find the marginal PDF of Y. We have fY(y)=10x24+y24+xy6dx=3y2+y+112,for 0y2. Thus, for 0y2, we obtain fX|Y(x|y)=fXY(x,y)fY(y)=3x2+3y2+2xy3y2+y+1,for 0x1. Thus, for 0y2, we have fX|Y(x|y)={3x2+3y2+2xy3y2+y+10x10otherwise
      2. We have P(X<12|Y=y)=1203x2+3y2+2xy3y2+y+1dx=13y2+y+1[x3+yx2+3y2x]120=32y2+y4+183y2+y+1. Note that, as we expect, P(X<12|Y=y) depends on y.


Conditional expectation and variance are similarly defined. Given Y=y, we need to replace fX(x) by fX|Y(x|y) in the formulas for expectation:
For two jointly continuous random variables X and Y, we have:
  1. Expected value of X given Y=y: E[X|Y=y]=xfX|Y(x|y)dx
  2. Conditional LOTUS: E[g(X)|Y=y]=g(x)fX|Y(x|y)dx
  3. Conditional variance of X given Y=y: Var(X|Y=y)=E[X2|Y=y](E[X|Y=y])2

Example
Let X and Y be as in Example 5.21. Find E[X|Y=1] and Var(X|Y=1).
  • Solution
    • E[X|Y=1]=xfX|Y(x|1)dx=10x3x2+3y2+2xy3y2+y+1|y=1dx=10x3x2+3+2x3+1+1dx(y=1)=15103x3+2x2+3xdx=712, E[X2|Y=1]=x2fX|Y(x|1)dx=15103x4+2x3+3x2dx=2150. So we have Var(X|Y=1)=E[X2|Y=1](E[X|Y=1])2=2150(712)2=2873600.


Independent Random Variables:

When two jointly continuous random variables are independent, we must have fX|Y(x|y)=fX(x). That is, knowing the value of Y does not change the PDF of X. Since fX|Y(x|y)=fXY(x,y)fY(y), we conclude that for two independent continuous random variables we must have fXY(x,y)=fX(x)fY(y).
Two continuous random variables X and Y are independent if fXY(x,y)=fX(x)fY(y), for all x,y. Equivalently, X and Y are independent if FXY(x,y)=FX(x)FY(y), for all x,y. If X and Y are independent, we have E[XY]=EXEY,E[g(X)h(Y)]=E[g(X)]E[h(Y)].

Suppose that we are given the joint PDF fXY(x,y) of two random variables X and Y. If we can write fXY(x,y)=f1(x)f2(y), then X and Y are independent.
Example
Determine whether X and Y are independent:
  1. fXY(x,y)={2ex2yx,y>00otherwise
  2. fXY(x,y)={8xy0<x<y<10otherwise
  • Solution
      1. We can write fXY(x,y)=[exu(x)][2e2yu(y)], where u(x) is the unit step function: u(x)={1x00otherwise Thus, we conclude that X and Y are independent.
      2. For this case, it does not seem that we can write fXY(x,y) as a product of some f1(x) and f2(y). Note that the given region 0<x<y<1 enforces that x<y. That is, we always have X<Y. Thus, we conclude that X and Y are not independent. To show this, we can obtain the marginal PDFs of X and Y and show that fXY(x,y)fX(x)fY(y), for some x,y. We have, for 0x1, fX(x)=1x8xydy=4x(1x2). Thus, fX(x)={4x(1x2)0<x<10otherwise Similarly, we obtain fY(y)={4y30<y<10otherwise As we see, fXY(x,y)fX(x)fY(y), thus X and Y are NOT independent.


Example
Consider the unit disc D={(x,y)|x2+y21}. Suppose that we choose a point (X,Y) uniformly at random in D. That is, the joint PDF of X and Y is given by fXY(x,y)={c(x,y)D0otherwise
  1. Find the constant c.
  2. Find the marginal PDFs fX(x) and fY(y).
  3. Find the conditional PDF of X given Y=y, where 1y1.
  4. Are X and Y independent?
  • Solution
      1. We have 1=fXY(x,y)dxdy= Thus, c=\frac{1}{\pi}.
      2. For -1 \leq x \leq 1, we have \begin{align}%\label{} \nonumber f_X(x)&=\int_{-\infty}^{\infty} f_{XY}(x,y)dy\\ \nonumber &=\int_{-\sqrt{1-x^2}}^{\sqrt{1-x^2}} \frac{1}{\pi}\hspace{5pt} dy\\ \nonumber &=\frac{2}{\pi}\sqrt{1-x^2}. \end{align} Thus, \begin{equation} \nonumber f_X(x) = \left\{ \begin{array}{l l} \frac{2}{\pi}\sqrt{1-x^2} & \quad -1 \leq x \leq 1 \\ & \quad \\ 0 & \quad \text{otherwise} \end{array} \right. \end{equation} Similarly, \begin{equation} \nonumber f_Y(y) = \left\{ \begin{array}{l l} \frac{2}{\pi}\sqrt{1-y^2} & \quad -1 \leq y \leq 1 \\ & \quad \\ 0 & \quad \text{otherwise} \end{array} \right. \end{equation}
      3. We have \begin{align}%\label{} \nonumber f_{X|Y}(x|y)&=\frac{f_{XY}(x,y)}{f_Y(y)}\\ \nonumber &=\left\{ \begin{array}{l l} \frac{1}{2\sqrt{1-y^2}} & \quad -\sqrt{1-y^2} \leq x \leq \sqrt{1-y^2} \\ & \quad \\ 0 & \quad \text{otherwise} \end{array} \right. \end{align} Note that the above equation indicates that, given Y=y, X is uniformly distributed on [-\sqrt{1-y^2},\sqrt{1-y^2}]. We write \begin{align}%\label{} \nonumber X|Y=y \hspace{5pt} \sim \hspace{5pt} Uniform(-\sqrt{1-y^2},\sqrt{1-y^2}). \end{align}
      4. Are X and Y independent? No, because f_{XY}(x,y)\neq f_X(x) f_Y(y).


    Law of Total Probability:

    Now, we'll discuss the law of total probability for continuous random variables. This is completely analogous to the discrete case. In particular, the law of total probability, the law of total expectation (law of iterated expectations), and the law of total variance can be stated as follows:

    Law of Total Probability:

    \begin{align}\label{eq:LOTP-cont} P(A)=\int_{-\infty}^{\infty}P(A|X=x)f_X(x) \hspace{5pt} dx \hspace{20pt} (5.16) \end{align}

    Law of Total Expectation:

    \begin{align}\label{eq:LOTE-cont} \nonumber E[Y]&=\int_{-\infty}^{\infty}E[Y|X=x]f_X(x) \hspace{5pt} dx \hspace{20pt} (5.17)\\ &=E[E[Y|X]] \end{align}

    Law of Total Variance:

    \begin{align}\label{eq:LOTV-cont} \textrm{Var}(Y)=E[\textrm{Var}(Y|X)]+\textrm{Var}(E[Y|X]) \hspace{20pt} (5.18) \end{align}
    Let's look at some examples.
    Example
    Let X and Y be two independent Uniform(0,1) random variables. Find P(X^3+Y>1).
    • Solution
      • Using the law of total probability (Equation 5.16), we can write \begin{align}%\label{} \nonumber P(X^3+Y>1)&= \int_{-\infty}^{\infty}P(X^3+Y>1|X=x)f_X(x) \hspace{5pt} dx\\ \nonumber &=\int_{0}^{1}P(x^3+Y>1|X=x) \hspace{5pt} dx\\ \nonumber &=\int_{0}^{1}P(Y>1-x^3) \hspace{5pt} dx &\textrm{(since $X$ and $Y$ are independent)} \\ \nonumber &=\int_{0}^{1}x^3 \hspace{5pt} dx &\textrm{(since $Y \sim Uniform(0,1)$) }\\ \nonumber &=\frac{1}{4}. \end{align}


    Example
    Suppose X \sim Uniform(1,2) and given X=x, Y is an exponential random variable with parameter \lambda=x, so we can write \begin{align}%\label{} \nonumber Y|X=x \hspace{10pt} \sim \hspace{10pt} Exponential(x). \end{align} We sometimes write this as \begin{align}%\label{} \nonumber Y|X \hspace{10pt} \sim \hspace{10pt} Exponential(X). \end{align}
    1. Find EY.
    2. Find Var(Y).
    • Solution
        1. We use the law of total expectation (Equation 5.17) to find EY. Remember that if Y \sim Exponential(\lambda), then EY=\frac{1}{\lambda}. Thus we conclude \begin{align} \nonumber E[Y|X=x]=\frac{1}{x}. \end{align} Using the law of total expectation, we have \begin{align} \nonumber EY&=\int_{-\infty}^{\infty} E[Y|X=x] f_X(x)dx\\ \nonumber &=\int_{1}^{2} E[Y|X=x] \cdot 1 dx\\ \nonumber &=\int_{1}^{2}\frac{1}{x}dx\\ \nonumber &= \ln 2. \end{align} Another way to write the above calculation is \begin{align} \nonumber EY&=E[E[Y|X]] &(\textrm{law of total expectation})\\ \nonumber &=E\left[\frac{1}{X}\right] &(\textrm{since } E[Y|X]=\frac{1}{X})\\ \nonumber &=\int_{1}^{2}\frac{1}{x}dx\\ \nonumber &= \ln 2. \end{align}
        2. To find Var(Y), we can write \begin{align} \nonumber Var(Y)&=E[Y^2]-(E[Y])^2\\ &=E[Y^2]-(\ln 2)^2\\ &=E\big[E[Y^2|X]\big]-(\ln 2)^2 & (\textrm{law of total expectation})\\ &=E\left[\frac{2}{X^2}\right]-(\ln 2)^2 & \big(\textrm{since } Y|X \sim Exponential(X)\big)\\ &=\int_{1}^{2}\frac{2}{x^2}dx-(\ln 2)^2 \\ &=1-(\ln 2)^2. \end{align} Another way to find Var(Y) is to apply the law of total variance: \begin{align} \textrm{Var}(Y)=E[\textrm{Var}(Y|X)]+\textrm{Var}(E[Y|X]). \end{align} Since Y|X \sim Exponential(X), we conclude \begin{align} &E[Y|X]=\frac{1}{X},\\ &Var(Y|X)=\frac{1}{X^2}. \end{align} Therefore \begin{align} \textrm{Var}(Y)&=E\left[\frac{1}{X^2}\right]+\textrm{Var}\left(\frac{1}{X}\right)\\ &=E\left[\frac{1}{X^2}\right]+E\left[\frac{1}{X^2}\right]-\left(E\left[\frac{1}{X}\right]\right)^2\\ &=E\left[\frac{2}{X^2}\right]-(\ln 2)^2\\ &=1-(\ln 2)^2. \end{align}




    The print version of the book is available on Amazon.

    Book Cover


    Practical uncertainty: Useful Ideas in Decision-Making, Risk, Randomness, & AI

    ractical Uncertaintly Cover