3. Probability Rules ¶
Below are the three commonly used probability rules.
Product Rule¶
Given two propositions $A$ and $B$, we have
$$
P(A,B) = P(A) * P(B | A).
$$
The product rule can be explained by establishing the beliefs of $A$ and $B$ in two different ways: parallel and sequential.
- Parallel: The probability that both $A$ and $B$ are true is directly denoted as $P(A,B)$ by definition.
- Sequential: First, we establish our belief for $A$ alone. The probability that $A$ is true is denoted as $P(A)$. Then, we establish our belief for $B$ given that $A$ is true. This conditional probability is $P(B|A)$. Combining these two beliefs together, the probability that both $A$ and $B$ are true is equivalent to "$A$ is true" $\mathsf{AND}$ "$B$ is true given $A$ is true", which is $P(A) * P(B|A)$.
Since we will reach the same belief through either parallel or sequential way, we have $P(A,B) = P(A) * P(B | A)$.
In the sequential way, we can also establish our belief for $B$ first, and then $A$ given $B$. Then we can obtain $P(A,B) = P(B) * P(A | B)$. In other words, the product rule works for any order of random variables.
We can further extend the product rule from two propositions to any number of propositions. Given a list of propositions $A_1, A_2, \dots, A_n$, we have
$$
P(A_1, A_2, \dots, A_n) = P(A_1) * P(A_2 | A_1) * \dots * P(A_n | A_1, A_2, \dots, A_{n-1}).
$$
This is the famous chain rule) for probability.
Sum Rule¶
Given two discrete random variables $X$ and $Y$, with their domains as $\Omega(X)$ and $\Omega(Y)$, then for each $x \in \Omega(X)$, we have
$$
P(X = x) = \sum_{y \in \Omega(Y)} P(X = x, Y = y).
$$
In other words, if we jointly consider a random variable $X$ with another random variable $Y$, then the marginal probability $P(X = x)$ is equivalent to the sum of its joint probabilities over all the possible $Y$ values.
For example, if we consider tomorrow's weather $X_{t+1}$ jointly with today's weather $X_t$, where there are three possible weathers $\{sunny, rainy, cloudy\}$, then we have
$$
\begin{aligned}
P(X_{t+1} = sunny) & = P(X_{t+1} = sunny, X_t = sunny) \\
& + P(X_{t+1} = sunny, X_t = rainy) \\
& + P(X_{t+1} = sunny, X_t = cloudy)
\end{aligned}
$$$$
\begin{aligned}
P(X_{t+1} = rainy) & = P(X_{t+1} = rainy, X_t = sunny) \\
& + P(X_{t+1} = rainy, X_t = rainy) \\
& + P(X_{t+1} = rainy, X_t = cloudy)
\end{aligned}
$$$$
\begin{aligned}
P(X_{t+1} = cloudy) & = P(X_{t+1} = cloudy, X_t = sunny) \\
& + P(X_{t+1} = cloudy, X_t = rainy) \\
& + P(X_{t+1} = cloudy, X_t = cloudy)
\end{aligned}
$$
We can further extend the sum rule from two random variables to any number of variables. Given random variables $X_1, X_2, \dots, X_m$ and $Y_1, Y_2, \dots, Y_n$, then for each $x_1 \in \Omega(X_1), x_2 \in \Omega(X_2), \dots x_m \in \Omega(X_m)$, we have
$$
P(X_1 = x_1, \dots, X_m = x_m) = \sum_{y_i \in \Omega(Y_i), i = 1, \dots, n} P(X_1 = x_1, \dots, X_m = x_m, Y_1 = y_1, \dots, Y_n = y_n).
$$
Normalisation Rule¶
Given a discrete random variable $X$ with its domain $\Omega(X)$, we have
$$
\sum_{x = \Omega(X)} P(X) = 1.
$$
It is obvious that the random variable must take one of the values in its domain. Therefore, the probabilities of all the possible values should add up to 1 (100%).
The normalisation rule holds under any condition as well. Given any conditional proposition $A$, we have
$$
\sum_{x = \Omega(X)} P(X | A) = 1.
$$
In the above weather forecast example, the normalisation rule without condition can be written as:
$$
P(X_{t+1} = sunny) + P(X_{t+1} = rainy) + P(X_{t+1} = cloudy) = 1,
$$
Similarly, the normalisation rule under the condition $X_t = sunny$ can be written as:
$$
P(X_{t+1} = sunny | X_t = sunny) + P(X_{t+1} = rainy | X_t = sunny) + P(X_{t+1} = cloudy | X_t = sunny) = 1,
$$
