Lecture 1.5

Probability Theory: An Example

Learning Objectives

After this lecture you should be able to:

Set up a joint probability problem from a verbal description, identifying priors and conditionals.
Compute a marginal distribution using the sum rule and the product rule.
Apply Bayes' theorem numerically to compute a posterior from a prior, a likelihood, and evidence.
Explain in words how observing an outcome updates a prior belief into a posterior.
State the definition of independence and verify it for a concrete example.

This lecture works through a single concrete example — picking fruits from colored boxes — to build hands-on intuition for the rules derived in lecture 1.4: marginalization, the product rule, Bayes' theorem, and independence.

1. Setup

We have two random variables:

$F \in \{\text{apple}, \text{orange}\}$ — the fruit picked.
$B \in \{\text{red}, \text{blue}\}$ — the box it came from.

Prior probabilities (which box we pick from):

$$p(B = \text{blue}) = \tfrac{6}{10}, \qquad p(B = \text{red}) = \tfrac{4}{10}$$

Conditional probabilities (fruit counts per box):

Blue box: 3 apples, 1 orange → $p(F=\text{apple} \mid B=\text{blue}) = \tfrac{3}{4}$, $\quad p(F=\text{orange} \mid B=\text{blue}) = \tfrac{1}{4}$
Red box: 2 apples, 6 oranges → $p(F=\text{apple} \mid B=\text{red}) = \tfrac{2}{8} = \tfrac{1}{4}$, $\quad p(F=\text{orange} \mid B=\text{red}) = \tfrac{6}{8} = \tfrac{3}{4}$

2. Marginal Fruit Probabilities

We want $p(F)$ — the probability of picking each fruit regardless of which box it came from. We apply the sum rule to marginalize out $B$, then expand each joint term with the product rule:

$$p(F = \text{apple}) = \sum_b p(F=\text{apple},\, B=b) = \sum_b p(F=\text{apple} \mid B=b)\, p(B=b)$$ $$= \underbrace{\tfrac{3}{4} \cdot \tfrac{6}{10}}_{\text{blue box}} + \underbrace{\tfrac{1}{4} \cdot \tfrac{4}{10}}_{\text{red box}} = \tfrac{18}{40} + \tfrac{4}{40} = \frac{11}{20}$$

For oranges we could repeat the same calculation, or use normalization directly:

$$p(F = \text{orange}) = 1 - \tfrac{11}{20} = \frac{9}{20}$$

Shortcut: whenever there are only two outcomes, normalization ($\sum_f p(f) = 1$) gives the second for free once you have the first. Computers handle the full sum; for hand calculations, use the shortcut.

3. Applying Bayes' Theorem

Now suppose we pick a fruit and see it is an orange. Which box did it most likely come from? Our prior said $p(B=\text{red}) = 40\%$, but knowing the fruit is orange should update that belief — the red box is mostly oranges.

Bayes' theorem gives the posterior:

$$p(B=\text{red} \mid F=\text{orange}) = \frac{p(F=\text{orange} \mid B=\text{red})\; p(B=\text{red})}{p(F=\text{orange})}$$

Filling in the known values:

$$= \frac{\dfrac{3}{4} \cdot \dfrac{4}{10}}{\dfrac{9}{20}} = \frac{\dfrac{12}{40}}{\dfrac{9}{20}} = \frac{12}{40} \cdot \frac{20}{9} = \frac{240}{360} = \frac{2}{3} \approx 66\%$$

Prior vs. Posterior

Before observing the fruit: $p(B=\text{red}) = 40\%$.
After observing an orange: $p(B=\text{red} \mid F=\text{orange}) \approx 66\%$.

The observation shifted our belief substantially. The red box is fruit-heavy in oranges, so seeing an orange is strong evidence for the red box. This is Bayes' theorem in action: new evidence updates a prior into a more informed posterior.

4. Independence

Definition: Independence

Two random variables $X$ and $Y$ are independent if and only if knowing one provides no information about the other. Formally:

$$p(X, Y) = p(X)\, p(Y)$$

An equivalent statement: $p(X \mid Y) = p(X)$ and $p(Y \mid X) = p(Y)$. Conditioning on an independent variable leaves the distribution unchanged.

Example: When Fruit and Box Are Independent

Modify the setup so both boxes contain 2 apples and 2 oranges (equal ratio). Then:

$$p(F=\text{apple} \mid B=\text{blue}) = \tfrac{1}{2} = p(F=\text{apple} \mid B=\text{red})$$

The fruit distribution is the same regardless of which box — the conditional equals the marginal. Observing the fruit tells you nothing about the box, and vice versa. $F$ and $B$ are independent.

In the original setup (unequal ratios), this symmetry is broken: seeing an orange pushes you toward the red box. That is precisely what makes probabilistic reasoning useful.