CS70: Jean Walrand: Lecture 23. What do we learn from observations?

Probability Basics Review Setup: I

1. Probability Basics Review I

2. Examples

Random Experiment. Flip a coin twice. Probability Space. I

3. Conditional Probability

I

Sample Space: Set of outcomes, Ω. Ω = {HH, HT , TH, TT } Probability: Pr [ω] for all ω ∈ Ω. Pr [HH] = · · · = Pr [TT ] = 1/4 1. 0 ≤ Pr [ω] ≤ 1. 2. ∑ω∈Ω Pr [ω] = 1.

I

n! ≈

 n n √ 2πn . e

  √ 4n 2n 4πn(2n/e)2n ≈√ . ≈ √ n 2 n πn [ 2πn(n/e) ]

Pr [E] ≈

50 √4 50π 2100

1 =√ ≈ .08. 50π

irb(main):006:0> (50*3.1415926)**(0.5) => 12.5331412662588 irb(main):007:0> 1/12.5331412662588 => 0.0797884567608089

Sample space: Ω = set of 100 coin tosses = {H, T }100 . |Ω| = 2 × 2 × · · · × 2 = 2100 . Uniform probability space: Pr [ω] =

1 . 2100

Event E = “100 coin tosses with exactly 50 heads” |E|? Choose 50
positions out of 100 to be heads. |E| = 100 50 . Pr [E] =

Event: A ⊆ Ω, Pr [A] = ∑ω∈Ω Pr [ω]. Pr [at least one H out of two tosses] = Pr [HT , TH, HH] = 3/4

Exactly 50 heads in 100 coin tosses.

Calculation. Stirling formula (for large n):

Exactly 50 heads in 100 coin tosses.




100 50 2100

.

Probability of even number of heads in 57 coin tosses Ω = 57 coin tosses. |Ω| = 257 .

Let E = {ω ∈ Ω | number of Hs in ω is even}. Fact: P(E) = 1/2.

Proof: Consider the correspondence: HA1 A2 · · · A56 ↔ TA1 A2 · · · A56 Here, A1 , . . . , A56 are 56 coin flips. It matches every even sequence to an odd sequence, and conversely. Hence, there are exactly as many odd as even sequences. ⇒ |E| = |Ω|/2 ⇒ P(E) =

|E| 1 = |Ω| 2

Probability more heads than tails in 100 coin tosses.

Ω = 100 coin tosses. |Ω| = 2100 .

Ω = 100 coin tosses. |Ω| = 2100 .

Recall event E = ‘equal heads and tails’ Event F = ‘more heads than tails’ Event G = ‘more tails than heads’ A 1-to-1 correspondence between outcomes in F and G! |F | = |G|. E, F and G are disjoint. Pr [E] ≈ 8%. |Ω| = |E| + |F | + |G|. ⇒ 1 = Pr [Ω] = Pr [E] + 2Pr [F ] ≈ 8% + 2Pr [F ]. Solve for |F |:

Probability of n heads in 100 coin tosses.

Event En = ‘n heads’; |En | =

Roll a red and a blue die.

Ω = {(a, b) : 1 ≤ a, b ≤ 6} = {1, 2, . . . , 6}2 .




100 n

Uniform: Pr [ω] =


100 |En | n = 100 |Ω| 2

Inclusion/Exclusion Note that,

whether or not the sample space has uniform distribution.

E1 ∪ E2 = ‘At least one die shows 6’ 6 6 11 Pr [E1 ] = , Pr [E2 ] = , Pr [E1 ∪ E2 ] = . 36 36 36

1 36

for all ω.

2. at least one die showing 6? E1 = {(6, b) : 1 ≤ b ≤ 6} = red die shows 6 E2 = {(a, 6) : 1 ≤ a ≤ 6} = blue die shows 6 E = E1 ∪ E2 = red or blue die (or both) show 6 |E| = |E1 | + |E2 | − |E1 ∩ E2 | [Inclusion/Exclusion] |E1 ∩ E2 | = {(6, 6)} |E| = 6 + 6 − 1 = 11 Pr [E] = 11/36

Roll a red and a blue die. Ω = {(a, b) : 1 ≤ a, b ≤ 6} = {1, 2, . . . , 6}2 .

Pr [A ∪ B] = Pr [A] + Pr [B] − Pr [A ∩ B]

E1 = ‘Red die shows 6’; E2 = ‘Blue die shows 6’

=

1. the red die showing 6? E1 = {(6, b) : 1 ≤ b ≤ 6}, |E1 | = 6, Pr [E1 ] = |E1 |/|Ω| = 1/6

n

pn = Pr [En ] =

1 |Ω|

What is the probability of

pn

|F | ≈ 46%

Roll a red and a blue die.

1 1 Uniform: Pr [ω] = |Ω| = 36 for all ω. What is the probability of

1. the dice sum to 7? E = {(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)}; |E| = 6. Counting argument: for each choice a of the value of the red die, there is exactly one choice b = 7 − a for the blue die, so there are 6 total choices. Pr [E] = |E|/|Ω| = 6/36 = 1/6.

2. the dice sum to 10? E = {(4, 6), (5, 5), (6, 4)} Pr [E] = |E|/|Ω| = 3/36 = 1/12.

Roll a red and a blue die.

Roll two blue dice.

Roll two blue dice

The key idea is that we do not distinguish the dice. Roll die 1, then die 2. Then forget the order. For instance, we consider that (2, 5) and (5, 2) are the same outcome. We designate this outcome by (2, 5). Thus,

Ω0 = {(a, b) | 1 ≤ a ≤ b ≤ 6}.

We see that Pr [(1, 3)] =

2 36

and Pr [(2, 2)] =

1 36 .

Two different models of the same random experiment. In Ω0 , Pr [(1, 3)] =

Roll two blue dice. Now what is the probability of at least one die showing 6?

Roll two blue dice.

What is the probability of the dice sum to 7?

Really not uniform! and not finite! I

Experiment: Toss three times a coin with Pr [H] = 2/3. I I

I

I I

11 36 ;

2 1 in Ω0 , Pr [B] = 5 × 36 + 1 × 36 .

In Ω, Pr [A] =

6 36 ;

2 in Ω0 , Pr [B] = 3 × 36 .

Of course, this is the same as for distinguishable dice!

Of course, this is the same as for distinguishable dice!

The event does not depend on the dice being distinguishable.

The event does not depend on the dice being distinguishable.

Ω = {HHH, HHT , HTH, HTT , THH, THT , TTH, TTT }. Pr [HHH] = ( 23 )3 ; Pr [HHT ] = ( 23 )2 ( 13 ); . . .

Toss a fair coin until you get a heads. I

In Ω, Pr [A] =

2 1 and Pr [(2, 2)] = 36 36

Ω = {H, TH, TTH, TTTH, . . .} Pr [H] = 12 , Pr [TH] = 14 , Pr [TTH] = 18 1 + · · · = 1. Still sums to 1. Indeed 12 + 14 + 18 + 16

Set notation review

Conditional probability: example. ⌦

⌦

A[B

B

A

Figure : Two events

⌦

⌦

Figure : Union (or)

A\B

New sample space: A; uniform still.

⌦

A\B

Event B = two heads. Figure : Symmetric difference (only one)

Conditional Probability: A non-uniform example Consider Ω = {1, 2, . . . , N} with Pr [n] = pn .

Pr [3|B] =

The probability of two heads if the first flip is heads. The probability of B given A is 1/2.

Another non-uniform example Consider Ω = {1, 2, . . . , N} with Pr [n] = pn .

p3 p3 = . p1 + p2 + p3 Pr [B]

ω∈ / B ⇒ Pr [ω|B] = 0.

Ω = {HH, HT , TH, TT }; uniform. Event A = one flip is heads. A = {HH, HT , TH}.

New sample space: A; uniform still.

A B

A¯

Figure : Intersection (and)

Two coin flips. One of the flips is heads. Probability of two heads?

Figure : Difference (A, not B)

⌦

Figure : Complement (not)

Two coin flips. First flip is heads. Probability of two heads? Ω = {HH, HT , TH, TT }; Uniform probability space. Event A = first flip is heads: A = {HH, HT }.

A similar example.

Pr [A|B] =

p3 Pr [A ∩ B] = . p1 + p2 + p3 Pr [B]

Event B = two heads. The probability of two heads if at least one flip is heads. The probability of B given A is 1/3.

Yet another non-uniform example Consider Ω = {1, 2, . . . , N} with Pr [n] = pn .

Pr [A|B] =

p2 + p3 Pr [A ∩ B] = . p1 + p2 + p3 Pr [B]

Conditional Probability.

Conditional Probability.

Conditional Probability. Definition: The conditional probability of B given A is Pr [B|A] =

Definition: The conditional probability of B given A is If in A, what is the probability of outcome ω? If ω 6∈ A, probability is 0 Otherwise: Ratio of probability of ω to total probability of A Pr [ω|A] =

Pr [ω] Pr [A]

Uniform Probability Space: Ratio of 1/|Ω| to |A|/|Ω| =⇒ 1/|A|. (Makes sense!) What do we learn from observations? You observe that the event B occurs. That changes your information about the probability of every event A. The conditional probability of A given B is Pr [A|B] =

Pr [A ∩ B] . Pr [B]

Note: Pr [A ∩ B] = Pr [B] × Pr [A|B] = Pr [A] × Pr [B|A].

Pr [B|A] =

∑ Pr [ω|A] =

ω∈B

A∩B

∑ω∈A∩B Pr [ω] Pr [A ∩ B] = Pr [A] Pr [A] A

B

Pr [A ∩ B] Pr [A]

In A! In B? Must be in A ∩ B. Pr [B|A] =

Pr [A∩B] Pr [A] .