Outline. Why Study Probability Theory? CPSC 531: Basic Probability Concepts

CPSC 531: Basic Probability Concepts Instructor: Anirban Mahanti Office: ICT 745 Email: [email protected] References: 1. “Introduction to Prob...
5 downloads 2 Views 122KB Size
CPSC 531: Basic Probability Concepts Instructor: Anirban Mahanti Office: ICT 745 Email: [email protected]

References: 1. “Introduction to Probability Models” by Sheldon Ross, Academic Press, Eight Edition, 2003. 2. “Probability and Statistics” by M. DeGroot and M. Schervish, Third edition, Addison Wesley, 2002. 3. “Probability and Random Processes for Electrical and Computer Engineers” by John Gubner, Cambridge Press, First Edition, 2006. Basic Probability Concepts

1

Outline ˆ Experiments, sample space, and events ˆ Review of set theory ˆ Probability: definition, property,

interpretation

ˆ Conditional probability ˆ Independent events ˆ Law of total probability and Bayes’ ˆ Counting Methods

Basic Probability Concepts

2

Why Study Probability Theory? ˆ Probability theory is useful in understanding,

studying, and analyzing complex real world systems ˆ Probability theory can be used to model and develop complex real world systems ˆ A good understanding of probability theory is needed to develop simulations 

E.g., How would you model network traffic? • Should we use Markovian models? • What is the impact of “self similarity” on design and analysis of communication networks

Basic Probability Concepts

3

1

Experiment ˆ Experiment – in probability theory refers to a

process whose outcome is not known in advance with certainty E.g., Suppose a coin is tossed 10 times. How many times will we get “heads”?  Experience tells us that, if the coin is fair, we will see on average 5 “heads”; we can arrive at this result by performing the experiment many times and noting down the observations  Instead of performing the experiments, we can use probability theory to develop a model of any system that yields uncertain measurements 

Basic Probability Concepts

4

Sample space, Outcome, Event ˆ Sample space (S) – is the collection of all

possible outcomes of a system or experiment. E.g., Rolling a six-sided dice can be modeled with the sample space S = {1, 2, 3, 4, 5, 6}  E.g., Model of the number of packets queued at a router that can buffer up to 50 packets would use the sample space S = {0, 1, 2, …, 50} 

ˆ Outcome – is element in the sample space  E.g., 2, 4, and 5 are outcomes in the rolling dice example

ˆ Event – is a subset of the sample space  E.g., A is an event that an even number is rolled: A = {2, 4, 6} Basic Probability Concepts

5

Review of Set Theory

Basic Probability Concepts

6

2

Countable vs. Uncountable Sets ˆ Some sets contain only finite number of

elements, while others contain infinitely many elements. An infinite set can be classified as: 

Countable - if there is a one-to-one correspondence between elements of the set and the set of natural numbers • E.g., the set of numbers from the roll of a dice • E.g., the set of all odd numbers • E.g., the set of all prime numbers



Uncountable – converse of countable; e.g., set of real numbers, numbers in the interval [0, 1] • E.g., set of possible temperatures on Jan 1 in a city Basic Probability Concepts

7

Set Operations (1) ˆ Consider a set of points: S ˆ Let A and B be two collection of points in S ˆ Every point in A is also in B, then A ⊂ B  If A ⊂ B and B ⊂ A, then A = B ˆ Empty set: Ø is the “null” set ˆ Complement: AC is the complement of set A

and contains all outcomes in the sample space S that do not belong to A

Basic Probability Concepts

8

Set Operations (2) ˆ Union: A ∪ B = a set that consists of points

that belong either to A or to B, or to both 

A ∪ B := {w Є S: w Є A or w Є B}

ˆ Intersection: A ∩ B = a set that consists of

points that belongs to both A and B 

A ∩ B := {w Є S: w Є A and w Є B}

ˆ Two sets are said to be disjoint or mutually

exclusive if A ∩ B = Ø

ˆ Two sets are said to be mutually exclusive and

exhaustive if A ∪ B = S and A ∩ B = Ø

Basic Probability Concepts

9

3

Set Identities (1) ˆ Commutative laws A ∪B =B ∪A A ∩B =B ∩A ˆ Distributive laws  A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C)  A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C) ˆ Associative laws  (A ∪ B) ∪ C = A ∪ (B ∪ C) = (A ∪ B) ∪ C  (A ∩ B) ∩ C = A ∩ (B ∩ C) = (A ∩ B) ∩ C Basic Probability Concepts

10

Basic Probability Concepts

11

Set Identities (2) ˆ Identity laws A ∪Ø = A A ∩U =A A ∪U =U A ∩Ø = Ø ˆ De Morgan’s laws  (A ∪ B)C = AC ∩ BC  (A ∩ B)C = AC ∪ BC

Definitions of Probability

Basic Probability Concepts

12

4

Classical Definition ˆ In the classical approach, the probability of an

event P(A) is assigned a priori without performing any experiments.

ˆ The probability of an event A is the ratio of

the number of outcomes NA favorable to event A to the total number N of possible outcomes of the experiment. P (A) =

NA N Basic Probability Concepts

13

Relative Frequency Definition ˆ Suppose n trials of an experiment are

undertaken. In these trials, an event A occurs nA times. The probability of event A, P(A), can be defined as follows:

P (A) = lim

n→∞

nA n

ˆ Note that the probability is assigned after (a

posterior) the experiments.

Basic Probability Concepts

14

Axiomatic Definition ˆ A probability measure, or simply a probability,

on a sample space S is a specification of numbers P(.) that satisfies the following axioms:

1) 2) 3)

0 ≤ P(A) ≤ 1 for every event A; probability of an event A is a number between 0 and 1. P(S) = 1; the probability of an outcome being a sample point in the sample space is 1. For every infinite sequence of mutually exclusive events A1, A2, … ⎛ ∞ ⎞ ∞ P ⎜⎜ U Ai ⎟⎟ = ∑ P(A i ) ⎝ i =1 ⎠ i =1 Basic Probability Concepts

15

5

Probability Models (1) ˆ Probability theory helps develop mathematical

models of complicated problems 

Example: Develop a mathematical model of rolling a fair die to determine “probability” of an even face. Let S = {1, 2, 3, 4, 5, 6} be the sample space. Let Ei = {i}, i = 1, 2, …, 6 be the event that die rolled the face i. Let E = {2, 4, 6} model the die rolling an even face. Here the classical definition of probability is used. P(E) = |E|/|S| = 1/2 Basic Probability Concepts

16

Probability Models (2) 

Some observations about the probability model: • • • •



P(Ø) = |Ø|/|S| = 0. P(A) ≥ 0 for every event A. If A and B are disjoint events, then P(A ∪ B) = P(A) + P(B) When the die is rolled, something happens; this is represented by the fact that P(S) = 1.

Our example considered the countable finite case. What about countable infinite outcomes? • Let S = {1, 2, …}. For w Є S, let p(w) be non-negative and ∞

∑ p( w) = 1.

w =1

For any A ⊂ S, P(A) = ∑ p(w).

Basic Probability Concepts

17

Probability Models (3) 

In many cases, the number of possible outcomes are uncountable. E.g., the duration of Web browsing sessions, the duration of a phone call, the heights of adults in a city etc. We can define P as follows:

P (A) =

Z

f (w)dw, A

A⊂S

for some non-negative function f such that:

∫ f ( w)dw = 1. S

Basic Probability Concepts

18

6

Properties of Probability ˆ (Complement) For any event A, P(AC) = 1 – P(A) ˆ (Impossible Set) P(Ø) = 0

ˆ (Monotonicity rule) If A ⊂ B, P(A) ≤ P(B)

ˆ (Inclusion-Exclusion rule) Given two events A

and B, P(A U B) = P(A) + P(B) – P(A ∩ B).

ˆ Other rules can be derived. The inclusion-

exclusion rule can be extended as follows: Suppose A, B, and C are events in S. Then

P (A∪B∪C) = P (A)+P (B)+P (C)−P (A∩B)−P (A∩C)−P (B∩C)+P (A∩B∩C)

Basic Probability Concepts

19

Example (1) ˆ Select one ball from a box containing white

(W), red (R), blue (B), and green (G) balls. Suppose that P(R) = 0.1 and P(B) = 0.5. What is the probability of selecting a white or a green ball? ˆ Solution: WURUBUG=S From axiom 3, we have P(S) = P(W) + P(R) + P(B) + P(G) Also, P(WUG) = P(W)+P(G) = 1 – P(R) – P(B) = 0.4

Basic Probability Concepts

20

Example (2) ˆ Rolling a fair dice: P(1) = P(2) = P(3) = P(4) =

P(5) = P(6) = 1/6, where P(i) is the probability of rolling a face with i dots. ˆ Question:  

P({1, 3}) = ? P({2, 4, 6}) = ?

Basic Probability Concepts

21

7

Conditional Probability

Basic Probability Concepts

22

Conditional Probability (1) ˆ The conditional probability of event A given that

event B has occurred is P( A | B) = 

P( A ∩ B) P( B)

The conditional probability P(A|B) is undefined if P(B)=0

ˆ The frequency interpretation: If an experiment is

repeated a large number of times, then event B occurs approximately P(B) fraction of the time and the event A and B both occur approximately P(A ∩ B) portion of time. Therefore, among the repetitions in which B occurs, the proportion of events in which A also occurs is P(A ∩ B) /P(B). P(A ∩ B) also denoted as P(AB).

Basic Probability Concepts

23

Conditional Probability (2) ˆ Example: Two balls are selected at random, without

replacement, from a bag containing r red balls and b blue balls. Determine probability p that the first selected ball is red and the second selected ball is blue. Solution A = event that first ball is red; B = event that second ball is blue P(A) = prob. first ball is red = r/(r+b) P(B|A) = prob. blue in second draw = b/(r+b-1) P(B|A) = P(AB)/P(A) P(AB) = (r.b)/((r+b)(r+b-1)) Basic Probability Concepts

24

8

Conditional Probability (3) ˆ P(AB) = P(A) P(A|B); extended, as shown next ˆ Suppose that A1, A2, A3, …, An are events such that

P(B) > 0 and P(A1A2A3 …An-1) > 0. Then,

P(A1A2A3…An) = P(A1)P(A2|A1) P(A3|A1A2)… P(An|A1A2A3 …An-1)

ˆ Example: Draw 4 balls, one by one, without

replacement, from a box containing r red balls and b blue balls. Probability of sequence red, red, blue, blue? Solution: Let Rj and Bj denote event that a red and blue ball is drawn, respectively. P(R1R2B3B4) = P(R1)P(R2|R1)P(B3|R1R2) P(B4|R1R2B3) complete the calculations on your own … Basic Probability Concepts

25

Independent Events (1) ˆ Independent events are those that don’t have

any effect on each other. That is, knowing one of them occurs does not provide any information about the occurrence of the other event. Mathematically, A and B are independent if P(A|B) = P(A) and P(B|A) = P(B)

ˆ From conditional probability definition, we

have P(AB) = P(A|B)P(B). Therefore, if A and B are independent events, P(AB) = P(A)P(B) Basic Probability Concepts

26

Independent Events (2) ˆ Example: Determine the probability pn that

exactly n tosses are required for a head to appear for the first time.

Solution We need to determine the probability of a sequence of (n-1) tails followed by a head. The probability of obtaining a heads or tails (for a fair coin) is ½. Therefore, pn = (½)n.

Basic Probability Concepts

27

9

Law of Total Probability and Bayes’ Rule

Basic Probability Concepts

28

Law of Total Probability ˆ Let B1, B2, B3, …, Bk be mutually disjoint and collectively

exhaustive events from the sample space S. Then, for any event A in S, we have k

P( A) = ∑ P( B j ) P( A | B j )

Explanation A = (B1A)U (B2A)U… (BkA). The (BjA)’s are disjoint events. Therefore, using laws of conditional probability we get: k

k

j =1

j =1

P ( A) = ∑ P ( B j A) = ∑ P( B j ) P ( A | B j ),

j =1

B1

B2

A

B3

B4

if P ( B j ) > 0 for j = 1,..., k Basic Probability Concepts

29

Bayes’ Theorem ˆ Partition: The events B1, B2, B3, …, Bk form a

partition of a set S if they are mutually disjoint and Uik=1 Bi = S

ˆ Bayes’ Theorem: Suppose that B1, B2, B3, …, Bk

form a partition of sample space S such that P(Bj)> 0 for j = 1, …, k. Let A be an event in S such that P(A)>0. Then, for i = 1, …, k, P( Bi | A) =

P( Bi ) P( A | Bi ) ∑ kj =1 P( B j ) P( A | B j ) Basic Probability Concepts

30

10

Counting Methods

Basic Probability Concepts

31

Types of Counting Problems ˆ Two types of counting problems:  Selection problems – select k balls from an urn containing n distinguishable balls  Allocation problems – allocate k indistinguishable balls to a set of n distinguishable urns ˆ Consider the selection problem  How do we replace selected balls? • With replacement – put ball back into urn • Without replacement 

How to count events? • Permutation – order of selection important • Combination – order of selection unimportant Basic Probability Concepts

32

Permutations (1) ˆ The number of ordered ways k items can be

selected without replacement from a pool of n items is: Pn,k = n x (n-1) x … X (n-k+1) = n!/(n-k)!

ˆ The number of ordered ways k items can be

selected with replacement from a pool of n items is: Prn,k = n x n x … (k times) = nk

Basic Probability Concepts

33

11

Permutations (2) ˆ When not all items are ordered: E.g., a club has

25 members and 2 officers (a president and a secretary). These officers are to be chosen from the members of the club. How many ways can the officers be selected? 

25 ways to select the first officer and 24 ways to select the second officer. Answer = 25x24 = 600

Basic Probability Concepts

34

Permutations (3) ˆ What is the probability that at least two

people share the same birthday? You may assume the following: 365 possible birthdays (ignore leap years) Our group consists of 2 ≤ k ≤ 365 people who are unrelated (no twins)  Each birthday is equally likely (birth rate independent of the time of year)  

Basic Probability Concepts

35

Combinations (1) ˆ The number of combinations possible when

selecting k items from a set of n items without replacement is denoted by Cn,k. ˆ Note that one way to find Pn,k is to compute Cn,k (i.e., unordered selection of k items) and multiply by the number of ways of ordering k items (i.e., k!). 

Therefore, Pn,k = Cn,k x k!

Cn , k =

Pn, k k!

=

n! k!(n − k )! Basic Probability Concepts

36

12

Combinations (2) ˆ The number of combinations possible when

selecting k items from a set of n items with replacement is denoted by Crn,k. Crn,k = Cn+k-1,k.

Basic Probability Concepts

37

Combinations (3) ˆ Example: A fair coin is tossed 10 times. Determine

the probability, p, of obtaining exactly 3 heads. Solution 

 

Each experiment has two possible outcome, heads or tails. Since the coins are fair, we can assume that each outcome is equally likely. Therefore, the total number of outcomes in this experiment is = 210 The number of different arrangements possible with 3 heads and 7 tails is = C10,3 Therefore,

p=

C10,3 210

= 0.1172

Basic Probability Concepts

38

Summary ˆ A review of basic probability and statistics

was presented. You probably enjoyed it!

ˆ Next, we will talk about discrete and

continuous random variables.

Basic Probability Concepts

39

13