Introduction to Artificial Intelligence Marc Toussaint April 7, 2015
The majority of slides of the earlier parts are adapted from Stuart Russell.
This is a direct concatenation and reformatting of all lecture slides and exercises from the Artificial Intelligence course (winter term 2014/15, U Stuttgart), including indexing to help prepare for exams. sequential decisions
deterministic
sequential decision problems
propositional
sequential assignment
MCTS
fwd/bwd chaining
bandits
UCB
graphical models
MDPs Decision Theory
probabilistic
on trees
minimax
utilities
multi-agent MDPs
FOL
constraint propagation
alpha/beta pruning
games
propositional logic
CSP
backtracking search BFS
relational
dynamic programming V(s), Q(s,a)
FOL belief propagation msg. passing
relational graphical models
HMMs relational MDPs
fwd/bwd msg. passing ML
learning
Reinforcement Learning Active Learning
Contents 1 Introduction
4
3 Search
5
Example: Romania (1:2) Example: Vacuum World (1:5) Problem Definition: Deterministic, fully observable (1:9) Example: The 8-Puzzle (1:15)
3.1 Tree Search Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
Tree search implementation: states vs nodes (1:25) Tree Search: General Algorithm (1:26) Breadth-first search (BFS) (1:29) Complexity of BFS (1:37) Uniform-cost search (1:38) Depth-first search (DFS) (1:39) Complexity of DFS (1:52) Iterative deepening search (1:54) Complexity of Iterative Deepening Search (1:63) Graph search and repeated states (1:65)
4 Informed search algorithms
13
Best-first Search (2:3) Greedy Search (2:5) Complexity of Greedy Search (2:14) A∗ search (2:15) A∗ : Proof 1 of Optimality (2:22) Complexity of A∗ (2:27) A∗ : Proof 2 of Optimality (2:28) Admissible heuristics (2:30) Memorybounded A∗ (2:34)
5 Constraint Satisfaction Problems
17
Constraint satisfaction problems (CSPs): Definition (3:2) Map-Coloring Problem (3:3)
5.1 Methods for solving CSPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Backtracking (3:10) Variable order: Minimum remaining values (3:18) Variable order: Degree heuristic (3:19) Value order: Least constraining value (3:20) Forward checking (3:21) Constraint propagation (3:25) Treestructured CSPs (3:33)
1
18
2
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
6 Optimization
22
Optimization problem: Definition (4:2) Local Search (4:5) Travelling Salesman Problem (TSP) (4:6) Local optima, plateaus (4:8) Iterated Local Search (4:9) Simulated Annealing (4:11) Genetic Algorithms (4:14)
6.1 A glimpse at general optimization problems . . . . . . . . . . . . . . . . . . . . . .
24
LP, QP, ILP, NLP (4:17) Slack Variables (4:19) n-queens as ILP (4:19) TSP as ILP (4:20) CSP as ILP (4:21)
7 Propositional Logic
26
Knowledge base: Definition (5:2) Wumpus World example (5:4) Logics: Definition, Syntax, Semantics (5:20) Entailment (5:21) Model (5:22) Inference (5:28) Propositional logic: Syntax (5:29) Propositional logic: Semantics (5:31) Logical equivalence (5:37) Validity (5:38) Satisfiability (5:38) Horn Form (5:40) Modus Ponens (5:40) Forward chaining (5:41) Completeness of Forward Chaining (5:51) Backward Chaining (5:52) Conjunctive Normal Form (5:64) Resolution (5:64) Conversion to CNF (5:65)
8 First Order Logic
37
FOL: Syntax (6:5) Universal quantification (6:7) Existential quantification (6:8)
8.1 FOL description of interactive domains . . . . . . . . . . . . . . . . . . . . . . . . .
38
Situation Calculus (6:21) Frame problem (6:22) Planning Domain Definition Language (PDDL) (6:24)
9 First Order Logic – Inference
41
Reduction to propositional inference (7:6) Unification (7:9) Generalized Modus Ponens (7:14) Forward Chaining (7:15) Backward Chaining (7:17) Resolution (7:19) Conversion to CNF (7:20)
10 Probabilities
44
Probabilities as (subjective) information calculus (8:2) Frequentist vs Bayesian (8:4)
10.1 Basic definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44
Definitions based on sets (8:6) Random variables (8:7) Probability distribution (8:8) Joint distribution (8:9) Marginal (8:9) Conditional distribution (8:9) Bayes’ Theorem (8:11) Multiple RVs, conditional independence (8:12)
10.2 Probability distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
45
Bernoulli and Binomial (8:14) Beta (8:15) Multinomial (8:18) Dirichlet (8:19) Conjugate priors (8:23) Dirac (8:26) Gaussian (8:27) Particle approximation of a distribution (8:29) Utilities and Decision Theory (8:32) Entropy (8:33) Kullback-Leibler divergence (8:34)
11 Bandits & UCT
50
Multi-armed Bandits (9:1) Exploration, Exploitation (9:6) Upper Confidence Bound (UCB) (9:8) Monte Carlo Tree Search (MCTS) (9:14) Upper Confidence Tree (UCT) (9:19)
12 Game Playing
53
Minimax (10:3) Alpha-Beta Pruning (10:6) Evaluation functions (10:11) UCT for games (10:12)
13 Graphical Models
55
Bayesian Network (11:3) Conditional independence in a Bayes Net (11:7) Inference: general meaning (11:12)
13.1 Inference Methods in Graphical Models . . . . . . . . . . . . . . . . . . . . . . . .
57
Inference in graphical models: overview (11:17) Monte Carlo (11:19) Importance sampling (11:22) Gibbs sampling (11:24) Variable elimination (11:27) Factor graph (11:30) Belief propagation (11:36) Message passing (11:36) Loopy belief propagation (11:39) Junction tree algorithm (11:41) Maximum a-posteriori (MAP) inference (11:45) Conditional random field* (11:46)
14 Dynamic Models
63
Markov Process (12:2) Filtering, Smoothing, Prediction (12:3) Hidden Markov Model (12:4) HMM: Inference (12:5) HMM inference (12:6) Kalman filter (12:9)
15 Reinforcement Learning
66
Markov Decision Process (MDP) (13:3) Value Function (13:4) Bellman optimality equation (13:8) Value Iteration (13:10) Q-Function (13:11) Q-Iteration (13:12) Proof of convergence of Q-Iteration (13:13) Temporal difference (TD) (13:19) Sarsa (13:21) Q-learning (13:22) Proof of convergence of Q-learning (13:24) Eligibility traces (13:26) Modelbased RL (13:34) Imitation Learning (13:37) Inverse RL (13:40) Policy gradients (13:44)
16 Reinforcement Learning – Exploration Epsilon-greedy exploration in Q-learning (14:4) Sample Complexity (14:6) PAC-MDP efficiency (14:7) ExplicitExploit-or-Explore* (14:9) R-Max (14:14) Bayesian RL (14:15) Optimistic heuristics (14:16)
74
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
17 Exercises 17.1 Exercise 1 . 17.2 Exercise 2 . 17.3 Exercise 3 . 17.4 Exercise 4 . 17.5 Exercise 5 . 17.6 Exercise 6 . 17.7 Exercise 7 . 17.8 Exercise 8 . 17.9 Exercise 9 . 17.10Exercise 10 Index
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
3
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . .
78 78 79 79 80 81 82 82 83 83 84 86
4
1
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Introduction
I wasn’t happy with the first slides (’Introduction’ and ’Intelligent Agents’). So I skip them here. They will also not be relevant for the exam. You may find them on the lecture webpage.
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
3
5
Search
1:4
Example: vacuum world Outline • Problem formulation & examples
Deterministic, fully observable, start in #5. Solution??
• Basic search algorithms 1:1
1:5
Example: Romania Example: vacuum world
On holiday in Romania; currently in Arad. Flight leaves tomorrow from Bucharest
Deterministic, fully observable, start in #5. Solution?? [Right, Suck]
Formulate goal: be in Bucharest, Sgoal = {Bucharest} Formulate problem: states: various cities, S = {Arad, Timisoara, . . . } actions: drive between cities, A = {edges between states} Find solution:
Non-observable, {1, 2, 3, 4, 5, 6, 7, 8} e.g., Right goes Solution??
start to
in
{2, 4, 6, 8}. 1:6
sequence of cities, e.g., Arad, Sibiu, Fagaras, Bucharest minimize costs with cost function, (s, a) 7→ c 1:2
Example: Romania
Example: vacuum world Deterministic, fully observable, start in #5. Solution?? [Right, Suck] Non-observable, start in {1, 2, 3, 4, 5, 6, 7, 8} e.g., Right goes to {2, 4, 6, 8}. Solution?? [Right, Suck, Lef t, Suck] Non-deterministic, start in #5 Murphy’s Law: Suck can dirty a clean carpet Local sensing: dirt, location only. Solution?? 1:7
1:3
Problem types Deterministic, fully observable (“single-state problem”) Agent knows exactly which state it will be in; solution is a sequence First state and world known → the agent does not rely on observations Non-observable (“conformant problem”) Agent may have no idea where it is; solution (if any) is a sequence Nondeterministic and/or partially observable (“contingency problem”) percepts provide new information about current state solution is a reactive plan or a policy often interleave search, execution Unknown state space (“exploration problem”)
Example: vacuum world Deterministic, fully observable, start in #5. Solution?? [Right, Suck] Non-observable, start in {1, 2, 3, 4, 5, 6, 7, 8} e.g., Right goes to {2, 4, 6, 8}. Solution?? [Right, Suck, Lef t, Suck] Non-deterministic, start in #5 Murphy’s Law: Suck can dirty a clean carpet Local sensing: dirt, location only. Solution?? [Right, if dirt then Suck] 1:8
6
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Deterministic, fully observable problem def. A deterministic, fully observable problem is defined by four items: initial state s0 ∈ S
e.g., s0 = Arad
successor function succ : S × A → S e.g., succ(Arad,Arad-Zerind) = Zerind goal states Sgoal ⊆ S e.g., s = Bucharest step cost function cost(s, a, s0 ), assumed to be ≥ 0
states??: integer dirt and robot locations (ignore dirt amounts
e.g., traveled distance, number of actions executed, etc.
etc.)
the path cost is the sum of step costs
actions??: Lef t, Right, Suck, N oOp goal test?? path cost??
A solution is a sequence of actions leading from s0 to a goal
1:12
An optimal solution is a solution with minimal path costs
Example: vacuum world state space graph 1:9
Example: vacuum world state space graph
states??: integer dirt and robot locations (ignore dirt amounts etc.) actions??: Lef t, Right, Suck, N oOp goal test??: no dirt
states??
path cost??
actions??
1:13
goal test??
Example: vacuum world state space graph
path cost?? 1:10
Example: vacuum world state space graph
states??: integer dirt and robot locations (ignore dirt amounts etc.) actions??: Lef t, Right, Suck, N oOp goal test??: no dirt states??: integer dirt and robot locations (ignore dirt amounts
path cost??: 1 per action (0 for N oOp)
etc.)
1:14
actions??
Example: The 8-puzzle
goal test?? path cost?? 1:11
Example: vacuum world state space graph
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
7
Example: The 8-puzzle
states?? actions?? goal test?? path cost?? 1:15
Example: The 8-puzzle states??: integer locations of tiles (ignore intermediate positions) actions??: move blank left, right, up, down (ignore unjamming etc.) goal test??: = goal state (given) path cost??: 1 per move [Note: optimal solution of n-Puzzle family is NP-hard]
states??: integer locations of tiles (ignore intermediate posi-
1:19
tions) actions??
3.1
goal test??
Tree Search Algorithms
path cost?? 1:20
1:16
Tree search algorithms Example: The 8-puzzle
Basic idea: offline, simulated exploration of state space by generating successors of already-explored states (a.k.a. expanding states) 1:21
Tree search example states??: integer locations of tiles (ignore intermediate positions) actions??: move blank left, right, up, down (ignore unjamming etc.) goal test?? path cost?? 1:17
1:22
Tree search example
Example: The 8-puzzle
1:23
states??: integer locations of tiles (ignore intermediate positions)
Tree search example
actions??: move blank left, right, up, down (ignore unjamming etc.) goal test??: = goal state (given) path cost?? 1:18
8
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 1:24
Depth-first search Depth-limited search
Implementation: states vs. nodes
Iterative deepening search 1:28
A state is a (representation of) a physical configuration A node is a data structure constituting part of a search tree
Breadth-first search
includes parent, children, depth, path cost g(x) States do not have parents, children, depth, or path cost!
Expand shallowest unexpanded node Implementation: fringe is a FIFO queue, i.e., new successors go at end
The E XPAND function creates new nodes, filling in the various fields and using the S UCCESSOR F N of the problem to create the corresponding states. 1:25
1:29
Breadth-first search
Implementation: general tree search function T REE -S EARCH( problem, fringe) returns a solution, or failure fringe ← I NSERT(M AKE -N ODE(I NITIAL -S TATE[problem]), fringe) loop do if fringe is empty then return failure node ← R EMOVE -F RONT(fringe) if G OAL -T EST(problem, S TATE(node)) then return node fringe ← I NSERTA LL(E XPAND(node, problem), fringe) function E XPAND( node, problem) returns a set of nodes successors ← the empty set for each action, result in S UCCESSOR -F N(problem, S TATE[node]) do s ← a new N ODE PARENT-N ODE[s] ← node; ACTION[s] ← action; S TATE[s] ← result PATH -C OST[s] ← PATH -C OST[node] + S TEP -C OST(S TATE[node], action, result) D EPTH[s] ← D EPTH[node] + 1 add s to successors return successors 1:26
Expand shallowest unexpanded node Implementation: fringe is a FIFO queue, i.e., new successors go at end
1:30
Breadth-first search Expand shallowest unexpanded node Implementation:
Search strategies
fringe is a FIFO queue, i.e., new successors go at end A strategy is defined by picking the order of node expansion Strategies are evaluated along the following dimensions: completeness—does it always find a solution if one exists? time complexity—number of nodes generated/expanded space complexity—maximum number of nodes in memory optimality—does it always find a least-cost solution? 1:31
Time and space complexity are measured in terms of b—maximum branching factor of the search tree
Breadth-first search
d—depth of the least-cost solution m—maximum depth of the state space (may be ∞)
Expand shallowest unexpanded node 1:27
Uninformed search strategies Uninformed strategies use only the information available in the problem definition Breadth-first search Uniform-cost search
Implementation: fringe is a FIFO queue, i.e., new successors go at end
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 1:32
Properties of breadth-first search
9
Depth-first search Expand deepest unexpanded node Implementation:
Complete??
fringe = LIFO queue, i.e., put successors at front 1:33
Properties of breadth-first search Complete?? Yes (if b is finite) Time?? 1:34
Properties of breadth-first search
1:39
Depth-first search
Complete?? Yes (if b is finite)
Expand deepest unexpanded node
Time?? 1 + b + b2 + b3 + . . . + bd + b(bd − 1) = O(bd+1 ), i.e., exp. in d
Implementation: fringe = LIFO queue, i.e., put successors at front
Space?? 1:35
Properties of breadth-first search Complete?? Yes (if b is finite) Time?? 1 + b + b2 + b3 + . . . + bd + b(bd − 1) = O(bd+1 ), i.e.,
1:40
exp. in d
Depth-first search
Space?? O(bd+1 ) (keeps every node in memory) Optimal??
Expand deepest unexpanded node 1:36
Implementation:
Properties of breadth-first search
fringe = LIFO queue, i.e., put successors at front
Complete?? Yes (if b is finite) Time?? 1 + b + b2 + b3 + . . . + bd + b(bd − 1) = O(bd+1 ), i.e., exp. in d Space?? O(bd+1 ) (keeps every node in memory) Optimal?? Yes (if cost = 1 per step); not optimal in general
1:41
Space is the big problem; can easily generate nodes at 100MB/sec so 24hrs = 8640GB. 1:37
Uniform-cost search
Depth-first search Expand deepest unexpanded node Implementation: fringe = LIFO queue, i.e., put successors at front
Expand least-cost unexpanded node Implementation: fringe = queue ordered by path cost, lowest first Equivalent to breadth-first if step costs all equal Complete?? Yes, if step cost ≥ Time?? # of nodes with g ≤ cost of optimal solution, O(bdC
∗
/e
where C ∗ is the cost of the optimal solution Space?? # of nodes with g ≤ cost of optimal solution, O(bdC
∗
1:42
)
/e
Depth-first search )
Optimal?? Yes—nodes expanded in increasing order of g(n) 1:38
Expand deepest unexpanded node Implementation: fringe = LIFO queue, i.e., put successors at front
10
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 1:47
Properties of depth-first search Complete?? 1:48
1:43
Depth-first search
Properties of depth-first search
Expand deepest unexpanded node
Complete?? No: fails in infinite-depth spaces, spaces with loops
Implementation:
Modify to avoid repeated states along path
fringe = LIFO queue, i.e., put successors at front
⇒ complete in finite spaces Time?? 1:49
Properties of depth-first search 1:44
Complete?? No: fails in infinite-depth spaces, spaces with loops Modify to avoid repeated states along path
Depth-first search
⇒ complete in finite spaces Expand deepest unexpanded node
Time?? O(bm ): terrible if m is much larger than d but if solutions are dense, may be much faster than breadth-
Implementation: first
fringe = LIFO queue, i.e., put successors at front
Space?? 1:50
Properties of depth-first search 1:45
Complete?? No: fails in infinite-depth spaces, spaces with loops Modify to avoid repeated states along path
Depth-first search
⇒ complete in finite spaces Time?? O(bm ): terrible if m is much larger than d
Expand deepest unexpanded node
but if solutions are dense, may be much faster than breadth-
Implementation: first
fringe = LIFO queue, i.e., put successors at front
Space?? O(bm), i.e., linear space! Optimal?? 1:51
Properties of depth-first search 1:46
Complete?? No: fails in infinite-depth spaces, spaces with loops
Depth-first search
Modify to avoid repeated states along path ⇒ complete in finite spaces
Expand deepest unexpanded node
Time?? O(bm ): terrible if m is much larger than d
Implementation: fringe = LIFO queue, i.e., put successors at front
but if solutions are dense, may be much faster than breadthfirst Space?? O(bm), i.e., linear space! Optimal?? No 1:52
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
11
Depth-limited search = depth-first search with depth limit l, i.e., nodes at depth l have no successors Recursive implementation: function D EPTH -L IMITED -S EARCH( problem, limit) returns soln/fail/cutoff R ECURSIVE -DLS(M AKE -N ODE(I NITIAL -S TATE[problem]), problem, limit) function R ECURSIVE -DLS(node, problem, limit) returns soln/fail/cutoff cutoff-occurred? ← false if G OAL -T EST(problem, S TATE[node]) then return node else if D EPTH[node] = limit then return cutoff else for each successor in E XPAND(node, problem) do result ← R ECURSIVE -DLS(successor, problem, limit) if result = cutoff then cutoff-occurred? ← true else if result 6= failure then return result if cutoff-occurred? then return cutoff else return failure
1:58
Properties of iterative deepening search 1:53
Complete?? 1:59
Iterative deepening search function I TERATIVE -D EEPENING -S EARCH( problem) returns a solution inputs: problem, a problem
Properties of iterative deepening search Complete?? Yes
for depth ← 0 to ∞ do result ← D EPTH -L IMITED -S EARCH( problem, depth) if result 6= cutoff then return result end
Time?? 1:60
1:54
Properties of iterative deepening search Complete?? Yes
Iterative deepening search l = 0
Time?? (d + 1)b0 + db1 + (d − 1)b2 + . . . + bd = O(bd ) Space?? 1:61
1:55
Properties of iterative deepening search Complete?? Yes
Iterative deepening search l = 1
Time?? (d + 1)b0 + db1 + (d − 1)b2 + . . . + bd = O(bd ) Space?? O(bd) Optimal?? 1:62 1:56
Properties of iterative deepening search Iterative deepening search l = 2
Complete?? Yes Time?? (d + 1)b0 + db1 + (d − 1)b2 + . . . + bd = O(bd ) Space?? O(bd) Optimal?? Yes, if step cost = 1 Can be modified to explore uniform-cost tree Numerical comparison for b = 10 and d = 5, solution at far left leaf: 1:57
Iterative deepening search l = 3
N (IDS)
=
50 + 400 + 3, 000 + 20, 000 + 100, 000 = 123, 450
N (BFS)
=
10 + 100 + 1, 000 + 10, 000 + 100, 000 + 999, 990 =
12
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 IDS does better because other nodes at depth d are not expanded BFS can be modified to apply goal test when a node is generated 1:63
Summary of algorithms Criterion
BreadthFirst
UniformCost
DepthFirst
DepthLimited
Iterative Deepening
Yes∗ bd+1 bd+1 Yes∗
Yes∗ ∗ bdC /e ∗ dC /e b Yes
No bm bm No
Yes, if l ≥ d bl bl No
Yes bd bd Yes∗
Complete? Time Space Optimal?
1:64
Loops: Repeated states Failure to detect repeated states can turn a linear problem into an exponential one!
1:65
Graph search function G RAPH -S EARCH( problem, fringe) returns a solution, or failure closed ← an empty set fringe ← I NSERT(M AKE -N ODE(I NITIAL -S TATE[problem]), fringe) loop do if fringe is empty then return failure node ← R EMOVE -F RONT(fringe) if G OAL -T EST(problem, S TATE[node]) then return node if S TATE[node] is not in closed then add S TATE[node] to closed fringe ← I NSERTA LL(E XPAND(node, problem), fringe) end
But: storing all visited nodes leads again to exponential space complexity (as for BFS) 1:66
Summary In BFS (or uniform-cost search), the fringe propagates layerwise, containing nodes of similar distance-from-start (cost-sofar), leading to optimal paths but exponential space complexity O(B d+1 ) In DFS, the fringe is like a deep light beam sweeping over the tree, with space complexity O(bm). Iteratively deepening it also leads to optimal paths. Graph search can be exponentially more efficient than tree search, but storing the visited nodes may lead to exponential space complexity as BFS. 1:67
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
4
Informed search algorithms
13
Greedy search We set the priority function equal to a heuristic f (n) = h(n) h(n) = estimate of cost from n to the closest goal
Outline
E.g., hSLD (n) = straight-line distance from n to Bucharest Greedy search expands the node that appears to be closest to
• Best-first search
goal
• A∗ search
2:5
• Heuristics 2:1
Greedy search example
Review: Tree search function T REE -S EARCH( problem, fringe) returns a solution, or failure fringe ← I NSERT(M AKE -N ODE(I NITIAL -S TATE[problem]), fringe) loop do if fringe is empty then return failure node ← R EMOVE -F RONT(fringe) if G OAL -T EST[problem] applied to S TATE(node) succeeds return node fringe ← I NSERTA LL(E XPAND(node, problem), fringe)
2:6
Greedy search example
A strategy is defined by picking the order of node expansion 2:2
Best-first search 2:7
Idea: use an arbitrary priority function f (n) for each node – actually f (n) is neg-priority: nodes with lower f (n) have higher priority
Greedy search example
f (n) should reflect which nodes could be on an optimal path – could is optimistic – the lower f (n) the more optimistic you are that n is on an optimal path ⇒ Expand the unexpanded node with highes priority Implementation: fringe is a queue sorted in decreasing order of priority 2:8
Special cases: greedy search
Greedy search example
A∗ search 2:3
Romania with step costs in km
2:9
Properties of greedy search Complete?? 2:4
2:10
14
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Properties of greedy search
n. (Also require h(n) ≥ 0, so h(G) = 0 for any goal G.)
Complete?? No–can get stuck in loops, e.g., with Oradea as goal,
E.g., hSLD (n) never overestimates the actual road distance Theorem: A∗ search is optimal (=finds the optimal path)
Iasi → Neamt → Iasi → Neamt →
2:15
Complete in finite space with repeated-state checking Time?? 2:11
A∗ search example
Properties of greedy search Complete?? No–can get stuck in loops, e.g., 2:16
Iasi → Neamt → Iasi → Neamt → Complete in finite space with repeated-state checking Time?? O(bm ), but a good heuristic can give dramatic improve-
A∗ search example
ment Space?? 2:12
Properties of greedy search 2:17
Complete?? No–can get stuck in loops, e.g., Iasi → Neamt → Iasi → Neamt →
A∗ search example
Complete in finite space with repeated-state checking Time?? O(bm ), but a good heuristic can give dramatic improvement Space?? O(bm )—keeps all nodes in memory Optimal?? 2:13
2:18
Properties of greedy search A∗ search example
Complete?? No–can get stuck in loops, e.g., Iasi → Neamt → Iasi → Neamt → Complete in finite space with repeated-state checking Time?? O(bm ), but a good heuristic can give dramatic improvement Space?? O(bm )—keeps all nodes in memory Optimal?? No
2:19
Greedy search does not care about the ’past’ (the cost-so-far). 2:14
A∗ search example
A∗ search Idea: combine information from the past and the future – neg-priority = cost-so-far + estimated cost-to-go Evaluation function f (n) = g(n) + h(n) g(n) = cost-so-far to reach n h(n) = estimated cost-to-go from n f (n) = estimated total cost of path through n to goal A∗ search uses an admissible (=optimistic) heuristic i.e., h(n) ≤ h∗ (n) where h∗ (n) is the true cost-to-go from
2:20
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
A∗ search example
15
Properties of A∗ Complete?? Yes, unless there are infinitely many nodes with f ≤ f (G) Time?? Exponential in [relative error in h × length of soln.] Space?? Keeps all nodes in memory Optimal?? 2:26
Properties of A∗ 2:21
Complete?? Yes, unless there are infinitely many nodes with f ≤ f (G)
Proof of optimality of A∗
Time?? Exponential in [relative error in h × length of soln.] Suppose some suboptimal goal G2 has been generated and is in the fringe (but has not yet been selected to be tested for goal condition!). Let n be an unexpanded node on a shortest path to an optimal goal G.
Space?? Keeps all nodes in memory Optimal?? Yes A∗ expands all nodes with f (n) < C ∗ A∗ expands some nodes with f (n) = C ∗ A∗ expands no nodes with f (n) > C ∗ 2:27
Optimality of A∗ (more useful) f (G2 )
=
g(G2 )
>
g(G)
since G2 is suboptimal
≥
f (n)
since h is admissible
Lemma: A∗ expands nodes in order of increasing f value∗
since h(G2 ) = 0
Gradually adds “f -contours” of nodes (cf. breadth-first adds layers) Contour i has all nodes with f = fi , where fi < fi+1
Since f (n) < f (G2 ), A∗ will expand n before is will select G2 from the fringe (for goal testing). Then, as G is added to the fringe, and since f (G) = g(G) < f (G2 ) = g(G2 ) it will select G before G2 for goal testing. 2:22
Properties of A∗ Complete?? 2:23 2:28
∗
Properties of A
Proof of lemma: Consistency
Complete?? Yes, unless there are infinitely many nodes with A heuristic is consistent if
f ≤ f (G)
h(n) ≤ c(n, a, n0 ) + h(n0 )
Time?? 2:24
If h is consistent, we have
Properties of A∗
f (n0 )
=
g(n0 ) + h(n0 )
=
g(n) + c(n, a, n0 ) + h(n0 )
f ≤ f (G)
≥
g(n) + h(n)
Time?? Exponential in [relative error in h × length of soln.]
=
f (n)
Complete?? Yes, unless there are infinitely many nodes with
Space??
I.e., f (n) is nondecreasing along any path. 2:25
2:29
16
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Admissible heuristics
Key point: the optimal solution cost of a relaxed problem is no greater than the optimal solution cost of the real problem
E.g., for the 8-puzzle:
2:33
h1 (n) = number of misplaced tiles
Memory-bounded A∗
h2 (n) = total Manhattan distance (i.e., no. of squares from desired location of each tile)
As with BFS, A∗ has exponential space complexity Iterative-deepening A∗ , works for integer path costs, but problematic for real-valued (Simplified) Memory-bounded A∗ (SMA∗ ): – Expand as usual until a memory bound is reach
h1 (S) =??
– Then, whenever adding a node, remove the worst node
h2 (S) =??
0
n from the tree – worst means: the n0 with highest f (n0 )
2:30
– To not loose information, backup the measured step-cost
Admissible heuristics
cost(˜ n, a, n0 ) to improve the heuristic h(˜ n) of its parent
E.g., for the 8-puzzle: ∗
SMA is complete and optimal if the depth of the optimal path is
h1 (n) = number of misplaced tiles
within the memory bound
h2 (n) = total Manhattan distance
2:34
(i.e., no. of squares from desired location of each tile)
Summary Combine information from the past and the future A heuristic function h(n) represents information about the future – it estimates cost-to-go optimistically
h1 (S) =?? 6
Good heuristics can dramatically reduce search cost
h2 (S) =?? 4+0+3+3+1+0+2+1 = 14 2:31
Greedy best-first search expands lowest h – incomplete and not always optimal
Dominance
∗
A search expands lowest f = g + h – neg-priority = cost-so-far + estimated cost-to-go
If h2 (n) ≥ h1 (n) for all n (both admissible)
– complete and optimal
then h2 dominates h1 and is better for search
– also optimally efficient (up to tie-breaks, for forward search)
Typical search costs: d = 14 IDS = 3,473,941 nodes A∗ (h1 ) = 539 nodes A∗ (h2 ) = 113 nodes d = 24 IDS ≈ 54,000,000,000 nodes A∗ (h1 ) = 39,135 nodes A∗ (h2 ) = 1,641 nodes Given any admissible heuristics ha , hb ,
Admissible heuristics can be derived from exact solution of relaxed problems Memory-bounded startegies exist 2:35
Outlook We postpone tree search with partial observations
h(n) = max(ha (n), hb (n))
– rather discuss this in a fully probabilistic setting later is also admissible and dominates ha , hb
We postpone tree search for games 2:32
Relaxed problems
– minimax extension to tree search – discuss state-of-the-art probabilistic Monte-Carlo tree search methods later
Admissible heuristics can be derived from the exact solution cost of a relaxed version of the problem If the rules of the 8-puzzle are relaxed so that a tile can move anywhere, then h1 (n) gives the shortest solution If the rules are relaxed so that a tile can move to any adjacent square, then h2 (n) gives the shortest solution
Next: Constraint Statisfaction Problems 2:36
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
5
Constraint Satisfaction Problems
17
Example: Map-Coloring contd.
Outline • CSP examples • Backtracking sequential assignment for CSPs • Problem structure and problem decomposition
• Later: general-purpose discrete (and continuous) optimization methods
Solutions are assignments satisfying all constraints, e.g., 3:1
{W = red, N = green, Q = red, E = green, V = red, S = blue, T = green} 3:4
Constraint satisfaction problems (CSPs) Constraint graph In previous lectures we consideres sequential decision problems
Binary CSP: each constraint relates at most two variables
CSPs are not sequential decision problems
Constraint graph: a bi-partite graph: nodes are variables, boxes
However, the basic methods address them by testing sequentially ’decisions’
are constraints In the map-coloring problem, all constraints relate two variables: boxes↔edges In general, constraints may constrain several (or one) vari-
CSP:
ables (|Ik | 6= 2)
We have n variables xi , each with domain Di , xi ∈ Di
N
We have K constraints Ck , each of which determines the feasible configurations of a subset of variables
W
The goal is to find a configuration X = (X1 , .., Xn ) of all vari-
c1 c2
ables that satisfies all constraints
c3
Q
S
E
c6
c8
c7
Formally Ck = (Ik , ck ) where Ik ⊆ {1, .., n} determines the
c5
c4
c9
V
subset of variables, and ck : DIk → {0, 1} determines whether a configuration xIk ∈ DIk of this subset of variables is feasible
T 3:5
3:2
Varieties of CSPs
Example: Map-Coloring
• Discrete variables: finite domains; each Di of size |Di | = d ⇒ O(dn ) complete assignments – e.g., Boolean CSPs, incl. Boolean satisfiability infinite domains (integers, strings, etc.) – e.g., job scheduling, variables are start/end days for each job – linear constraints solvable, nonlinear undecidable • Continuous variables – e.g., start/end times for Hubble Telescope observations – linear constraints solvable in poly time by LP methods Variables W , N , Q, E, V , S, T
3:6
(E = New South Wales)
Domains Di = {red, green, blue} for all variables
Varieties of constraints
Constraints: adjacent regions must have different colors e.g., W 6= N , or
Unary constraints involve a single variable, |Ik | = 1
(W, N ) ∈ {(red, green), (red, blue), (green, red), (green, blue), . . .} 3:3
e.g., S 6= green Binary constraints involve pairs of variables, |Ik | = 2 e.g., S 6= W
18
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 ⇒ b = d and there are dn leaves
Higher-order constraints involve 3 or more variables, |Ik | > 2 e.g., Sudoku
Depth-first search for CSPs with single-variable assignments Having “soft constraints” (preferences, cost, probabilities) leads
is called backtracking search
to general optimization and probabilistic inference problems
Backtracking search is the basic uninformed algorithm for CSPs 3:7
Can solve n-queens for n ≈ 25 3:11
Real-world CSPs Backtracking search Assignment problems function B ACKTRACKING -S EARCH(csp) returns solution/failure return R ECURSIVE -B ACKTRACKING({ }, csp)
e.g., who teaches what class Timetabling problems
function R ECURSIVE -B ACKTRACKING(assignment, csp) returns soln/failure if assignment is complete then return assignment var ← S ELECT-U NASSIGNED -VARIABLE(VARIABLES[csp], assignment, csp) for each value in O RDERED -D OMAIN -VALUES(var, assignment, csp) do if value is consistent with assignment given C ONSTRAINTS[csp] then add [var = value] to assignment result ← R ECURSIVE -B ACKTRACKING(assignment, csp) if result 6= failure then return result remove [var = value] from assignment return failure
e.g., which class is offered when and where? Hardware configuration Spreadsheets Transportation scheduling Factory scheduling Floorplanning
Notice that many real-world problems involve real-valued variables 3:8
5.1
3:12
Backtracking example
Methods for solving CSPs 3:13 3:9
Backtracking example
Sequential assignment approach Let’s start with the straightforward, dumb approach, then fix it States are defined by the values assigned so far • Initial state: the empty assignment, { } • Successor function: assign a value to an unassigned variable that does not conflict with current assignment ⇒ fail if no feasible assignments (not fixable!)
3:14
• Goal test: the current assignment is complete
Backtracking example 1) Every solution appears at depth n with n variables ⇒ use depth-first search 2) b = (n − `)d at depth `, hence n!dn leaves! 3:10
Backtracking sequential assignment Two variable assignment decisions are commutative, i.e., [W = red then N = green] same as [N = green then W = red] We can fix a single next variable to assign a value to at each node This does not compromise completeness (ability to find the solution)
3:15
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
19
Backtracking example Given a variable, choose the least constraining value: the one that rules out the fewest values in the remaining variables
Combining these heuristics makes 1000 queens feasible 3:20 3:16
Forward checking Improving backtracking efficiency
Idea: Keep track of remaining legal values for unassigned variables
Simple heuristics can give huge gains in speed:
Terminate search when any variable has no legal values
1. Which variable should be assigned next? 2. In what order should its values be tried? 3. Can we detect inevitable failure early? 4. Can we take advantage of problem structure? 3:21 3:17
Forward checking Minimum remaining values
Idea: Keep track of remaining legal values for unassigned variables
Minimum remaining values (MRV):
Terminate search when any variable has no legal values
choose the variable with the fewest legal values
3:18 3:22
Degree heuristic
Forward checking
Tie-breaker among MRV variables Idea: Keep track of remaining legal values for unassigned vari-
Degree heuristic: choose the variable with the most constraints on remaining variables
ables Terminate search when any variable has no legal values
3:19
Least constraining value
3:23
20
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Forward checking
for every value x of X there is some allowed y
Idea: Keep track of remaining legal values for unassigned variables Terminate search when any variable has no legal values
3:27
Arc consistency (for pair-wise constraints) Simplest form of propagation makes each arc consistent X → Y is consistent iff for every value x of X there is some allowed y Use a data structure D OMAIN [X] to explicitly store Di for each node 3:24
Constraint propagation If X loses a value, neighbors of X need to be rechecked Forward checking propagates information from assigned to unassigned variables, but doesn’t provide early detection for all failures:
3:28
Arc consistency (for pair-wise constraints) Simplest form of propagation makes each arc consistent X → Y is consistent iff for every value x of X there is some allowed y
N and S cannot both be blue! Idea: propagate the implied constraints serveral steps further Generally, this is called constraint propagation 3:25
If X loses a value, neighbors of X need to be rechecked Arc consistency detects failure earlier than forward checking Can be run as a preprocessor or after each assignment
Arc consistency (for pair-wise constraints)
3:29
Arc consistency algorithm (for pair-wise constraints)
Simplest form of propagation makes each arc consistent X → Y is consistent iff for every value x of X there is some allowed y
function AC-3( csp) returns the CSP, possibly with reduced domains inputs: csp, a pair-wise CSP with variables {X1 , X2 , . . . , Xn } local variables: queue, a queue of arcs, initially all the arcs in csp while queue is not empty do (Xi , Xj ) ← R EMOVE -F IRST(queue) if R EMOVE -I NCONSISTENT-VALUES(Xi , Xj ) then for each Xk in N EIGHBORS[Xi ] do add (Xk , Xi ) to queue 3:26
Arc consistency (for pair-wise constraints) Simplest form of propagation makes each arc consistent X → Y is consistent iff
function R EMOVE -I NCONSISTENT-VALUES( Xi , Xj ) returns true iff D OM[Xi ] changed changed ← false for each x in D OMAIN[Xi ] do if no value y in D OMAIN[Xj ] allows (x,y) to satisfy the constraint Xi ↔ Xj then delete x from D OMAIN[Xi ]; changed ← true return changed
O(n2 d3 ), can be reduced to O(n2 d2 )
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 3:30
21
3. For j from 1 to n, assign Xj consistently with P arent(Xj ) This is forward sequential assignment (trivial backtracking)
Constraint propagation
3:34
Nearly tree-structured CSPs
See textbook for details for non-pair-wise constraints Very closely related to message passing in probabilistic models
Conditioning: instantiate a variable, prune its neighbors’ domains In practice: design approximate constraint propagation for specific problem E.g.: Sudoku: If Xi is assigned, delete this value from all peers
3:31
Problem structure
Cutset conditioning: instantiate (in all ways) a set of variables such that the remaining constraint graph is a tree Cutset size c ⇒ runtime O(dc · (n − c)d2 ), very fast for small c
N W
c1 c2
c3
3:35
Q
c5
c4
c9
S
Summary
E
c6
c8
c7
CSPs are a fundamental kind of problem:
V
finding a feasible configuration of n variables the set of constraints defines the (graph) structure of the prob-
T
lem
Tasmania and mainland are independent subproblems
Sequential assignment approach Backtracking = depth-first search with one variable assigned
Identifiable as connected components of constraint graph 3:32
per node Variable ordering and value selection heuristics help significantly
Tree-structured CSPs
Forward checking prevents assignments that guarantee later failure Constraint propagation (e.g., arc consistency) does additional work to constrain values and detect inconsistencies The CSP representation allows analysis of problem structure
Theorem: if the constraint graph has no loops, the CSP can be solved in O(n d2 ) time
Tree-structured CSPs can be solved in linear time If after assigning some variables, the remaining structure is a
Compare to general CSPs, where worst-case time is O(dn )
tree → linear time feasibility check by tree CSP
This property also applies to logical and probabilistic reasoning! 3:33
Algorithm for tree-structured CSPs 1. Choose a variable as root, order variables from root to leaves such that every node’s parent precedes it in the ordering
2. For j from n down to 2, apply R EMOVE I NCONSISTENT(P arent(Xj ), Xj ) This is backward constraint propagation
3:36
22
6
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Optimization
Iterative improvement The majority of optimization methods iteratively manipulate x to monotonely improve x, e.g.: – line search, backtracking, trust region methods
Outline
– gradient-based, (Quasi-) Newton methods – interior point methods, Simplex method
• Local Search
– primal-dual Newton
• Iterated Local Search
– local search, pattern search, Nelder-Mead
• Simulated annealing & Genetic algorithms (briefly)
Exceptions: – Global Optimization, Bayesian Optimization
• General formulation of optimization problems
– stochastic search, simulated annealing, evolutionary algorithms
• LP, QP, ILP, non-linear program • ILP formulations of n-queens, CSP, TSP
4:4 4:1
Local search (greedy downhill, hill climbing) Optimization problems We have n variables xi ,
We assume there is a finite neighborhood N(x) defined for every
continuous x ∈ Rn , or discrete xi ∈
x
{1, .., d}, or mixed Greedy local search (variant 1): An optimization problem (or mathematical program) is defined by min f (x) x
s.t.
g(x) ≤ 0, h(x) = 0
where g : Rn → Rk defines k inequality constraints,
Input: Initial x, function f (x) Output: Local minimum x ˆ of f (x) 1: repeat 2: x ˆ←x 3: x ← argminy∈N(x) f (y) 4: until f (ˆ x) ≤ f (x)
and h : Rn → Rl defines l equality constraints
Variant 2: x ← the “first” y ∈ N(x) such that f (y) < f (x)
Optimization is a central thread through all of science: – Machine Learning, Robotics, Computer Vision
4:5
– Engineering, Control Theory – Economics, Operations Research
Example: Travelling Salesman Problem (TSP)
– Physics, Chemistry, Molecular Biology – Social Sciences
Goal: Find the shortest closed tour visiting n cities.
Computational modelling of natural phenomena often via opti-
Start with any complete tour; modify 2 arcs to make the tour
mality principles
shorter 4:2
Stefan Funke gives an excellent lecture on Discrete Optimization (WS) – max-flow and min-cut on graphs – Linear Programs, esp. Simplex methods
Variants of this approach get within 1% of optimum very quickly
– Integer Linear Programming and LP-relaxations
with thousands of cities In TSP, this neighborhood is called 2-opt (modifying 2 arcs).
I offer a lecture on Continuous Optimization (SS) – Gradient and Newton methods
3-opt or 4-opt are larger neighborhoods. 4:6
– Lagrangian, log-barrier, augmented lagrangian methods, primal-dual – Local & stochastic search, global optimization, Bayesian optimization 4:3
Example: n-queens Goal: Put n queens on an n × n board with no two queens on the same row, column, or diagonal
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 Start with any configuration of n queens; move a queen to re-
23
Example: Tavelling Salesman Problem
duce number of conflicts LocalSearch uses the simple 2-opt or 3-opt neighborhood (→ quick) Iterated Local Search uses 4-opt meta-neighborhood (double bridges) 4:10
Simulated annealing Almost always solves n-queens problems almost instantaneously for very large n, e.g., n = 1million
Idea: Escape local minimum by allowing some “bad” moves but gradually decrease their size and frequency
4:7
Input: initial x, function f (x), proposal distribution q(y|x), initial temp. T Output: Global minimum x ˆ of f (x) 1: repeat 2: Sample y from the neighborhood of x, y ∼ q(y|x) 3: Acceptance probability A = f (x)−f (y) q(x|y) T min 1, e q(y|x) 4: With probability A update x ← y 5: Decrease T , e.g. T ← (1 − )T for small 6: until T = 0 or x converges
Local search contd. Useful to consider solution space landscape
Typically q(x|y) = q(y|x) The new sample y is always accepted if y is better than x (f (y) ≤ f (x)) Random-restart local search overcomes local optima problem—
If y is worse than x, only accept with probability e
f (x)−f (y) T
trivially complete
4:11
Random sideways moves escape from plateaus, but loop on flat
Properties of simulated annealing
optima 4:8
At fixed “temperature” T , state occupation probability reaches Boltzman distribution
Iterated Local Search (6= random restarts) p(x) = αe
−f (x) kT
Random restarts may be rather expensive, sampling initial x is T decreased slowly enough =⇒ always reach best state x∗ =
fully uninformed Idea: Escape local minimum x by restarting in a meta-neighborhood
argminx f (x) because e
N (x) ∗
−f (x∗ ) kT
/e
−f (x) kT
=e
f (x)−f (x∗ ) kT
1 for small T
Is this necessarily an interesting guarantee?? Input: Initial x, function f (x) Output: Local minimum x ˆ of f (x) 1: repeat 2: For all meta-neigbors yi ∈ N∗ (x) compute yˆi ← LocalSearch(yi ) 3: x ← argminy∈{ˆ y1 ,..,ˆ yI } f (y) 4: until x converges
Devised by Metropolis et al., 1953, for physical process modelling 4:12
Local beam search (maintain k candidates) Idea: keep k candidates instead of 1; choose top k of all their
LocalSearch uses a simple/quick neighborhood N(x)
successors
The meta-neighborhood N∗ (x) enables large jumps towards alternative local optima
Not the same as k searches run in parallel! Searches that find good candidates recruit other searches
Variant 2: x ← the “first” yi ∈ N∗ (x) such that f (ˆ yi ) < f (x)
to join them
Stochastic variant: Meta-neighborhood N∗ (x) is replaced by a
Problem: quite often, all k candidates end up on same local hill
transition prob. q ∗ (y|x)
Idea: choose k successors randomly, biased towards good ones 4:9
4:13
24
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Genetic algorithms
– Sequential Quadratic Programming (SQP), Log-barrier, Augmented Lagrangian, primal-dual
= stochastic local beam search + generate successors from
4:17
pairs of candidates
The art is in finding a reduction How can a real-world problem be encoded as an optimization problem? 4:18 4:14
Genetic algorithms contd.
Example: n-queens as Integer Linear Program binary indicator variables xij for a queen at position (i, j), i, j =
GAs require solutions encoded as strings (GPs use trees or pro-
1, .., n Constraints:
grams)
– row constraints ∀i :
P
xij ≤ 1 P i xij ≤ 1 P – diagonal cnstr. ∀i∈{−n+1,..,n−1} : j:j,i+j∈{1,..,n} xi+j,j ≤ 1 P – diagonal cnstr. ∀i∈{−n+1,..,n−1} : j:j,i−j∈{1,..,n} xi−j,j ≤ 1
Crossover helps iff substrings are meaningful components
j
– column constraints ∀j :
Objective Function: arbitrary (e.g. f (x) = 1)! We encoded everything in the constraints!
GAs 6= evolution: e.g., real genes encode replication machinery!
Better alternative: Optimize the number of constraint violations: instead of “≤ 1” write “≤ 1 + ξk ” in all constraints the slack variables ξ = (ξ1 , .., ξK ) become part of the state
Move general view:
add the constraints ξk ≥ 0
keeping multiple candidates allows us to use
objective function f (x, ξ) =
more general neighborhoods N(x1 , .., xK ) or meta-neighborhoods
P
k
ξk
related to Phase I optimization of finding a feasible x
4:15
6.1
A glimpse at general optimization problems
4:19
Example: TSP as Integer Linear Program binary indicator variables xij for (ij) ∈ tour
4:16
Optimization problems
city-visit-times ti ∈ {1, .., n} Objective: cost f (x) =
Linear Program (LP) >
min c x s.t. x
Gx ≤ h, Ax = b
– Simplex Algorithm, Interior point method (Log-barrier), Augmented Lagrangian, primal-dual – LP in standard form: minx c>x s.t.
P
ij
cij xij of the tour
Constraints: P – Columns sum to 1: ∀j : i xij = 1 P – Rows sum to 1: ∀i : j xij = 1 – city-visit-times ti must fulfill: ∀2≤i6=j≤n : ti − tj ≤ n − 2 − (n − 1)xij
x ≥ 0, Ax = b
Quadratic Program (QP) (Q is positive definite) 1 min x>Qx + c>x s.t. Gx ≤ h, Ax = b x 2 – Log-barrier, Augmented Lagrangian, primal-dual Newton
(There are alternative formulations.) 4:20
Example: CSP as Integer Linear Program
Integer Linear Program (ILP) min c>x s.t. x
Ax = b, xi ∈ {1, .., di }
– LP-relaxations & backtracking, specialized methods, graph cut methods Non-linear program (Convex Program: f, g convex and h linear) min f (x) x
s.t.
g(x) ≤ 0, h(x) = 0
binary indicator variables xiv = [Xi = v] for every CSP variable Xi Constraints: – “Xi can have only one value”: probabilities..)
∀i :
P
v
xiv = 1 (cf.
– If [Xi = v ∧Xj = w] is constraint-violating, add a constraint xiv + xjw ≤ 1
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 – Do this for EVERY forbidden local configuration (MANY constraints) Objective Function: arbitrary (e.g. f (x) = 1)! We encoded everything in the constraints!
Better alternative: Translate the constraints into soft constraints xiv + xjw ≤ 1 + ξk Minimize
P
k
ξk
s.t.
ξk ≥ 0
(There exists a more efficient formulation for MaxSAT in conjunctive normal form.) 4:21
Summary Many problems can be reduced to optimization problems Local Search, esp. Iterated Local Search is often effective in practice In continuous domains, when gradients, Hessians are given → full field of optimization Ongoing research in global & Bayesian optimization 4:22
25
26
7
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Propositional Logic Performance measure gold +1000, death -1000 -1 per step, -10 for using the arrow Environment Squares adjacent to wumpus are smelly Squares adjacent to pit are breezy Glitter iff gold is in the same square Shooting kills wumpus if you are facing it The wumpus kills you if in the same square Shooting uses up the only arrow Grabbing picks up gold if in same square Releasing drops the gold in same square Actuators Left turn, Right turn,
Outline • Knowledge-based agents • Wumpus world • Logic in general—models and entailment • Propositional (Boolean) logic • Equivalence, validity, satisfiability • Inference rules and theorem proving
Forward, Grab, Release, Shoot, Climb
– forward chaining
Sensors Breeze, Glitter, Stench, Bump, Scream
– backward chaining – resolution
5:4 5:1
Wumpus world characterization Knowledge bases
Observable?? 5:5
Knowledge base = set of sentences of a formal language
Wumpus world characterization Declarative approach to building an agent (or other system): T ELL it what it needs to know
Observable?? No—only local perception
Then it can A SK itself what to do—answers should follow from
Deterministic??
the KB
5:6
Agents can be viewed at the knowledge level
Wumpus world characterization
i.e., what they know, regardless of how implemented Or at the implementation level i.e., data structures in KB and algorithms that manipulate
Observable?? No—only local perception Deterministic?? Yes—outcomes exactly specified
them
5:7
5:2
Wumpus world characterization
A simple knowledge-based agent
Observable?? No—only local perception function KB-AGENT( percept) returns an action static: KB, a knowledge base t, a counter, initially 0, indicating time
Deterministic?? Yes—outcomes exactly specified Static?? Yes—Wumpus and Pits do not move
T ELL(KB, M AKE -P ERCEPT-S ENTENCE( percept, t)) action ← A SK(KB, M AKE -ACTION -Q UERY(t)) T ELL(KB, M AKE -ACTION -S ENTENCE(action, t)) t←t + 1 return action
Discrete?? 5:8
Wumpus world characterization
The agent must be able to: Represent states, actions, etc. Incorporate new percepts Update internal representations of the world Deduce hidden properties of the world Deduce appropriate actions
Observable?? No—only local perception Deterministic?? Yes—outcomes exactly specified Static?? Yes—Wumpus and Pits do not move 5:3
Discrete?? Yes Single-agent??
Wumpus World description
5:9
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
27
Wumpus world characterization Observable?? No—only local perception Deterministic?? Yes—outcomes exactly specified Static?? Yes—Wumpus and Pits do not move Discrete?? Yes Single-agent?? Yes—Wumpus is essentially a natural feature 5:10
Exploring a wumpus world 5:13
Exploring a wumpus world
5:11
5:14
Exploring a wumpus world Exploring a wumpus world
5:12 5:15
Exploring a wumpus world Exploring a wumpus world
28
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Breeze in (1,2) and (2,1) ⇒ no safe actions Assuming pits uniformly distributed, (2,2) has pit w/ prob 0.86, vs. 0.31
Smell in (1,1) ⇒ cannot move Can use a strategy of coercion: shoot straight ahead wumpus was there ⇒ dead ⇒ safe wumpus wasn’t there ⇒ safe 5:19 5:16
Logic in general Exploring a wumpus world Logics are formal languages for representing information such that conclusions can be drawn Syntax defines the sentences in the language Semantics define the “meaning” of sentences; i.e., define truth of a sentence in a world E.g., the language of arithmetic x + 2 ≥ y is a sentence; x2 + y > is not a sentence x + 2 ≥ y is true iff the number x + 2 is no less than the number y x + 2 ≥ y is true in a world where x = 7, y = 1 x + 2 ≥ y is false in a world where x = 0, y = 6 5:20
5:17
Entailment Entailment means that one thing follows from another:
Exploring a wumpus world
KB |= α
Knowledge base KB entails sentence α if and only if α is true in all worlds where KB is true E.g., the KB containing “the Giants won” and “the Reds won” entails “Either the Giants won or the Reds won” E.g., x + y = 4 entails 4 = x + y Entailment is a relationship between sentences (i.e., syntax) that is based on semantics 5:21 5:18
Other tight spots
Models Given a logical sentence, when is its truth uniquely defined in a world?
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 Logicians typically think in terms of models, which are formally
29
Wumpus models
structured worlds (e.g., full abstract description of a world, configuration of all variables, world state) with respect to which truth can uniquely be evaluated We say m is a model of a sentence α if α is true in m M (α) is the set of all models of α Then KB |= α if and only if M (KB) ⊆ M (α)
E.g. KB = Giants won and Reds won α = Giants won KB = wumpus-world rules + observations α1 = “[1,2] is safe”, KB |= α1 , proved by model checking 5:22 5:26
Entailment in the wumpus world Wumpus models Situation after detecting nothing in [1,1], moving right, breeze in [2,1] Consider possible models for ?s assuming only pits 3 Boolean choices ⇒ 8 possible models 5:23
Wumpus models
KB = wumpus-world rules + observations α2 = “[2,2] is safe”, KB 6|= α2 5:27
Inference Inference in the general sense means: Given some pieces of information (prior, observed variabes, knowledge base) what is 5:24
the implication (the implied information, the posterior) on other things (non-observed variables, sentence)
Wumpus models KB `i α = sentence α can be derived from KB by procedure i Consequences of KB are a haystack; α is a needle. Entailment = needle in haystack; inference = finding it Soundness: i is sound if whenever KB `i α, it is also true that KB |= α Completeness: i is complete if whenever KB |= α, it is also true that KB `i α Preview: we will define a logic (first-order logic) which is expressive enough to say almost anything of interest, and for which there exists a sound and complete inference procedure. That is, the procedure will answer any question whose answer follows from what is known by the KB.
KB = wumpus-world rules + observations 5:25
5:28
30
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Propositional logic: Syntax
5:33
Propositional logic is the simplest logic—illustrates basic ideas
Wumpus world sentences
The proposition symbols P1 , P2 etc are sentences If S is a sentence, ¬S is a sentence (negation)
Let Pi,j be true if there is a pit in [i, j].
If S1 and S2 are sentences, S1 ∧ S2 is a sentence (conjunction)
Let Bi,j be true if there is a breeze in [i, j].
If S1 and S2 are sentences, S1 ∨ S2 is a sentence (disjunction) If S1 and S2 are sentences, S1 ⇒ S2 is a sentence (implication)
¬P1,1
If S1 and S2 are sentences, S1 ⇔ S2 is a sentence (bicondi-
¬B1,1
tional)
B2,1 5:29
“Pits cause breezes in adjacent squares”
Propositional logic: Syntax grammar hsentencei hatomic sentencei hcomplex sentencei
→ → →
hatomic sentencei | hcomplex sentencei true | false | P | Q | R | . . . ¬ hsentencei | (hsentencei ∧ hsentencei) | (hsentencei ∨ hsentencei) | (hsentencei ⇒ hsentencei) | (hsentencei ⇔ hsentencei)
B1,1
⇔
(P1,2 ∨ P2,1 )
B2,1
⇔
(P1,1 ∨ P2,2 ∨ P3,1 )
“A square is breezy if and only if there is an adjacent pit” 5:34
5:30
Truth tables for inference
Propositional logic: Semantics
B1,1 B2,1 P1,1 P1,2 P2,1 P2,2 P3,1
Each model specifies true/false for each proposition symbol E.g. P1,2 P2,2 P3,1 true true false (With these symbols, 8 possible models, can be enumerated automatically.)
R1
R2
R3
R4
R5
KB
false false false false false false false true true true true false false false false false false false false true true true false true false false .. .. .. .. .. .. .. .. .. .. .. .. .. . . . . . . . . . . . . . false true false false false false false true true false true true false false true false false false false true true true true true true true false true false false false true false true true true true true true false true false false false true true true true true true true true
Rules for evaluating truth with respect to a model m:
¬S is true iff S is false false true false false true false false true false false true true false .. .. .. .. .. .. .. .. .. .. .. .. .. S1 ∧ S2 is true iff S1 is true and S2 is true . . . . . . . . . . . . . S1 ∨ S2 is true iff S1 is true or S2 is true true true true true true true true false true true false true false S1 ⇒ S2 is true iff S1 is false or S2 is true Enumerate rows (different assignments to symbols), i.e., is false iff S1 is true and S2 is false if KB is true in row, check that α is too S1 ⇔ S2 is true iff S1 ⇒ S2 is true and S2 ⇒ S1 is true Simple recursive process evaluates an arbitrary sentence, e.g., 5:35 ¬P1,2 ∧ (P2,2 ∨ P3,1 ) = true ∧ (false ∨ true) = true ∧ true = true 5:31
Inference by enumeration
Truth tables for connectives P
Q
¬P
P ∧Q
P ∨Q
P ⇒Q
P ⇔Q
false false true true
false true false true
true true false false
false false false true
false true true true
true true false true
true false false true 5:32
Wumpus world sentences Let Pi,j be true if there is a pit in [i, j]. Let Bi,j be true if there is a breeze in [i, j]. ¬P1,1 ¬B1,1 B2,1 “Pits cause breezes in adjacent squares”
Depth-first enumeration of all models is sound and complete function TT-E NTAILS ?(KB, α) returns true or false inputs: KB, the knowledge base, a sentence in propositional logic α, the query, a sentence in propositional logic symbols ← a list of the proposition symbols in KB and α return TT-C HECK -A LL(KB, α, symbols, [ ]) function TT-C HECK -A LL(KB, α, symbols, model) returns true or false if E MPTY ?(symbols) then if PL-T RUE ?(KB, model) then return PL-T RUE ?(α, model) else return true else do P ← F IRST(symbols); rest ← R EST(symbols) return TT-C HECK -A LL(KB, α, rest, E XTEND(P , true, model)) and TT-C HECK -A LL(KB, α, rest, E XTEND(P , false, model))
O(2n ) for n symbols 5:36
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
31
Logical equivalence
Forward and backward chaining
Two sentences are logically equivalent iff true in same models:
Applicable when KB is in Horn Form
α ≡ β if and only if α |= β and β |= α
Horn Form (restricted) KB = conjunction of Horn clauses
(α ∧ β)
≡
(β ∧ α) commutativity of ∧
(α ∨ β)
≡
(β ∨ α) commutativity of ∨
– proposition symbol; or
((α ∧ β) ∧ γ)
≡
(α ∧ (β ∧ γ)) associativity of ∧
– (conjunction of symbols) ⇒ symbol
((α ∨ β) ∨ γ)
≡
(α ∨ (β ∨ γ)) associativity of ∨
¬(¬α)
≡
α
(α ⇒ β)
≡
(¬β ⇒ ¬α) contraposition
(α ⇒ β)
≡
(¬α ∨ β) implication elimination
(α ⇔ β)
≡
((α ⇒ β) ∧ (β ⇒ α)) biconditional elimination
¬(α ∧ β)
≡
(¬α ∨ ¬β)
De Morgan
¬(α ∨ β)
≡
(¬α ∧ ¬β)
De Morgan
(α ∧ (β ∨ γ))
≡
((α ∧ β) ∨ (α ∧ γ)) distributivity of ∧ over ∨
(α ∨ (β ∧ γ))
≡
((α ∨ β) ∧ (α ∨ γ)) distributivity of ∨ over ∧
Horn clause =
E.g., C ∧ (B ⇒ A) ∧ (C ∧ D ⇒ B)
double-negation elimination
Modus Ponens (for Horn Form): complete for Horn KBs α1 , . . . , αn ,
α1 ∧ · · · ∧ αn ⇒ β β
Can be used with forward chaining or backward chaining. These algorithms are very natural and run in linear time 5:40
5:37
Validity and satisfiability Forward chaining A sentence is valid if it is true in all models, e.g., true, A ∨ ¬A,
A ⇒ A,
Idea: fire any rule whose premises are satisfied in the KB,
(A ∧ (A ⇒ B)) ⇒ B
Validity is connected to inference via the Deduction Theorem: KB |= α if and only if (KB ⇒ α) is valid
add its conclusion to the KB, until query is found P ⇒Q L∧M ⇒P
A sentence is satisfiable if it is true in some model e.g., A ∨ B,
B∧L⇒M
C
A∧P ⇒L
A sentence is unsatisfiable if it is true in no models
A∧B ⇒L
e.g., A ∧ ¬A
A
Satisfiability is connected to inference via the following:
B
KB |= α if and only if (KB ∧ ¬α) is unsatisfiable
5:41
i.e., prove α by reductio ad absurdum 5:38
Forward chaining example Proof methods Proof methods divide into (roughly) two kinds:
Application of inference rules – Legitimate (sound) generation of new sentences from old – Proof = a sequence of inference rule applications Can use inference rules as operators in a standard search alg. – Typically require translation of sentences into a normal form Model checking truth table enumeration (always exponential in n) improved backtracking, e.g., Davis–Putnam–Logemann–Loveland (see book)
5:42
heuristic search in model space (sound but incomplete) e.g., min-conflicts-like hill-climbing algorithms 5:39
Forward chaining example
32
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
5:43
Forward chaining example
5:46
Forward chaining example
5:44
Forward chaining example
5:47
Forward chaining example
5:45
Forward chaining example
5:48
Forward chaining example
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
33
Backward chaining Idea: work backwards from the query q: to prove q by BC, check if q is known already, or prove by BC all premises of some rule concluding q Avoid loops: check if new subgoal is already on the goal stack Avoid repeated work: check if new subgoal 1) has already been proved true, or 2) has already failed 5:52
5:49
Backward chaining example
Forward chaining algorithm function PL-FC-E NTAILS ?(KB, q) returns true or false inputs: KB, the knowledge base, a set of propositional Horn clauses q, the query, a proposition symbol local variables: count, a table, indexed by clause, initially the number of premises inferred, a table, indexed by symbol, each entry initially false agenda, a list of symbols, initially the symbols known in KB while agenda is not empty do p ← P OP(agenda) unless inferred[p] do inferred[p] ← true for each Horn clause c in whose premise p appears do decrement count[c] if count[c] = 0 then do if H EAD[c] = q then return true P USH(H EAD[c], agenda) return false
5:53
5:50
Backward chaining example Proof of completeness FC derives every atomic sentence that is entailed by KB 1. FC reaches a fixed point where no new atomic sentences are derived 2. Consider the final state as a model m, assigning true/false to symbols 3. Every clause in the original KB is true in m Proof : Suppose a clause a1 ∧ . . . ∧ ak ⇒ b is false in m Then a1 ∧ . . . ∧ ak is true in m and b is false in m Therefore the algorithm has not reached a fixed point! 4. Hence m is a model of KB 5. If KB |= q, q is true in every model of KB, including m
5:54
General idea: construct any model of KB by sound inference, check α 5:51
Backward chaining example
34
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
5:55
Backward chaining example
5:58
Backward chaining example
5:56
Backward chaining example
5:59
Backward chaining example
5:57
Backward chaining example
5:60
Backward chaining example
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
35
where `i and mj are complementary literals. E.g., P1,3 ∨ P2,2 , P1,3
¬P2,2
Resolution is sound and complete for propositional logic 5:64
Conversion to CNF B1,1 ⇔ (P1,2 ∨ P2,1 ) 1. Eliminate ⇔, replacing α ⇔ β with (α ⇒ β) ∧ (β ⇒ α). (B1,1 ⇒ (P1,2 ∨ P2,1 )) ∧ ((P1,2 ∨ P2,1 ) ⇒ B1,1 ) 5:61
Backward chaining example
2. Eliminate ⇒, replacing α ⇒ β with ¬α ∨ β. (¬B1,1 ∨ P1,2 ∨ P2,1 ) ∧ (¬(P1,2 ∨ P2,1 ) ∨ B1,1 ) 3. Move ¬ inwards using de Morgan’s rules and double-negation: (¬B1,1 ∨ P1,2 ∨ P2,1 ) ∧ ((¬P1,2 ∧ ¬P2,1 ) ∨ B1,1 ) 4. Apply distributivity law (∨ over ∧) and flatten: (¬B1,1 ∨ P1,2 ∨ P2,1 ) ∧ (¬P1,2 ∨ B1,1 ) ∧ (¬P2,1 ∨ B1,1 )
5:65
Resolution algorithm 5:62
Proof by contradiction, i.e., show KB ∧ ¬α unsatisfiable function PL-R ESOLUTION(KB, α) returns true or false inputs: KB, the knowledge base, a sentence in propositional logic α, the query, a sentence in propositional logic
Forward vs. backward chaining FC is data-driven, cf. automatic, unconscious processing, e.g., object recognition, routine decisions May do lots of work that is irrelevant to the goal BC is goal-driven, appropriate for problem-solving, e.g., Where are my keys? How do I get into a PhD pro-
clauses ← the set of clauses in the CNF representation of KB ∧ ¬α new ← { } loop do for each Ci , Cj in clauses do resolvents ← PL-R ESOLVE(Ci , Cj ) if resolvents contains the empty clause then return true new ← new ∪ resolvents if new ⊆ clauses then return false clauses ← clauses ∪ new
gram? 5:66
Complexity of BC can be much less than linear in size of KB 5:63
Resolution
Resolution example KB = (B1,1 ⇔ (P1,2 ∨ P2,1 )) ∧ ¬B1,1
α = ¬P1,2
Conjunctive Normal Form (CNF—universal) conjunction of disjunctions of literals | {z } clauses E.g., (A ∨ ¬B) ∧ (B ∨ ¬C ∨ ¬D) Resolution inference rule (for CNF): complete for propositional logic 5:67
`1 ∨ · · · ∨ `k , m1 ∨ · · · ∨ mn `1 ∨ · · · ∨ `i−1 ∨ `i+1 ∨ · · · ∨ `k ∨ m1 ∨ · · · ∨ mj−1 ∨ mj+1 ∨ · · · ∨ mn
36
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Summary
Modus Ponens rule: complete for Horn KBs
α1 ,...,αn ,
α1 ∧···∧αn ⇒ β
Resolution rule: complete for propositional logic in CNF, let “`i = Logical agents apply inference to a knowledge base
¬mj ”:
to derive new information and make decisions
`1 ∨···∨`k , m1 ∨···∨mn `1 ∨···∨`i−1 ∨`i+1 ∨···∨`k ∨m1 ∨···∨mj−1 ∨mj+1 ∨···∨mn
Basic concepts of logic:
5:70
– syntax: formal structure of sentences – semantics: truth of sentences wrt models – entailment: necessary truth of one sentence given another – inference: deriving sentences from other sentences – soundness: derivations produce only entailed sentences – completeness: derivations can produce all entailed sentences Wumpus world requires the ability to represent partial and negated information, reason by cases, etc. Forward, backward chaining are linear-time, complete for Horn clauses Resolution is complete for propositional logic Propositional logic lacks expressive power 5:68
Dictionary: logic in general a logic: a language, elements α are sentences, (grammar example: slide 34) model m: a world/state description that allows us to evaluate α(m) ∈ {true, false} uniquely for any sentence α, M (α) = {m : α(m) = true} entailment α |= β:
M (α) ⊆ M (β), “∀m : α(m) ⇒ β(m)”
(Folgerung) equivalence α ≡ β: iff (α |= β and β |= α) KB: a set of sentences inference procedure i can infer α from KB: KB `i α soundness of i: KB `i α implies KB |= α
(Korrektheit)
completeness of i: KB |= α implies KB `i α 5:69
Dictionary: propositional logic conjunction: α ∧ β, disjunction: α ∨ β, negation: ¬α implication: α ⇒ β ≡ ¬α∨β, biconditional: α ⇔ β ≡ (α ⇒ β)∧ (β ⇒ α) Note: |= and ≡ are statements about sentences in a logic; ⇒ and ⇔ are symbols in the grammar of propositional logic α valid: true for any model, e.g.: KB |= α iff [(KB ⇒ α) is valid]
(allgemeingultig) ¨
α unsatisfiable: true for no model, e.g.: KB |= α iff [(KB ∧¬α) is unsatisfiable] literal: A or ¬A, clause: disjunction of literals, CNF: conjunction of clauses Horn clause: symbol | (conjunction of symbols ⇒ symbol), Horn form: conjunction of Horn clauses
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
8
37
First Order Logic
6:4
Syntax of FOL: Basic elements Constants Predicates Functions Variables Connectives Equality Quantifiers
Outline • Why FOL? • Syntax and semantics of FOL • Example sentences
KingJohn, 2, U CB, . . . Brother, >, . . . Sqrt, Lef tLegOf, . . . x, y, a, b, . . . ∧ ∨ ¬ ⇒ ⇔ = ∀∃ 6:5
• Wumpus world in FOL 6:1
Pros and cons of propositional logic
First Order Logic: Syntax grammar hsentencei
→
hatomic sentencei | hcomplex sentencei | [∀ | ∃] hvariablei hsentencei
hatomic sentencei
→
predicate(htermi,. . . ) | htermi=htermi
htermi
→
function(htermi,. . . ) | constant | variable
hcomplex sentencei
→
¬ hsentencei | (hsentencei [∧ | ∨ | ⇒ | ⇔ ] hsente
Pros: Propositional logic is declarative: pieces of syntax correspond to facts Propositional logic allows partial/disjunctive/negated information (unlike most data structures and databases) Propositional logic is compositional: meaning of B1,1 ∧ P1,2 is derived from meaning of B1,1 and of P1,2
6:6
Meaning in propositional logic is context-independent (unlike natural language, where meaning depends on context)
Universal quantification ∀ hvariablesi hsentencei
Cons:
Everyone at Berkeley is smart:
Propositional logic has very limited expressive power
(unlike
natural language)
∀ x At(x, Berkeley) ⇒ Smart(x) ∀x P
E.g., cannot say “pits cause breezes in adjacent squares”
is true in a model m iff P is true with x being
each possible object in the model
except by writing one sentence for each square
6:7 6:2
Existential quantification
First-order logic
∃ hvariablesi hsentencei
Whereas propositional logic assumes world contains facts, first-order logic (like natural language) assumes the world con-
Someone at Stanford is smart: ∃ x At(x, Stanf ord) ∧ Smart(x)
tains
∃x P • Objects: people, houses, numbers, theories, Ronald Mc-
is true in a model m iff P is true with x being
some possible object in the model
Donald, colors, baseball games, wars, centuries . . .
6:8
• Relations: red, round, bogus, prime, multistoried . . ., brother of, bigger than, inside, part of, has color, occurred after, owns, comes between, . . .
Properties of quantifiers ∀ x ∀ y is the same as ∀ y ∀ x
• Functions: father of, best friend, third inning of, one more than, end of . . .
∃ x ∃ y is the same as ∃ y ∃ x ∃ x ∀ y is not the same as ∀ y ∃ x
6:3
∃ x ∀ y Loves(x, y) “There is a person who loves everyone in the world”
Logics in general Language
Ontological Commitment
Epistemological Commitment
Propositional logic First-order logic Temporal logic Probability theory Fuzzy logic
facts facts, objects, relations facts, objects, relations, times facts facts + degree of truth
true/false/unknown true/false/unknown true/false/unknown degree of belief known interval value
∀ y ∃ x Loves(x, y) “Everyone in the world is loved by at least one person” Quantifier duality: each can be expressed using the other ∀ x Likes(x, IceCream) ∃ x Likes(x, Broccoli)
¬∃ x ¬Likes(x, IceCream) ¬∀ x ¬Likes(x, Broccoli)
38
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Example sentences
6:9
Brothers are siblings
Truth in first-order logic
∀ x, y Brother(x, y) ⇒ Sibling(x, y). “Sibling” is symmetric
Sentences are true with respect to a model and an interpretation
6:14
Model contains ≥ 1 objects (domain elements) and relations among them
Example sentences
Interpretation specifies referents for constant symbols → objects (domain elements)
Brothers are siblings
predicate symbols → relations
∀ x, y Brother(x, y) ⇒ Sibling(x, y).
function symbols → functional relations
“Sibling” is symmetric
An atomic sentence predicate(term1 , . . . , termn ) is true iff the objects referred to by term1 , . . . , termn
∀ x, y Sibling(x, y) ⇔ Sibling(y, x).
are in the relation referred to by predicate
One’s mother is one’s female parent 6:15
6:10
Example sentences
Models for FOL: Example
Brothers are siblings ∀ x, y Brother(x, y) ⇒ Sibling(x, y). “Sibling” is symmetric ∀ x, y Sibling(x, y) ⇔ Sibling(y, x). One’s mother is one’s female parent ∀ x, y M other(x, y) ⇔ (F emale(x) ∧ P arent(x, y)). A first cousin is a child of a parent’s sibling 6:16
Example sentences Brothers are siblings ∀ x, y Brother(x, y) ⇒ Sibling(x, y).
6:11
“Sibling” is symmetric ∀ x, y Sibling(x, y) ⇔ Sibling(y, x).
Models for FOL: Lots!
One’s mother is one’s female parent Entailment in propositional logic can be computed by enumerat-
∀ x, y M other(x, y) ⇔ (F emale(x) ∧ P arent(x, y)).
ing models
A first cousin is a child of a parent’s sibling
We can enumerate the FOL models for a given KB vocabulary:
∀ x, y F irstCousin(x, y) ⇔ ∃ p, ps P arent(p, x)∧Sibling(ps, p)∧
For each number of domain elements n from 1 to ∞
P arent(ps, y)
For each k-ary predicate Pk in the vocabulary
6:17
For each possible k-ary relation on n objects For each constant symbol C in the vocabulary For each choice of referent for C from n objects . . .
8.1
FOL description of interactive domains
Computing entailment by enumerating FOL models is not easy! 6:18
6:12
Knowledge base for the wumpus world
Example sentences
“Perception”
Brothers are siblings
∀ b, g, t P ercept([Smell, b, g], t) ⇒ Smelt(t) 6:13
∀ s, b, t P ercept([s, b, Glitter], t) ⇒ AtGold(t) Reflex: ∀ t AtGold(t) ⇒ Action(Grab, t)
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 Reflex with internal state: do we have the gold already?
39
Each axiom is “about” a predicate (not an action per se):
∀ t AtGold(t) ∧ ¬Holding(Gold, t) ⇒ Action(Grab, t) P true afterwards
Holding(Gold, t) cannot be observed
⇔
[an action made P true
∨
P true already and no action made P false
⇒ keeping track of change is essential 6:19
For holding the gold:
Deducing hidden properties
∀ a, s Holding(Gold, Result(a, s)) ⇔ Properties of locations:
[(a = Grab ∧ AtGold(s))
∀ x, t At(Agent, x, t) ∧ Smelt(t) ⇒ Smelly(x)
∨ (Holding(Gold, s) ∧ a 6= Release)]
∀ x, t At(Agent, x, t) ∧ Breeze(t) ⇒ Breezy(x)
6:23
Squares are breezy near a pit: ∀ y Breezy(y) ⇔ [∃ x P it(x) ∧ Adjacent(x, y)]
Planning Domain Definition Language (PDDL)
Implies two rules:
The Situation Calculus is very general, but not concise
Diagnostic rule—infer cause from effect
The AI community developed as simpler format (based on STRIPS)
∀ y Breezy(y) ⇒ ∃ x P it(x) ∧ Adjacent(x, y)
for the 1998/2000 International Planning Competition (IPC)
Causal rule—infer effect from cause ∀ x, y P it(x) ∧ Adjacent(x, y) ⇒ Breezy(y) 6:20
Keeping track of change: Situation Calculus Facts hold in situations, rather than eternally E.g., Holding(Gold, N ow) rather than just Holding(Gold) Situation calculus is one way to represent change in FOL: Adds a situation argument to each non-eternal predicate 6:24
E.g., N ow in Holding(Gold, N ow) denotes a situation Situations are connected by the Result function
PDDL
Result(a, s) is the situation that results from doing a in s The precondition specifies if an action predicate is applicable in a given situation The effect determines the changed facts
Frame assumption: All facts not mentioned in the effect remain unchanged. 6:21
Describing actions I: Frame problem
The majority of state-of-the-art AI planners use this format FFplan: (B. Nebel, Freiburg) a forward chaining heuristic
“Effect” axiom—describe changes due to action
state space planner
∀ s AtGold(s) ⇒ Holding(Gold, Result(Grab, s)) For probabilistic versions of PDDL: T. Lang and M. Toussaint: Planning with noisy probabilistic relational rules. JAIR, 2010.
“Frame” axiom—describe non-changes due to action ∀ s HaveArrow(s) ⇒ HaveArrow(Result(Grab, s))
6:25
Frame problem: find an elegant way to handle non-change (a) representation—avoid frame axioms
Planning as FOL inference
(b) inference—avoid repeated “copy-overs” to keep track of state
A general approach to planning is to query the KB for a plan that 6:22
Describing actions II Successor-state axioms solve the representational frame problem
fulfills a goal condition
There is debate and ongoing research on this versus fwd search 6:26
40
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Substitution Suppose a wumpus-world agent is using an FOL KB and perceives a smell and a breeze (but no glitter) at s: T ELL(KB, P ercept([Smell, Breeze, N one], s)) A SK(KB, ∃ a Action(a, s)) I.e., does KB entail any particular actions at s? Answer: Y es, {a/Shoot}
← substitution (binding list)
Given a sentence S and a substitution σ, Sσ denotes the result of plugging σ into S; e.g., S = Smarter(x, y) σ = {x/Hillary, y/Bill} Sσ = Smarter(Hillary, Bill) A SK (KB,S) returns some/all σ such that KB |= Sσ 6:27
Making plans as inference over plans Represent plans as action sequences [a1 , a2 , . . . , an ] P lanResult(p, s) is the result of executing p in s Then the query A SK(KB, ∃ p Holding(Gold, P lanResult(p, S0 ))) has the solution {p/[F orward, Grab]} Definition of P lanResult in terms of Result: ∀ s P lanResult([ ], s) = s ∀ a, p, s P lanResult([a|p], s) = P lanResult(p, Result(a, s)) 6:28
Summary First-order logic: – objects and relations are semantic primitives – syntax: constants, functions, predicates, equality, quantifiers Increased expressive power: sufficient to define wumpus world Situation calculus: – convention for describing actions and change in FOL – can formulate planning as inference on a situation calculus KB Planning Domain Definition Language (PDDL): – more common restricted language – more concise because of the frame assumption – directly lends to forward chaining methods (like FFplan) 6:29
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
9
First Order Logic – Inference
41
Existential instantiation contd. UI can be applied several times to add new sentences; the new KB is logically equivalent to the old
Outline
EI can be applied once to replace the existential sentence; the new KB is not equivalent to the old,
• Reducing first-order inference to propositional inference
but is satisfiable iff the old KB was satisfiable
• Unification
7:5
• Generalized Modus Ponens
Reduction to propositional inference
• Forward and backward chaining • Resolution
Suppose the KB contains just the following: 7:1
∀ x King(x) ∧ Greedy(x) ⇒ Evil(x) King(John) Greedy(John) Brother(Richard, John)
A brief history of reasoning 450B . C. 322B . C. 1565 1847 1879 1922 1930 1930 1931 1960 1965
Stoics Aristotle Cardano Boole Frege Wittgenstein ¨ Godel Herbrand ¨ Godel Davis/Putnam Robinson
propositional logic, inference (maybe) “syllogisms” (inference rules), quantifiers probability theory (propositional logic + uncertainty) propositional logic (again) first-order logic proof by truth tables ∃ complete algorithm for FOL complete algorithm for FOL (reduce to propositional) ¬∃ complete algorithm for arithmetic “practical” algorithm for propositional logic “practical” algorithm for FOL—resolution
Instantiating the universal sentence in all possible ways, we have King(John) ∧ Greedy(John) ⇒ Evil(John) King(Richard) ∧ Greedy(Richard) ⇒ Evil(Richard) King(John) Greedy(John) Brother(Richard, John)
7:2
The new KB is propositionalized: proposition symbols are King(Joh
Universal instantiation (UI)
7:6
Every instantiation of a universally quantified sentence is en-
Reduction contd.
tailed by it: ∀v α S UBST({v/g}, α)
Idea: propositionalize KB and query, apply resolution, return result
for any variable v and ground term g
Problem: with function symbols, there are infinitely many ground 7:3
terms, e.g., F ather(F ather(F ather(John)))
Existential instantiation (EI)
Theorem: Herbrand (1930). If a sentence α is entailed by an FOL KB,
For any sentence α, variable v, and constant symbol k
it is entailed by a finite subset of the propositional KB
that does not appear elsewhere in the knowledge base:
Idea: For n = 0 to ∞ do
∃v α S UBST({v/k}, α)
create a propositional KB by instantiating with depth-n terms see if α is entailed by this KB Problem: works if α is entailed, loops if α is not entailed Theorem: Turing (1936), Church (1936), entailment in FOL is
E.g., ∃ x Crown(x) ∧ OnHead(x, John) yields
semidecidable Crown(C1 ) ∧ OnHead(C1 , John)
7:7
provided C1 is a new constant symbol, called a Skolem constant
Problems with propositionalization
Another example: from ∃ x d(xy )/dy = xy we obtain Propositionalization seems to generate lots of irrelevant sentences.
d(ey )/dy = ey
E.g., from provided e is a new constant symbol 7:4
∀ x King(x) ∧ Greedy(x) ⇒ Evil(x) King(John) ∀ y Greedy(y) Brother(Richard, John)
42
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 it seems obvious that Evil(John), but propositionalization pro-
Unification
duces lots of facts such as Greedy(Richard) that are irrelevant With p k-ary predicates and n constants, there are p · nk instan-
We can get the inference immediately if we can find a substitution θ
tiations
such that King(x) and Greedy(x) match King(John) and Greedy(y
With function symbols, it gets much much worse! 7:8
θ = {x/John, y/John} works U NIFY(α, β) = θ if αθ = βθ
Unification We can get the inference immediately if we can find a substitution θ such that King(x) and Greedy(x) match King(John) and Greedy(y) θ = {x/John, y/John} works
p Knows(John, x) Knows(John, x) Knows(John, x) Knows(John, x)
q Knows(John, Jane) Knows(y, OJ) Knows(y, M other(y)) Knows(x, OJ)
θ {x/Jane} {x/OJ, y/John} {y/John, x/M other(
U NIFY(α, β) = θ if αθ = βθ 7:12
p Knows(John, x) Knows(John, x) Knows(John, x) Knows(John, x)
q Knows(John, Jane) Knows(y, OJ) Knows(y, M other(y)) Knows(x, OJ)
θ
Unification We can get the inference immediately if we can find a substitution θ 7:9
such that King(x) and Greedy(x) match King(John) and Greedy(y θ = {x/John, y/John} works
Unification
U NIFY(α, β) = θ if αθ = βθ We can get the inference immediately if we can find a substitution θ such that King(x) and Greedy(x) match King(John) and Greedy(y) θ = {x/John, y/John} works U NIFY(α, β) = θ if αθ = βθ
p Knows(John, x) Knows(John, x) Knows(John, x) Knows(John, x)
q Knows(John, Jane) Knows(y, OJ) Knows(y, M other(y)) Knows(x, OJ)
θ {x/Jane}
p Knows(John, x) Knows(John, x) Knows(John, x) Knows(John, x)
q Knows(John, Jane) Knows(y, OJ) Knows(y, M other(y)) Knows(x, OJ)
θ {x/Jane} {x/OJ, y/John} {y/John, x/M other( f ail
Standardizing apart eliminates overlap of variables, e.g., Knows(z17 7:13
Generalized Modus Ponens (GMP) 7:10
Unification
p1 0 , p2 0 , . . . , pn 0 , (p1 ∧ p2 ∧ . . . ∧ pn ⇒ q) qθ
where pi 0 θ = pi θ for all i
We can get the inference immediately if we can find a substitution θ
p1 0 is King(John) p2 0 is Greedy(y) θ is {x/John, y/John} qθ is Evil(John)
such that King(x) and Greedy(x) match King(John) and Greedy(y) θ = {x/John, y/John} works U NIFY(α, β) = θ if αθ = βθ
p Knows(John, x) Knows(John, x) Knows(John, x) Knows(John, x)
q Knows(John, Jane) Knows(y, OJ) Knows(y, M other(y)) Knows(x, OJ)
θ {x/Jane} {x/OJ, y/John}
p1 is King(x) p2 is Greedy(x) q is Evil(x)
GMP used with KB of definite clauses (exactly one positive literal) All variables assumed universally quantified 7:14
7:11
Forward chaining algorithm
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
43
Resolution: brief summary Full first-order version: function FOL-FC-A SK(KB, α) returns a substitution or false repeat until new is empty new ← { } for each sentence r in KB do ( p 1 ∧ . . . ∧ p n ⇒ q) ← S TANDARDIZE -A PART(r) for each θ such that (p 1 ∧ . . . ∧ p n )θ = (p 01 ∧ . . . ∧ p 0n )θ for some p 01 , . . . , p 0n in KB q 0 ← S UBST(θ, q) if q 0 is not a renaming of a sentence already in KB or new then do add q 0 to new φ ← U NIFY(q 0 , α) if φ is not fail then return φ add new to KB return false
`1 ∨ · · · ∨ `k , m1 ∨ · · · ∨ mn (`1 ∨ · · · ∨ `i−1 ∨ `i+1 ∨ · · · ∨ `k ∨ m1 ∨ · · · ∨ mj−1 ∨ mj+1 ∨ · · · ∨ mn )θ
where U NIFY(`i , ¬mj ) = θ. For example, ¬Rich(x) ∨ U nhappy(x) Rich(Ken) U nhappy(Ken) with θ = {x/Ken} Apply resolution steps to CN F (KB ∧ ¬α); complete for FOL 7:19
7:15
Conversion to CNF
Properties of forward chaining
Everyone who loves all animals is loved by someone: Sound and complete for first-order definite clauses
∀ x [∀ y Animal(y) ⇒ Loves(x, y)] ⇒ [∃ y Loves(y, x)]
(proof similar to propositional proof)
1. Eliminate biconditionals and implications
Datalog = first-order definite clauses + no functions (e.g., crime ∀ x [¬∀ y ¬Animal(y) ∨ Loves(x, y)] ∨ [∃ y Loves(y, x)]
KB) k
FC terminates for Datalog in poly iterations: at most p·n literals 2. Move ¬ inwards: ¬∀ x, p ≡ ∃ x ¬p, ¬∃ x, p ≡ ∀ x ¬p:
May not terminate in general if α is not entailed This is unavoidable: entailment with definite clauses is semidecidable
∀ x [∃ y ¬(¬Animal(y) ∨ Loves(x, y))] ∨ [∃ y Loves(y, x)] ∀ x [∃ y ¬¬Animal(y) ∧ ¬Loves(x, y)] ∨ [∃ y Loves(y, x)] ∀ x [∃ y Animal(y) ∧ ¬Loves(x, y)] ∨ [∃ y Loves(y, x)]
7:16 7:20
Backward chaining algorithm Conversion to CNF contd. function FOL-BC-A SK(KB, goals, θ) returns a set of substitutions inputs: KB, a knowledge base goals, a list of conjuncts forming a query (θ already applied) θ, the current substitution, initially the empty substitution { } local variables: answers, a set of substitutions, initially empty if goals is empty then return {θ} q 0 ← S UBST(θ, F IRST(goals)) for each sentence r in KB where S TANDARDIZE -A PART(r) = ( p 1 ∧ . . . ∧ p n ⇒ q) and θ0 ← U NIFY(q, q 0 ) succeeds new goals ← [ p 1 , . . . , p n |R EST(goals)] answers ← FOL-BC-A SK(KB, new goals, C OMPOSE(θ0 , θ)) ∪ answers return answers
3. Standardize variables: each quantifier should use a different one ∀ x [∃ y Animal(y) ∧ ¬Loves(x, y)] ∨ [∃ z Loves(z, x)] 4. Skolemize: a more general form of existential instantiation. Each existential variable is replaced by a Skolem function of the enclosing universally quantified variables: ∀ x [Animal(F (x)) ∧ ¬Loves(x, F (x))] ∨ Loves(G(x), x)
7:17
5. Drop universal quantifiers:
Properties of backward chaining
[Animal(F (x)) ∧ ¬Loves(x, F (x))] ∨ Loves(G(x), x)
Depth-first recursive proof search: space is linear in size of proof
6. Distribute ∧ over ∨:
Incomplete due to infinite loops ⇒ fix by checking current goal against every goal on stack
[Animal(F (x))∨Loves(G(x), x)]∧[¬Loves(x, F (x))∨Loves(G(x), x
Inefficient due to repeated subgoals (both success and failure) ⇒ fix using caching of previous results (extra space!)
7:21
Widely used (without improvements!) for logic programming 7:18
44
10
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Probabilities
• Utilities, decision theory, entropy, KLD 8:3
Probability: Frequentist and Bayesian Objective Probability • Frequentist probabilities are defined in the limit of an infinite number of trials Example: “The probability of a particular coin landing heads up is 0.43”
The double slit experiment: x
• Bayesian (subjective) probabilities quantify degrees of belief Example: “The probability of it raining tomorrow is 0.3” – Not possible to repeat “tomorrow”
θ
8:4
P
10.1
Basic definitions 8:5
Probabilities & Sets • Sample Space/domain O, e.g. O = {1, 2, 3, 4, 5, 6}
• Probability P : A ⊂ O 7→ [0, 1] e.g., P ({1}) = 16 , P ({4}) = 16 , P ({2, 5}) = 31 , 8:1
• Axioms: ∀A, B ⊆ O
Probability Theory
– Nonnegativity P (A) ≥ 0 – Additivity P (A ∪ B) = P (A) + P (B) if A ∩ B = { }
• Why do we need probabilities?
– Normalization P (O) = 1 – Obvious: to express inherent (objective) stochasticity of the world
• Implications 0 ≤ P (A) ≤ 1
• But beyond this: (also in a “deterministic world”): – lack of knowledge!
P ({ }) = 0 A ⊆ B ⇒ P (A) ≤ P (B)
– hidden (latent) variables P (A ∪ B) = P (A) + P (B) − P (A ∩ B)
– expressing uncertainty
P (O \ A) = 1 − P (A)
– expressing information (and lack of information) – Subjective Probability
8:6
Probabilities & Random Variables • Probability Theory: an information calculus 8:2
• For a random variable X with discrete domain dom(X) = O we write:
Outline • Basic definitions – Random variables – joint, conditional, marginal distribution – Bayes’ theorem • Probability distributions: – Binomial & Beta – Multinomial & Dirichlet – Conjugate priors – Gauss – Dirak & Particles
∀x∈O : 0 ≤ P (X = x) ≤ 1 P P (X = x) = 1 x∈O Example: A dice can take values O = {1, .., 6}. X is the random variable of a dice throw. P (X = 1) ∈ [0, 1] is the probability that X takes value 1.
• A bit more formally: a random variable is a map from a measureable space to a domain (sample space) and thereby introduces a probability measure on the domain (“assigns a probability to each possible value”) 8:7
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
45
Probabilty Distributions likelihood · prior normalization
posterior =
• P (X = 1) ∈ R denotes a specific probability
8:11
P (X) denotes the probability distribution (function over O)
Multiple RVs: Example: A dice can take values O = {1, 2, 3, 4, 5, 6}. By P (X) we discribe the full distribution over possible values {1, .., 6}. These are 6 numbers that sum to one, usually stored in a table, e.g.: [ 16 , 16 , 61 , 16 , 16 , 16 ]
• Analogously for n random variables X1:n (stored as a rank n tensor) Joint:
• In implementations we typically represent distributions over discrete random variables as tables (arrays) of numbers
P (X1:n )
Marginal:
P (X1:n ),
X2:n
P (X1 |X2:n ) =
Conditional:
• Notation for summing over a RV:
P
P (X1 ) =
P (X1:n ) P (X2:n )
• X is conditionally independent of Y given Z iff:
In equation we often needP to sum over RVs. We then write X P (X) · · · P as shorthand for the explicit notation x∈dom(X) P (X = x) · · ·
P (X|Y, Z) = P (X|Z)
8:8
Joint distributions
• Product rule and Bayes’ Theorem: P (X1:n ) =
Qn
i=1
P (X, Z, Y ) P (X|Y, Z) P (Y |Z) P (Z)
P (Xi |Xi+1:n )
P (X1 |X2:n ) = P (X2 |X1 ,X3:n ) P (X1 |X3:n ) P (X2 |X3:n )
Assume we have two random variables X and Y
=
P (X|Y, Z) =
P (Y |X,Z) P (X|Z) P (Y |Z)
P (X, Y |Z) =
P (X,Z|Y ) P (Y ) P (Z)
8:12
• Definitions: Joint:
P (X, Y )
Marginal:
10.2
P (X) =
Conditional:
P
Y
Probability distributions
P (X, Y )
P (X|Y ) =
P (X,Y ) P (Y )
The conditional is normalized: ∀Y :
P
X
Bishop, C. M.: Pattern Recognition and Machine Learning. Springer, 2006 http://research. microsoft.com/en-us/um/ people/cmbishop/prml/
P (X|Y ) = 1
• X is independent of Y iff: P (X|Y ) = P (X) (table thinking: all columns of P (X|Y ) are equal) 8:9
Joint distributions
8:13
Bernoulli & Binomial
joint: P (X, Y ) P marginal: P (X) = Y P (X, Y ) conditional:
P (X|Y ) =
• We have a binary random variable x ∈ {0, 1}
P (X,Y ) P (Y )
{0, 1}) The Bernoulli distribution is parameterized by a single scalar µ,
• Implications of these definitions: Product rule:
(i.e. dom(x) =
P (x = 1 | µ) = µ ,
P (X, Y ) = P (X|Y ) P (Y ) = P (Y |X) P (X)
P (x = 0 | µ) = 1 − µ
Bern(x | µ) = µx (1 − µ)1−x Bayes’ Theorem:
P (X|Y ) =
P (Y |X) P (X) P (Y )
• We have a data set of random variables D = {x1 , .., xn }, each xi ∈ {0, 1}. If each xi ∼ Bern(xi | µ) we have 8:10
P (D | µ) =
Bayes’ Theorem
Qn
i=1
argmax log P (D | µ) = argmax µ
P (X|Y ) =
Bern(xi | µ) =
P (Y |X) P (X) P (Y )
µ
n X
Qn
i=1
µxi (1 − µ)1−xi
xi log µ + (1 − xi ) log(1 − µ) =
i=1
• The Binomial distribution is the distribution over the count m = Pn i=1 xi
n µm (1 − µ)n−m , Bin(m | n, µ) = m
n! n = (n − m)! m! m
1X ni
46
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 8:14
Beta
Multinomial • We have an integer random variable x ∈ {1, .., K}
The probability of a single x can be parameterized by µ = (µ1 , .., µK )
How to express uncertainty over a Bernoulli parameter µ • The Beta distribution is over the interval [0, 1], typically the pa-
P (x = k | µ) = µk
rameter µ of a Bernoulli: with the constraint
1 Beta(µ | a, b) = µa−1 (1 − µ)b−1 B(a, b) with mean hµi =
a a+b
and mode µ∗ =
a−1 a+b−2
PK
k=1
µk = 1 (probabilities need to be nor-
malized) • We have a data set of random variables D = {x1 , .., xn }, each
for a, b > 1
xi ∈ {1, .., K}. If each xi ∼ P (xi | µ) we have
• The crucial point is: – Assume we are in a world with a “Bernoulli source” (e.g., binary bandit), but don’t know its parameter µ – Assume we have a prior distribution P (µ) = Beta(µ | a, b)
P (D | µ) = where mk =
i=1
Pn
µx i =
i=1 [xi
Qn
i=1
[x =k]
QK
k=1
µk i
=
QK
k=1
m
µk k
= k] is the count of [xi = k]. The ML
estimator is
– Assume we collected some P data D = {x1 , .., xn }, Pxi ∈ {0, 1}, with counts aD = i xi of [xi = 1] and bD = i (1 − xi ) of [xi = 0] – The posterior is
Qn
argmax log P (D | µ) = µ
1 (m1 , .., mK ) n
• The Multinomial distribution is this distribution over the counts
P (D | µ) P (µ) ∝ Bin(D | µ) Beta(µ | a, b) P (µ | D) = P (D)
mk
∝ µaD (1 − µ)bD µa−1 (1 − µ)b−1 = µa−1+aD (1 − µ)b−1+bD
Mult(m1 , .., mK | n, µ) ∝
QK
k=1
m
µk k
= Beta(µ | a + aD , b + bD ) 8:18 8:15
Dirichlet
Beta
How to express uncertainty over a Multinomial parameter µ The prior is Beta(µ | a, b), the posterior is Beta(µ | a + aD , b + bD )
• Conclusions: – The semantics of a and b are counts of [xi = 1] and [xi = 0], respectively – The Beta distribution is conjugate to the Bernoulli (explained later) – With the Beta distribution we can represent beliefs (state of knowledge) about uncertain µ ∈ [0, 1] and know how to update this belief given data 8:16
• The Dirichlet distribution is over the K-simplex, that is, over P µ1 , .., µK ∈ [0, 1] subject to the constraint K k=1 µk = 1: Dir(µ | α) ∝
QK
k=1
α −1
µk k
It is parameterized by α = (α1 , .., αK ), has mean hµi i = and mode µ∗i =
Pαi −1 j αj −K
Pαi j αj
for ai > 1.
• The crucial point is: – Assume we are in a world with a “Multinomial source” (e.g., an integer bandit), but don’t know its parameter µ – Assume we have a prior distribution P (µ) = Dir(µ | α) – Assume we collected someP data D = {x1 , .., xn }, xi ∈ {1, .., K}, with counts mk = i [xi = k]
Beta
– The posterior is P (D | µ) P (µ) ∝ Mult(D | µ) Dir(µ | a, b) P (D) QK Q m QK αk −1 αk −1+mk ∝ k=1 µk k = K k=1 µk k=1 µk
P (µ | D) =
= Dir(µ | α + m) 8:19
Dirichlet
from Bishop
The prior is Dir(µ | α), the posterior is Dir(µ | α + m)
8:17
• Conclusions:
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
47
Conjugate priors
– The semantics of α is the counts of [xi = k] – The Dirichlet distribution is conjugate to the Multinomial – With the Dirichlet distribution we can represent beliefs (state of knowledge) about uncertain µ of an integer random variable and know how to update this belief given data
likelihood Binomial Bin(D | µ) Multinomial Mult(D | µ) Gauss N(x | µ, Σ) 1D Gauss N(x | µ, λ-1 ) nD Gauss N(x | µ, Λ-1 ) nD Gauss N(x | µ, Λ-1 )
8:20
Dirichlet Illustrations for α = (0.1, 0.1, 0.1), α = (1, 1, 1) and α = (10, 10, 10):
conjugate Beta Beta(µ | a, b) Dirichlet Dir(µ | α) Gauss N(µ | µ0 , A) Gamma Gam(λ | a, b) Wishart Wish(Λ | W, ν) Gauss-Wishart N(µ | µ0 , (βΛ)-1 ) Wish(Λ | W, ν) 8:24
from Bishop
Distributions over continuous domain
8:21
Motivation for Beta & Dirichlet distributions • Bandits: – If we have binary [integer] bandits, the Beta [Dirichlet] distribution is a way to represent and update beliefs
8:25
Distributions over continuous domain • Let x be a continuous RV. The probability density function (pdf) p(x) ∈ [0, ∞) defines the probability
– The belief space becomes discrete: The parameter α of the prior is continuous, but the posterior updates live on a discrete “grid” (adding counts to α)
b
Z P (a ≤ x ≤ b) =
p(x) dx ∈ [0, 1] a
– We can in principle do belief planning using this • Reinforcement Learning: – Assume we know that the world is a finite-state MDP, but do not know its transition probability P (s0 | s, a). For each (s, a), P (s0 | s, a) is a distribution over the integer s0 – Having a separate Dirichlet distribution for each (s, a) is a way to represent our belief about the world, that is, our belief about P (s0 | s, a) – We can in principle do belief planning using this → Bayesian Reinforcement Learning • Dirichlet distributions are also used to model texts (word distributions in text), images, or mixture distributions in general
The (cumulative) probability distribution F (y) = P (x ≤ y) = Ry dx p(x) ∈ [0, 1] is the cumulative integral with limy→∞ F (y) = −∞ 1 (In discrete domain: probability distribution and probability mass function P (x) ∈ [0, 1] are used synonymously.)
• Two basic examples: Gaussian: N(x | µ, Σ) =
>
1
e− 2 (x−µ)
1 | 2πΣ | 1/2
Σ-1 (x−µ)
Dirac or d (“point particle”) d(x) = 0 except at x = 0, 8:22
Conjugate priors
R
d(x) dx =
1 d(x) =
∂ H(x) ∂x
where H(x) = [x ≥ 0] = Heavyside step func-
tion 8:26
• Assume you have data D = {x1 , .., xn } with likelihood
Gaussian distribution
P (D | θ)
N (x|µ, σ 2 )
that depends on an uncertain parameter θ
2σ
• 1-dim: N(x | µ, σ 2 ) =
Assume you have a prior P (θ)
1 | 2πσ 2 | 1/2
1
2
e− 2 (x−µ)
/σ 2 µ
• n-dim Gaussian in normal form: • The prior P (θ) is conjugate to the likelihood P (D | θ) iff the posterior P (θ | D) ∝ P (D | θ) P (θ)
N(x | µ, Σ) =
1 1 exp{− (x − µ)> Σ-1 (x − µ)} 2 | 2πΣ | 1/2
with mean µ and covariance matrix Σ. In canonical form:
is in the same distribution class as the prior P (θ) N[x | a, A] =
exp{− 12 a>A-1 a} 1 exp{− x> A x + x>a} 2 | 2πA-1 | 1/2
(1)
• Having a conjugate prior is very convenient, because then you with precision matrix A = Σ-1 and coefficient a = Σ-1 µ (and
know how to update the belief given data 8:23
mean µ = A-1 a).
48
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
• Gaussian identities: see http://ipvs.informatik.uni-stuttgart.
Motivation for particle distributions
de/mlr/marc/notes/gaussians.pdf 8:27
• Numeric representation of “difficult” distributions – Very general and versatile – But often needs many samples
Motivation for Gaussian distributions
• Distributions over games (action sequences), sample based planning, MCTS
• Gaussian Bandits
• State estimation, particle filters
• Control theory, Stochastic Optimal Control • State estimation, sensor processing, Gaussian filtering (Kalman
• etc 8:31
filtering) • Machine Learning
Utilities & Decision Theory
• etc 8:28
• Given a space of events O (e.g., outcomes of a trial, a game, etc) the utility is a function
Particle Approximation of a Distribution
U : O→R
• We approximate a distribution p(x) over a continuous domain Rn
• The utility represents preferences as a single scalar – which is not always obvious (cf. multi-objective optimization)
• A particle distribution q(x) is a weighed set S = {(x , w i
i
)}N i=1
of
N particles – each particle has a “location” xi ∈ Rn and a weight wi ∈ R P – weights are normalized, i wi = 1 q(x) :=
N X
wi d(x − xi )
• Decision Theory making decisions (that determine p(x)) that maximize expected utility Z E{U }p =
U (x) p(x) x
• Concave utility functions imply risk aversion (and convex, risk-
i=1
taking)
where d(x − xi ) is the d-distribution.
8:32
• Given weighted particles, we can estimate for any (smooth) f : Z hf (x)ip =
f (x)p(x)dx ≈
PN
i=1
wi f (xi )
Entropy • The neg-log (− log p(x)) of a distribution reflects something like
x
See An Introduction to MCMC for Machine Learning www.cs. ubc.ca/˜nando/papers/mlintro.pdf
“error”: – neg-log of a Guassian ↔ squared error – neg-log likelihood ↔ prediction error
8:29
• The (− log p(x)) is the “optimal” coding length you should assign
Particle Approximation of a Distribution
to a symbol x. This will minimize the expected length of an encoding
Histogram of a particle representation:
Z H(p) =
p(x)[− log p(x)] x
• The entropy H(p) = Ep(x) {− log p(x)} of a distribution p is a measure of uncertainty, or lack-of-information, we have about x 8:33
Kullback-Leibler divergence* • Assume you use a “wrong” distribution q(x) to decide on the coding length of symbols drawn from p(x). The expected length of a encoding is Z 8:30
p(x)[− log q(x)] ≥ H(p) x
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 • The difference D p q =
Z p(x) log x
p(x) ≥0 q(x)
is called Kullback-Leibler divergence Proof of inequality, using the Jenson inequality: Z −
p(x) log x
q(x) ≥ − log p(x)
Z p(x) x
q(x) =0 p(x) 8:34
Some more continuous distributions* Gaussian Dirac or d Student’s t (=Gaussian for ν → ∞, otherwise heavy tails)
Exponential
N(x | a, A) > -1 1 1 e− 2 (x−a) A | 2πA | 1/2
= (x−a)
∂ d(x) = ∂x H(x) ν+1 2 p(x; ν) ∝ [1 + xν ]− 2
p(x; λ) = [x ≥ 0] λe−λx
(distribution over single event time)
Laplace (“double exponential”)
Chi-squared Gamma
p(x; µ, b) =
1 − | x−µ | /b e 2b
p(x; k) ∝ [x ≥ 0] xk/2−1 e−x/2 p(x; k, θ) ∝ [x ≥ 0] xk−1 e−x/θ 8:35
49
50
11
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Bandits & UCT
Bandits: Formal Problem Definition • Let at ∈ {1, .., n} be the choice of machine at time t Let yt ∈ R be the outcome
Multi-armed Bandits • A policy or strategy maps all the history to a new choice: π : [(a1 , y1 ), (a2 , y2 ), ..., (at-1 , yt-1 )] 7→ at
• Problem: Find a policy π that maxh
PT
t=1
yt i
or maxhyT i
• There are n machines • Each machine i returns a reward y ∼ P (y; θi ) The machine’s parameter θi is unknown
or other objectives like discounted infinite horizon maxh
P∞
t=1
γ t yt i
• Your goal is to maximize the reward, say, collected over the first T trials 9:1
9:5
Exploration, Exploitation • “Two effects” of choosing a machine: – You collect more data about the machine → knowledge
Bandits – applications
– You collect reward • Online advertisement
• For example – Exploration: Choose the next action at to minhH(bt )i – Exploitation: Choose the next action at to maxhyt i 9:6
• Clinical trials, robotic scientist
Digression: Active Learning • “Active Learning”
• Efficient optimization 9:2
“Experimental Design” “Exploration in Reinforcement Learning”
Bandits
All of these are strongly related to trying to minimize (also) H(bt )
• The bandit problem is an archetype for – Sequential decision making – Decisions that influence knowledge as well as rewards/states
Gaussian Processes:
– Exploration/exploitation • The same aspects are inherent also in global optimization, active learning & RL
• The Bandit problem formulation is the basis of UCB – which is (from Rasmussen & Williams)
the core of serveral planning and decision making methods
9:7
• Bandit problems are commercially very relevant 9:3
Upper Confidence Bounds (UCB)
Upper Confidence Bound (UCB) 1: 2:
9:4
3: 4:
Initialization: Play each machine once repeat Play the machine i that maximizes yˆi + β until
q
2 ln n ni
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
51
UCB for Gauss yˆi is the average reward of machine i so far
• If we have a single Gaussian bandits, we can compute P the mean estimator µ ˆ = n1 i yi P 2 1 the empirical variance σ ˆ 2 = n−1 i (yi − µ)
ni is how often machine i has been played so far P n = i ni is the number of rounds so far β is often chosen as β = 1
and the variance of the mean estimator Var{µ} = sˆ2 /n
The bound is derived from the Hoeffding inequality • µ ˆ and Var{µ} describe our posterior Gaussian belief over the See Finite-time analysis of the multiarmed bandit problem, Auer, Cesa-Bianchi & Fischer, Machine learning, 2002.
true underlying µ Using the err-function we can get exact quantiles
9:8
• Alternative strategies:
UCB algorithms 90%-quantile(µi ) • UCB algorithms determine a confidence interval such that µ ˆi + β
yˆi − σi < hyi i < yˆi + σi
p
√ Var{µi } = m ˆ i + βσ ˆ/ n 9:11
with high probability.
UCB - Discussion
UCB chooses the upper bound of this confidence interval
• UCB over-estimates the reward-to-go (under-estimates cost-to• Optimism in the face of uncertainty
go), just like A∗ – but does so in the probabilistic setting of bandits
• Strong bounds on the regret (sub-optimality) of UCB (e.g. Auer et al.)
• The fact that regret bounds exist is great! 9:9
• UCB became a core method for algorithms (including planners)
UCB for Bernoulli
to decide what to explore:
• If we have a single Bernoulli bandits, we can count
In tree search, the decision of which branches/actions to explore is itself a decision problem. An “intelligent agent” (like UBC) can
a = 1 + #wins ,
b = 1 + #losses
be used within the planner to make decisions about how to grow the tree.
• Our posterior over the Bernoulli parameter µ is Beta(µ | a, b) • The mean is hµi =
a a+b
The mode (most likely) is µ∗ = The variance is Var{µ} =
a−1 a+b−2
9:12
Monte Carlo Tree Search
for a, b > 1
ab (a+b+1)(a+b)2
9:13
One can numerically compute the inverse cumulative Beta distribution → get exact quantiles
Monte Carlo Tree Search (MCTS) • MCTS is very successful on Computer Go and other games
• Alternative strategies:
• MCTS is rather simple to implement • MCTS is very general: applicable on any discrete domain
argmax 90%-quantile(µi ) i
• Key paper: argmaxhµi i + β
´ Bandit based Monte-Carlo Planning, ECML Kocsis & Szepesvari:
p Var{µi }
2006.
i
9:10
• Survey paper: Browne et al.: A Survey of Monte Carlo Tree Search Methods, 2012.
52
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
• Tutorial presentation:
9:17
http://web.engr.oregonstate.edu/˜afern/ icaps10-MCP-tutorial.ppt 9:14
Monte Carlo methods
Flat Monte Carlo • The goal of MCTS is to estimate the utility (e.g., expected payoff
• General, the term Monte Carlo simulation refers to methods that
D) depending on the first action a chosen:
generate many i.i.d. random samples xi ∼ P (x) from a distribu-
Q(s0 , a) = E{D|s0 , a}
tion P (x). Using the samples one can estimate expectations of anything that depends on x, e.g. f (x): Z hf i =
P (x) f (x) dx ≈ x
where expectation is taken with w.r.t. the whole future randomized actions (including a potential opponent)
N 1 X f (xi ) N i=1
• Flat Monte Carlo does so by rolling out many random simula(In this view, Monte Carlo approximates an integral.)
tions (using a R OLLOUT P OLICY) without growing a tree The key difference/advantage of MCTS over flat MC is that the
• Example: What is the probability that a solitair would come out
tree growth focusses computational effort on promising actions
successful? (Original story by Stan Ulam.) Instead of trying to
9:18
analytically compute this, generate many random solitairs and
Upper Confidence Tree (UCT)
count.
• UCT uses UCB to realize the T REE P OLICY, i.e. to decide where • The method developed in the 40ies, where computers became
to expand the tree
faster. Fermi, Ulam and von Neumann initiated the idea. von Neumann called it “Monte Carlo” as a code name. 9:15
• B ACKUP updates all parents of vl as n(v) ← n(v) + 1 (count how often has it been played)
Generic MCTS scheme
Q(v) ← Q(v) + D (sum of rewards received)
• T REE P OLICY chooses child nodes based on UCB: s 2 ln n(v) Q(v 0 ) argmax +β 0) 0 n(v n(v 0 ) v ∈∂(v) or choose v 0 if n(v 0 ) = 0 9:19 from Browne et al. 1: 2: 3: 4: 5:
6: 7: 8:
start tree V = {v0 } while within computational budget do vl ← T REE P OLICY(V ) chooses a leaf of V append vl to V D ← R OLLOUT P OLICY(V ) rolls out a full simulation, with return D B ACKUP(vl , D) updates the values of all parents of vl end while return best child of v0 9:16
Generic MCTS scheme • In comparision to other planners it always computes full roll outs to a terminal state. No heuristics to estimate the utility of a state are needed. • The tree grows unbalanced • The T REE P OLICY decides where the tree is expanded – and needs to trade off exploration vs. exploitation • The R OLLOUT P OLICY is necessary to simulate a roll out. It could be a random policy; at least a randomized policy.
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
12
Game Playing
53
Properties of minimax Complete?? Yes, if tree is finite (chess has specific rules for this)
Outline
Optimal?? Yes, against an optimal opponent. Otherwise?? • Minimax
Time complexity?? O(bm )
• α–β pruning
Space complexity?? O(bm) (depth-first exploration)
• UCT for games
For chess, b ≈ 35, m ≈ 100 for “reasonable” games 10:1
⇒ exact solution completely infeasible But do we need to explore every path?
Game tree (2-player, deterministic, turns)
10:5
α–β pruning example
10:2
Minimax Perfect play for deterministic, perfect-information games Idea: choose move to position with highest minimax value = best achievable payoff against best play E.g., 2-ply game:
10:3
Minimax algorithm function M INIMAX -D ECISION(state) returns an action inputs: state, current state in game return the a in ACTIONS(state) maximizing M IN -VALUE(R ESULT(a, state)) function M AX -VALUE(state) returns a utility value if T ERMINAL -T EST(state) then return U TILITY(state) v ← −∞ for a, s in S UCCESSORS(state) do v ← M AX(v, M IN -VALUE(s)) return v function M IN -VALUE(state) returns a utility value if T ERMINAL -T EST(state) then return U TILITY(state) v←∞ for a, s in S UCCESSORS(state) do v ← M IN(v, M AX -VALUE(s)) return v 10:4
54
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 10:6
Suppose we have 100 seconds, explore 104 nodes/second ⇒ 106 nodes per move ≈ 358/2
Why is it called α–β?
⇒ α–β reaches depth 8 ⇒ pretty good chess program 10:10
Evaluation functions
α is the best value (to
MAX )
If V is worse than α,
MAX
Define β similarly for
MIN
found so far off the current path
will avoid it ⇒ prune that branch For chess, typically linear weighted sum of features 10:7
E VAL(s) = w1 f1 (s) + w2 f2 (s) + . . . + wn fn (s)
The α–β algorithm
e.g., w1 = 9 with
function A LPHA -B ETA -D ECISION(state) returns an action return the a in ACTIONS(state) maximizing M IN -VALUE(R ESULT(a, state)) function M AX -VALUE(state, α, β) returns a utility value inputs: state, current state in game α, the value of the best alternative for MAX along the path to state β, the value of the best alternative for MIN along the path to state if T ERMINAL -T EST(state) then return U TILITY(state) v ← −∞ for a, s in S UCCESSORS(state) do v ← M AX(v, M IN -VALUE(s, α, β)) if v ≥ β then return v α ← M AX(α, v) return v
f1 (s) = (number of white queens) – (number of black queens), etc. 10:11
Upper Confidence Tree (UCT) for games Standard backup updates all parents of vl as n(v) ← n(v) + 1 (count how often has it been played) Q(v) ← Q(v) + ∆ (sum of rewards received)
In games use a “negamax” backup: While iterating upward, flip sign ∆ ← −∆ in each iteration
function M IN -VALUE(state, α, β) returns a utility value same as M AX -VALUE but with roles of α, β reversed
Survey of MCTS applications: 10:8
Browne et al.: A Survey of Monte Carlo Tree Search Methods, 2012. 10:12
Properties of α–β
Brief notes on game theory
Pruning does not affect final result
• (Small) zero-sum games can be represented by a payoff matrix
Good move ordering improves effectiveness of pruning
• Uji denotes the utility of player 1 if she chooses the pure (=de-
A simple example of the value of reasoning about which computations are relevant (a form of metareasoning)
terministic) strategy i and player 2 chooses the pure strategy j. U T = −U
Zero-sum games: Uji = −Uij , 10:9
Resource limits Standard approach: • Use C UTOFF -T EST instead of T ERMINAL -T EST e.g., depth limit (perhaps add quiescence search)
• Fining a minimax optimal mixed strategy p is a Linear Program max w w
s.t. U p ≥ w ,
X
pi = 1 ,
p≥0
i
Note that U p ≥ w implies minj (U p)j ≥ w. • Gainable payoff of player 1: maxp minq q T U p Minimax-Theorem: maxp minq q T U p = minq maxp q T U p Minimax-Theorem ↔ optimal p with w ≥ 0 exists
• Use E VAL instead of U TILITY i.e., evaluation function that estimates desirability of position
10:13
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
13
Graphical Models
55
• The joint distribution can be factored as P (X1:n ) =
n Y
P (Xi | Parents(Xi ))
i=1
Outline
• Missing links imply conditional independence • Ancestral simulation to sample from joint distribution
• A. Introduction
11:5
– Motivation and definition of Bayes Nets – Conditional independence in Bayes Nets – Examples
Example
• B. Inference in Graphical Models – Sampling methods (Rejection, Importance, Gibbs) – Variable Elimination & Factor Graphs – Message passing, Loopy Belief Propagation
(Heckermann 1995) P(B=bad) =0.02
P(F=empty)=0.05
Battery
Fuel
11:1 Gauge
Graphical Models
P(G=empty|B=good,F=not empty)=0.04 P(G=empty|B=good,F=empty)=0.97 P(G=empty|B=bad,F=not empty)=0.10 P(G=empty|B=bad,F=empty)=0.99
• The core difficulty in modelling is specifying TurnOver
What are the relevant variables?
Start P(S=no|T=yes,F=not empty)=0.01 P(S=no|T=yes,F=empty)=0.92 P(S=no|T=no,Fnot empty)=1.00 P(S=no|T=no,F=empty)=1.00
P(T=no|B=good)=0.03 P(T=no|B=bad)=0.98
How do they depend on each other? (Or how could they depend on each other → learning)
⇐⇒ P (S, T, G, F, B) = P (B) P (F ) P (G|F, B) P (T |B) P (S|T, F ) • Graphical models are a simple, graphical notation for • Table sizes: LHS = 25 − 1 = 31 RHS = 1 + 1 + 4 + 2 + 4 = 12
1) which random variables exist
11:6
2) which random variables are “directly coupled” Thereby they describe a joint probability distribution P (X1 , .., Xn )
Bayes Nets & conditional independence
over n random variables. • Independence: Indep(X, Y ) ⇐⇒ P (X, Y ) = P (X) P (Y ) • 2 basic variants: – Bayesian Networks – Factor Graphs Field)
• Conditional independence: (aka. directed model, belief network)
Indep(X, Y |Z) ⇐⇒ P (X, Y |Z) = P (X|Z) P (Y |Z)
(aka. undirected model, Markov Random 11:2
X
Bayesian Networks
Z
• A Bayesian Network is a – directed acyclic graph (DAG) – where each node represents a random variable Xi
Z
Y X
Z Y
X
Y
(head-to-head)
(tail-to-tail)
(head-to-tail)
Indep(X, Y ) ¬Indep(X, Y |Z)
¬Indep(X, Y ) Indep(X, Y |Z)
¬Indep(X, Y ) Indep(X, Y |Z) 11:7
– for each node we have a conditional probability distribution P (Xi | Parents(Xi ))
• In the simplest case (discrete RVs), the conditional distribution is represented as a conditional probability table (CPT) 11:3
Example
P (X, Y, Z) = P (X) P (Y ) P (Z|X, Y ) P P (X, Y ) = P (X) P (Y ) Z P (Z|X, Y ) = P (X) P (Y )
• Tail-to-tail: Indep(X, Y |Z) P (X, Y, Z) = P (Z) P (X|Z) P (Y |Z)
drinking red wine → longevity? 11:4
Bayesian Networks • DAG → we can sort the RVs; edges only go from lower to higher index
• Head-to-head: Indep(X, Y )
P (X, Y |Z) = P (X, Y, Z) = P (Z) = P (X|Z) P (Y |Z)
• Head-to-tail: Indep(X, Y |Z) P (X, Y, Z) = P (X) P (Z|X) P (Y |Z) P (X, Y |Z) =
P (X,Y,Z) P (Z)
=
P (X,Z) P (Y |Z) P (Z)
= P (X|Z) P (Y |Z)
56
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 11:8
Inference • Inference: Given some pieces of information (prior, observed
General rules for determining conditional independence in a Bayes net:
variabes) what is the implication (the implied information, the posterior) on a non-observed variable
• In a Bayes Nets: Assume there is three groups of RVs: – Z are observed random variables
• Given three groups of random variables X, Y, Z
– X and Y are hidden random variables Indep(X, Y |Z) ⇐⇒ every path from X to Y is “blocked by Z”
• A path is “blocked by Z” ⇐⇒ on this path... – ∃ a node in Z that is head-to-tail w.r.t. the path, or
– We want to do inference about X, not Y
Given some observed variables Z, compute the posterior marginal P (X | Z) for some hidden variable X.
– ∃ a node in Z that is tail-to-tail w.r.t. the path, or P (X | Z) =
– ∃ another node A which is head-to-head w.r.t. the path and neither A nor any of its descendants are in Z 11:9
Example
X P (X, Z) 1 = P (X, Y, Z) P (Z) P (Z) Y
where Y are all hidden random variables except for X
• Inference requires summing over (eliminating) hidden variables. 11:12 (Heckermann 1995) P(B=bad) =0.02
P(F=empty)=0.05
Battery
Fuel
Example: Holmes & Watson • Mr. Holmes lives in Los Angeles. One morning when Holmes
Gauge
leaves his house, he realizes that his grass is wet. Is it due to
P(G=empty|B=good,F=not empty)=0.04 P(G=empty|B=good,F=empty)=0.97
rain, or has he forgotten to turn off his sprinkler? – Calculate P (R|H), P (S|H) and compare these values to the prior probabilities.
P(G=empty|B=bad,F=not empty)=0.10 P(G=empty|B=bad,F=empty)=0.99
TurnOver P(T=no|B=good)=0.03 P(T=no|B=bad)=0.98
Indep(T, F )?
Start
– Calculate P (R, S|H). Note: R and S are marginally independent, but conditionally dependent
P(S=no|T=yes,F=not empty)=0.01 P(S=no|T=yes,F=empty)=0.92 P(S=no|T=no,Fnot empty)=1.00 P(S=no|T=no,F=empty)=1.00
Indep(B, F |S)?
Indep(B, S|T )? 11:10
• Holmes checks Watson’s grass, and finds it is also wet. – Calculate P (R|H, W ), P (S|H, W ) – This effect is called explaining away
What can we do with Bayes nets? JavaBayes: run it from the html page
• Inference: Given some pieces of information (prior, observed
http://www.cs.cmu.edu/˜javabayes/Home/applet.html
variabes) what is the implication (the implied information, the
11:13
posterior) on a non-observed variable
Example: Holmes & Watson • Decision Making: If utilities and decision variables are defined
P(R=yes)=0.2
P(S=yes)=0.1
Rain
Sprinkler
→ compute optimal decisions in probabilistic domains Watson
• Learning: – Fully Bayesian Learning: Inference over parameters (e.g., β)
P(W=yes|R=yes)=1.0 P(W=yes|R=no)=0.2
Holmes P(H=yes|R=no,S=no)=0.0 P(H=yes|R=yes,S=yes)=1.0 P(H=yes|R=yes,S=no)=1.0 P(H=yes|R=no,S=yes)=0.9
P (H, W, S, R) = P (H|S, R) P (W |R) P (S) P (R)
– Maximum likelihood training: Optimizing parameters
P (R|H) =
• Structure Learning (Learning/Inferring the graph structure itself): Decide which model (which graph structure) fits the data best; thereby uncovering conditional independencies in the data.
=
1 X P (H|S, R) P (S) P (R) P (H) S
1 1 (1.0 · 0.2 · 0.1 + 1.0 · 0.2 · 0.9) = 0.2 P (H = 1) P (H = 1) 1 1 P (R = 0 | H = 1) = (0.9 · 0.8 · 0.1 + 0.0 · 0.8 · 0.9) = 0.072 P (H = 1) P (H = 1) P (R = 1 | H = 1) =
11:11
X P (R, W, S, H) X 1 = P (H|S, R) P (W |R) P (S) P (R) P (H) P (H) W,S W,S
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 11:14
57
In this view, Monte Carlo methods approximate an integral. • Motivation: p(x) itself is too complicated to express analytically or compute hf (x)ip directly
• These types of calculations can be automated → Variable Elimination Algorithm (discussed later) 11:15
13.1
Inference Methods in Graphical Models
• Example: What is the probability that a solitair would come out successful? (Original story by Stan Ulam.) Instead of trying to analytically compute this, generate many random solitairs and count. • Naming: The method developed in the 40ies, where computers became faster. Fermi, Ulam and von Neumann initiated the idea. von Neumann called it “Monte Carlo” as a code name. 11:19
11:16
Rejection Sampling Inference methods in graphical models • We have a Bayesian Network with RVs X1:n , some of which are • Sampling: – Rejection samping, importance sampling, Gibbs sampling – More generally, Markov-Chain Monte Carlo (MCMC) methods
observed: Xobs = yobs , obs ⊂ {1 : n}
• The goal is to compute marginal posteriors P (Xi | Xobs = yobs ) conditioned on the observations.
• Message passing: – Exact inference on trees (includes the Junction Tree Algorithm)
• We generate a set of K (joint) samples of all variables S = {xk1:n }K k=1
– Belief propagation
Each sample xk1:n = (xk1 , xk2 , .., xkn ) is a list of instantiation of all
• Other approximations/variational methods – Expectation propagation
RVs. 11:20
– Specialized variational methods depending on the model
Rejection Sampling • Reductions: – Mathematical Programming (e.g. LP relaxations of MAP)
• To generate a single sample xk1:n :
– Compilation into Arithmetic Circuits (Darwiche at al.) 11:17
1. Sort all RVs in topological order; start with i = 1 2. Sample a value xki ∼ P (Xi | xkParents(i) ) for the ith RV con-
Sampling
ditional to the previous samples xk1:i-1
• Read Andrieu et al: An Introduction to MCMC for Machine Learn-
3. If i ∈ obs compare the sampled value xki with the observation yi . Reject and repeat from a) if the sample is not
ing (Machine Learning, 2003)
equal to the observation. 4. Repeat with i ← i + 1 from 2.
• Here I’ll discuss only thee basic methods: – Rejection sampling
• We compute the marginal probabilities from the sample set S:
– Importance sampling – Gibbs sampling 11:18
P (Xi = x | Xobs = yobs ) ≈
countS (xki = x) K
or pair-wise marginals:
Monte Carlo methods • Generally, a Monte Carlo method is a method to generate a set
P (Xi = x, Xj = x0 | Xobs = yobs ) ≈
countS (xki = x ∧ xkj = x0 ) K
of (potentially weighted) samples that approximate a distribution 11:21
p(x). In the unweighted case, the samples should be i.i.d. xi ∼ p(x) In the general (also weighted) case, we want particles that allow
Importance sampling (with likelihood weighting)
to estimate expectations of anything that depends on x, e.g. f (x):
• Rejecting whole samples may become very inefficient in large Z hf (x)ip =
f (x) p(x) dx = lim x
N →∞
N X i=1
Bayes Nets! wi f (xi )
58
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Gibbs sampling*
• New strategy: We generate a weighted sample set S = {(xk1:n , wk )}K k=1
• As for rejection sampling, Gibbs sampling generates an unweighted sample set S which can directly be used to compute marginals.
where each sample
xk1:n
is associated with a weight w
k
In practice, one often discards an initial set of samples (burn-in) to avoid starting biases.
• In our case, we will choose the weights proportional to the likelihood P (Xobs = yobs | X1:n = xk1:n ) of the observations conditional to the sample xk1:n
• Gibbs sampling is a special case of MCMC sampling. Roughly, MCMC means to invent a sampling process, where the
11:22
next sample may stochastically depend on the previous (Markov property), such that the final sample set is guaranteed to corre-
Importance sampling
spond to P (X1:n ). → An Introduction to MCMC for Machine Learning
• To generate a single sample (wk , xk1:n ):
11:25 k
1. Sort all RVs in topological order; start with i = 1 and w =
Sampling – conclusions
1 2. a) If i 6∈ obs, sample a value xki ∼ P (Xi | xkParents(i) ) for the ith RV conditional to the previous samples xk1:i-1 b) If i ∈ obs, set the value
xki
= yi and update the weight
• Sampling algorithms are very simple, very general and very popular – they equally work for continuous & discrete RVs – one only needs to ensure/implement the ability to sample from conditional distributions, no further algebraic manipulations
according to likelihood wk ← wk P (Xi = yi | xk1:i-1 )
– MCMC theory can reduce required number of samples
3. Repeat with i ← i + 1 from 2. • In many cases exact and more efficient approximate inference is • We compute the marginal probabilities as: PK P (Xi = x | Xobs = yobs ) ≈
possible by actually computing/manipulating whole distributions in the algorithms instead of only samples.
wk [xki = x] PK k k=1 w
k=1
11:26
Variable Elimination
and likewise pair-wise marginals, etc. Notation: [expr] = 1 if expr is true and zero otherwise
11:27 11:23
Variable Elimination example X2
Gibbs sampling*
F4 µ µ 3 3 X1
• In Gibbs sampling we also generate a sample set S – but in this case the samples are not independent from each other. The next sample “modifies” the previous one: • First, all observed RVs are clamped to their fixed value xki = yi for any k. • To generate the (k + 1)th sample, iterate through the latent vari-
F3
µ1 (X
F5 µ2 µ4
µ5
X3
X5
P (x5 ) P = x1 ,x2 ,x3 ,x4 ,x6 P (x1 ) P (x2 |x1 ) P (x3 |x1 ) P (x4 |x2 ) P (x5 |x3 ) P (x6 |x2 , P P = x1 ,x2 ,x3 ,x6 P (x1 ) P (x2 |x1 ) P (x3 |x1 ) P (x5 |x3 ) P (x6 |x2 , x5 ) P( x4 |
F1 (
=
P
=
P
P (x1 ) P (x2 |x1 ) P (x3 |x1 ) P (x5 |x3 ) P (x6 |x2 , x5 ) µ1 (x2 ) P P (x1 ) P (x2 |x1 ) P (x3 |x1 ) P (x5 |x3 ) µ1 (x2 ) (x6 |x2 , x x6 P | {z
x1 ,x2 ,x3 ,x6 x1 ,x2 ,x3
F2 (x2 ,x5 ,x6
ables i 6∈ obs, updating:
=
P
=
P
=
P
=
P
=
P
x ,x ,x P (x1 ) P (x2 |x1 ) P (x3 |x1 ) P (x5 |x3 ) µ1 (x2 ) µ2 (x2 , x5 ) P P 1 2 3 = x2 ,x3 P (x5 |x3 ) µ1 (x2 ) µ2 (x2 , x5 ) P (x1 ) P (x2 |x1 ) P (x3 |x1 ) x1 | {z }
xk+1 ∼ P (Xi | xk1:n\i ) i
F3 (x1 ,x2 ,x3 )
∼ P (Xi | xk1 , xk2 , .., xki-1 , xki+1 , .., xkn ) Y P (Xj = xkj | Xi , xkParents(j)\i ) ∼ P (Xi | xkParents(i) )
P (x5 |x3 ) µ1 (x2 ) µ2 (x2 , x5 ) µ3 (x2 , x3 ) P P (x5 |x3 ) 1 (x2 ) µ2 (x2 , x5 ) µ3 (x2 , x3 ) x2 µ | {z }
x2 ,x3 x3
F4 (x3 ,x5 )
j:i∈Parents(j)
That is, each xk+1 is resampled conditional to the other (neighi
x3 x3
P (x5 |x3 ) µ4 (x3 , x5 ) P (x5 |x3 ) µ4 (x3 , x5 ) | {z } F5 (x3 ,x5 )
boring) current sample values.
= µ5 (x5 ) 11:24
11:28
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Variable Elimination example – lessons learnt • There is a dynamic programming principle behind Variable Elimination: – For eliminating X5,4,6 we use the solution of eliminating X4,6 – The “sub-problems” are represented by the F terms, their solutions by the remaining µ terms
59
Variable Elimination Algorithm • eliminate single variable(F, i)
6:
Input: list F of factors, variable id i Output: list F of factors find relevant subset Fˆ ⊆ F of factors coupled to i: Fˆ = {k : i ∈ ∂k} ˆ with neighborhood ∂ k ˆ = all variables create new factor k in Fˆ except i P Q compute µkˆ (X∂ kˆ ) = Xi k∈Fˆ fk (X∂k ) remove old factors Fˆ and append new factor µˆ to F
7:
return F
1: 2: 3:
4:
– We’ll continue to discuss this 4 slides later! 5:
• The factorization of the joint – determines in which order Variable Elimination is efficient – determines what the terms F (...) and µ(...) depend on
• elimination algorithm(µ, F, M )
• We can automate Variable Elimination. For the automation, all that matters is the factorization of the joint.
1:
2:
11:29
3: 4: 5:
Factor graphs
6:
• In the previous slides we introduces the box
k
notation to indi-
7:
Input: list F of factors, tuple M of desired output variables ids Output: single factor µ over variables XM define all variables present in F : V = vars(F ) define variables to be eliminated: E = V \ M for all i ∈ E: eliminate single variable(F, i) for all remaining factors, compute the product µ = Q f ∈F f return µ
cate terms that depend on some variables. That’s exactly what
11:32
factor graphs represent.
Variable Elimination on trees • A Factor graph is a – bipartite graph
Y3 1
– where each circle node represents a random variable Xi
Y8
– each box node represents a factor fk , which is a function fk (X∂k )
Y1
2
6
X Y4
F1 (Y1,8 , X)
F3 (Y3,4,5 , X)
Y2
– the joint probability distribution is given as 4
P (X1:n ) =
K Y
7
3
5
Y5
F2 (Y2,6,7 , X)
Y6 Y7
fk (X∂k )
k=1
The subtrees w.r.t. X can be described as
Notation: ∂k is shorthand for Neighbors(k) 11:30
F1 (Y1,8 , X) = f1 (Y8 , Y1 ) f2 (Y1 , X) F2 (Y2,6,7 , X) = f3 (X, Y2 ) f4 (Y2 , Y6 ) f5 (Y2 , Y7 )
Bayes Net → factor graph
F3 (Y3,4,5 , X) = f6 (X, Y3 , Y4 ) f7 (Y4 , Y5 ) X4
X2 X1
The joint distribution is: X6
P (Y1:8 , X) = F1 (Y1,8 , X) F2 (Y2,6,7 , X) F3 (Y3,4,5 , X) 11:33
X3
• Bayesian Network:
X5
P (x1:6 ) = P (x1 ) P (x2 |x1 ) P (x3 |x1 ) P (x4 |x2 ) P (x5 |x3 ) P (x6 |x2 , x5 )
X2
Variable Elimination on trees
X4
Y3 µ2→X
Y1 2
X1
• Factor Graph:
X6 X3
Y9
1
Y8 F1 (Y1,8 , X)
6
X µ6→X µ3→X
3
X5
8
Y4
7
F3 (Y3,4,5 , X)
Y2
Y5
4
P (x1:6 ) = f1 (x1 , x2 ) f2 (x3 , x1 ) f3 (x2 , x4 ) f4 (x3 , x5 ) f5 (x2 , x5 , x6 )
5
F2 (Y2,6,7 , X)
Y6 Y7
→ each CPT in the Bayes Net is just a factor (we neglect the We can eliminate each tree independently. The remaining terms
special semantics of a CPT) 11:31
(messages) are: P µF1 →X (X) = Y1,8 F1 (Y1,8 , X)
60
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 µF2 →X (X) =
P
F2 (Y2,6,7 , X)
• Message passing exemplifies how to exploit the factorization
µF3 →X (X) =
P
F3 (Y3,4,5 , X)
structure of the joint distribution for the algorithmic implemen-
Y2,6,7 Y3,4,5
tation
The marginal P (X) is the product of subtree messages P (X) = µF1 →X (X) µF2 →X (X) µF3 →X (X) 11:34
• Note: These are recursive equations. They can be resolved exactly if and only if the dependency structure (factor graph) is
Variable Elimination on trees – lessons learnt
a tree. If the factor graph had loops, this would be a “loopy recursive equation system”...
• The “remaining terms” µ’s are called messages
11:37
Intuitively, messages subsume information from a subtree
• Marginal = product of messages, P (X) =
Q
Message passing variants
µFk →X , is very
k
intuitive: – Fusion of independent information from the different subtrees – Fusing independent information ↔ multiplying probability tables
• Message passing has many important applications: – Many models are actually trees: In particular chains esp. Hidden Markov Models – Message passing can also be applied on non-trees (↔ loopy graphs) → approximate inference (Loopy Belief Propagation)
• Along a (sub-) tree, messages can be computed recursively 11:35
– Bayesian Networks can be “squeezed” to become trees → exact inference in Bayes Nets! (Junction Tree Algorithm) 11:38
Message passing • General equations (belief propagation (BP)) for recursive message computation (writing µk→i (Xi ) instead of µFk →X (X)):
• If the graphical model is not a tree (=has loops): – The recursive message equations cannot be resolved.
µ ¯ j→k (Xj )
µk→i (Xi ) =
X
fk (X∂k )
X∂k\i
z Y
Y j∈∂k\i
|
Loopy Belief Propagation
– However, we could try to just iterate them as update equations...
}| { µk0 →j (Xj )
k0 ∈∂j\k
{z
F (subtree)
} • Loopy BP update equations:
Q
j∈∂k\i :
excl. i Q
k0 ∈∂j\k
(initialize with µk→i = 1)
branching at factor k, prod. over adjacent variables j
: branching at variable j, prod. over adjacent factors k
0
µnew k→i (Xi ) =
X
fk (X∂k )
X∂k\i
Y
Y
j∈∂k\i
k0 ∈∂j\k
µold k0 →j (Xj )
excl. k 11:39
µ ¯ j→k (Xj ) are called “variable-to-factor messages”: store them for efficiency
Loopy BP remarks Y3
µ1→Y1
Y9
1
Y8
µ2→X
Y1 2
F1 (Y1,8 , X)
6
X µ6→X µ3→X
3
4
8
Y4 µ7→Y47 F3 (Y3,4,5 , X)
Y2
µ4→Y2
µ8→Y4
µ5→Y2 5
Y5
Example messages: P µ2→X = f2 (Y1 , X) µ1→Y1 (Y1 ) PY1 µ6→X = f6 (Y3 , Y4 , X) µ7→Y4 (Y4 ) PY3 ,Y4 µ3→X = Y f3 (Y2 , X) µ4→Y2 (Y2 ) µ5→Y2 (Y2 ) 2
F2 (Y2,6,7 , X)
Y6 Y7
11:36
Message passing remarks • Computing these messages recursively on a tree does nothing else than Variable Elimination Q ⇒ P (Xi ) = k∈∂i µk→i (Xi ) is the correct posterior marginal
• However, since it stores all “intermediate terms”, we can compute ANY marginal P (Xi ) for any i
• Problem of loops intuitively: loops ⇒ branches of a node to not represent independent information! – BP is multiplying (=fusing) messages from dependent sources of information • No convergence guarantee, but if it converges, then to a state of marginal consistency X X b(X∂k ) = b(X∂k0 ) = b(Xi ) X∂k\i
X ∂k0 \i
and to the minimum of the Bethe approximation of the free energy (Yedidia, Freeman, & Weiss, 2001) • We shouldn’t be overly disappointed: – if BP was exact on loopy graphs we could efficiently solve NP hard problems... – loopy BP is a very interesting approximation to solving an NP hard problem – is hence also applied in context of combinatorial optimization (e.g., SAT problems) • Ways to tackle the problems with BP convergence: – Damping (Heskes, 2004: On the uniqueness of loopy belief propagation fixed points) – CCCP (Yuille, 2002: CCCP algorithms to minimize the Bethe and Kikuchi free energies: Convergent alternatives to belief propagation) – Tree-reweighted MP (Kolmogorov, 2006: Convergent tree-reweighted message passing for energy minimization)
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
61 11:43
11:40
Junction Tree Algorithm
Junction Tree Algorithm Example
• Many models have loops X2
Instead of applying loopy BP in the hope of getting a good approximation, it is possible to convert every model into a tree by redefinition of
X4 X4 X6
X1
X1
RVs. The Junction Tree Algorithms converts a loopy model into a tree.
X3
X2,5
X2,3
X6
X5
• Loops are resolved by defining larger variable groups (separators) on which messages are defined
• If we eliminate in order 4, 6, 5, 1, 2, 3, we get remaining terms 11:41
(X2 ), (X2 , X5 ), (X2 , X3 ), (X2 , X3 ), (X3 )
Junction Tree Example • Example:
which translates to the Junction Tree on the right A
B
A
B
C
D
C
D
11:44
Maximum a-posteriori (MAP) inference • Often we want to compute the most likely global assignment
• Join variable B and C to a single separator MAP X1:n = argmax P (X1:n )
A
X1:n
B, C D
This can be viewed as a variable substitution: rename the tuple
of all random variables. This is called MAP inference and can be solved P by replacing all by max in the message passing equations – the algorithm is called Max-Product Algorithm and is a generalization of Dynamic Programming methods like Viterbi or Dijkstra.
(B, C) as a single random variable • Application: Conditional Random Fields
• A single random variable may be part of multiple separators – but only along a running intersection
f (y, x) = φ(y, x)>β =
k X
φj (y∂j , x)βj = log
j=1
11:42
Junction Tree Algorithm
k hY
eφj (y∂j ,x)βj
i
j=1
with prediction x 7→ y ∗ (x) = argmax f (x, y) y
Finding the argmax is a MAP inference problem! This is frequently needed in the innerloop of CRF learning algorithms.
• Standard formulation: Moralization & Triangulation
11:45
A clique is a fully connected subset of nodes in a graph. 1) Generate the factor graph (classically called “moralization”)
Conditional Random Fields
2) Translate each factor to a clique: Generate the undirected graph
• The following are interchangable:
“Random Field” ↔ “Markov R
where undirected edges connect all RVs of a factor 3) Triangulate the undirected graph 4) Translate each clique back to a factor; identify the separators between factors
• Formulation in terms of variable elimination: 1) Start with a factor graph
• Therefore, a CRF is a conditional factor graph: – A CRF defines a mapping from input x to a factor graph over y – Each feature φj (y∂j , x) depends only on a subset ∂j of variables y∂j – If y∂j are discrete, a feature φj (y∂j , x) is usually an indicator feature (see lecture 03); the corresponding parameter βj is then one entry of a factor fk (y∂j ) that couples these variables
2) Choose an order of variable elimination 11:46
3) Keep track of the “remaining µ terms” (slide 14): which RVs would they depend on? → this identifies the separators
62
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
What we didn’t cover
• A very promising line of research is solving inference problems using mathematical programming. This unifies research in the areas of optimization, mathematical programming and probabilistic inference. Linear Programming relaxations of MAP inference and CCCP methods are great examples. 11:47
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
14
Dynamic Models
63
• A Hidden Markov Model (HMM) is defined as the joint distribution P (X0:T , Y0:T ) = P (X0 ) ·
Motivation
T Y
P (Xt |Xt-1 ) ·
t=1
– Robotics slides – Speech recognition
T Y
P (Yt |Xt ) .
t=0
X0
X1
X2
X3
XT
Y0
Y1
Y2
Y3
YT
– Music 12:4 12:1
Inference in an HMM – a tree!
Markov processes (Markov chains) Construct a Bayes net from these variables: parents?
Ffuture (X2:T , Y3:T )
Fpast (X0:2 , Y0:1 )
Markov assumption: Xt depends on bounded subset of X0:t−1 First-order Markov process: P (Xt | X0:t−1 ) = P (Xt | Xt−1 )
X0
X1
X2
X3
XT
Y0
Y1
Y2
Y3
YT
Second-order Markov process: P (Xt | X0:t−1 ) = P (Xt | Xt−2 , Xt−1 )
Fnow (X2 , Y2 )
• The marginal posterior P (Xt | Y1:T ) is the product of three messages
Sensor Markov assumption: P (Yt | X0:t , Y0:t−1 ) = P (Yt | Xt ) Stationary process: transition model P (Xt | Xt−1 ) and
P (Xt | Y1:T ) ∝ P (Xt , Y1:T ) = µpast (Xt ) µnow (Xt ) µfuture (Xt ) |{z} | {z } |{z} %
α
β
sensor model P (Yt | Xt ) fixed for all t 12:2
Different inference problems in Markov Mod-
• For all a < t and b > t – Xa conditionally independent from Xb given Xt – Ya conditionally independent from Yb given Xt
els
“The future is independent of the past given the present” Markov property • P (xt | y0:T ) marginal posterior • P (xt | y0:t ) filtering • P (xt | y0:a ), t > a prediction • P (xt | y0:b ), t < b smoothing • P (y0:T ) likelihood calculation
• Viterbi alignment: Find sequence
x∗0:T
(conditioning on Yt does not yield any conditional independences) 12:5
Inference in HMMs
that maximizes P (x0:T | y0:T )
(This is done using max-product, instead of sum-product message passing.)
Ffuture (X2:T , Y3:T )
Fpast (X0:2 , Y0:1 ) X0
X1
X2
X3
XT
Y0
Y1
Y2
Y3
YT
Fnow (X2 , Y2 )
Applying the general message passing equations:
12:3 forward msg.
backward msg.
• We assume we have
slice – a discrete latent variable Xt in each time slice
X
P (xt |xt-1 ) αt-1 (xt-1 ) %t-1 (xt-1 )
xt-1
Hidden Markov Models as Graphical Model
– observed (discrete or continuous) variables Yt in each time
µXt-1 →Xt (xt ) =: αt (xt ) =
α0 (x0 ) = P (x0 ) X P (xt+1 |xt ) βt+1 (xt+1 ) %t+1 (xt+1 )
µXt+1 →Xt (xt ) =: βt (xt ) =
xt+1
βT (x0 ) = 1 observation msg. posterior marginal posterior marginal
µYt →Xt (xt ) =: %t (xt ) = P (yt | xt ) q(xt ) ∝ αt (xt ) %t (xt ) βt (xt )
q(xt , xt+1 ) ∝ αt (xt ) %t (xt ) P (xt+1 |xt ) %t+1 (xt+1 ) βt+1
– some observation model P (Yt | Xt ; θ) – some transition model P (Xt | Xt-1 ; θ)
12:6
64
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Inference in HMMs – implementation notes
Kalman Filter example
• The message passing equations can be implemented by reinterpreting them as matrix equations: Let αt , β t , %t be the vectors corresponding to the probability tables αt (xt ), βt (xt ), %t (xt ); and let P be the matrix with enties P (xt | xt-1 ). Then
• filtering of a position (x, y) ∈ R2 :
1: 2: 3: 4: 5:
α0 = π, β T = 1 fort=1:T -1 : αt = P (αt-1 · %t-1 ) fort=T -1:0 : β t = P> (β t+1 · %t+1 ) fort=0:T : q t = αt · %t · β t fort=0:T -1 : Qt = P · [(β t+1 · %t+1 ) (αt · %t )>]
where · is the element-wise product! Here, q t is the vector with entries q(xt ), and Qt the matrix with entries q(xt+1 , xt ). Note that the equation for Qt describes Qt (x0 , x) = P (x0 |x)[(βt+1 (x0 )%t+1 (x0 ))(αt (x)%t (x))]. 12:7
Inference in HMMs: classical derivation
12:10
Given our knowledge of Belief propagation, inference in HMMs is simple. For reference, here is a more classical derivation:
Kalman Filter example • smoothing of a position (x, y) ∈ R2 :
P (y0:T | xt ) P (xt ) P (y0:T ) P (y0:t | xt ) P (yt+1:T | xt ) P (xt ) = P (y0:T ) P (y0:t , xt ) P (yt+1:T | xt ) = P (y0:T ) αt (xt ) βt (xt ) = P (y0:T )
P (xt | y0:T ) =
αt (xt ) := P (y0:t , xt ) = P (yt |xt ) P (y0:t-1 , xt ) X = P (yt |xt ) P (xt | xt-1 ) αt-1 (xt-1 ) xt-1
βt (xt ) := P (yt+1:T | xt ) =
X
P (yt+1:T | xt+1 ) P (xt+1 | xt )
x+1
=
i Xh βt+1 (xt+1 ) P (yt+1 |xt+1 ) P (xt+1 | xt ) x+1
12:11
Note: αt here is the same as αt · %t on all other slides! 12:8
HMM remarks
HMM example: Learning Bach • A machine “listens” (reads notes of) Bach pieces over and over
• The computation of forward and backward messages along the
again → It’s supposed to learn how to write Bach pieces itself (or at
Markov chain is also called forward-backward algorithm • The EM algorithm to learn the HMM parameters is also called
least harmonize them).
Baum-Welch algorithm • If the latent variable xt is continuous xt ∈ Rd instead of discrete, then such a Markov model is also called state space
• Harmonizing Chorales in the Style of J S Bach Moray Allan & Chris Williams (NIPS 2004)
model. • If the continuous transitions and observations are linear Gaus-
– observed sequence Y0:T Soprano melody
sian P (xt+1 |xt ) = N(xt+1 | Axt +a, Q) ,
• use an HMM
P (yt |xt ) = N(yt | Cxt +c, W )
– latent sequence X0:T chord & and harmony:
then the forward and backward messages αt and βt are also Gaussian. → forward filtering is also called Kalman filtering → smoothing is also called Kalman smoothing • Sometimes, computing forward and backward messages (in disrete or continuous context) is also called Bayesian filtering/smoothing 12:9
12:12
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
HMM example: Learning Bach • results: http://www.anc.inf.ed.ac.uk/demos/hmmbach/
• See also work by Gerhard Widmer http://www.cp.jku.at/ people/widmer/ 12:13
Dynamic Bayesian Networks – Arbitrary BNs in each time slide – Special case: MDPs, speech, etc 12:14
65
66
15
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Reinforcement Learning • Stationary MDP: – We assume P (s0 | s, a) and P (r|s, a) independent of time R – We also define R(s, a) := E{r|s, a} = r P (r|s, a) dr
Long history of RL in AI
13:3
Idea of programming a computer to learn by trial and error (Tur-
State value function
ing, 1954) SNARCs (Stochastic Neural-Analog Reinforcement Calculators) (Minsky, 54)
• The value (expected discounted return) of policy π when started in state s:
Checkers playing program (Samuel, 59) Lots of RL in the 60s (e.g., Waltz & Fu 65; Mendel 66; Fu 70) MENACE (Matchbox Educable Naughts and Crosses Engine
V π (s) = Eπ {r0 + γr1 + γ 2 r2 + · · · | s0 = s}
(Mitchie, 63) discounting factor γ ∈ [0, 1]
RL based Tic Tac Toe learner (GLEE) (Mitchie 68) Classifier Systems (Holland, 75) Adaptive Critics (Barto & Sutton, 81)
• Definition of optimality: behavior π ∗ is optimal iff
Temporal Differences (Sutton, 88)
∗
from Satinder Singh’s Introduction to RL, videolectures.com
∀s : V π (s) = V ∗ (s)
where V ∗ (s) = max V π (s) π
(simultaneously maximising the value in all states)
• Long history in Psychology 13:1
(In MDPs there always exists (at least one) optimal deterministic policy.)
Outline
13:4
• Markov Decision Processes as formal model – Definition – Value/Q-function – Planning as computing V /Q given a model
• Learning
An example for a value function...
– Temporal Difference & Q-learning – Limitations of the model-free view – Model-based RL
• Exploration • Briefly
demo:
test/mdp runVI
– Imitation Learning & Inverse RL – Continuous states and actions (LSPI, Policy Gradients) 13:2
Values provide a gradient towards desirable states 13:5
Markov Decision Process Value function a0
a1
a2
s0
s1
s2
• The value function V is a central concept in all of RL! r0
P (s0:T +1 , a0:T , r0:T ; π) = P (s0 )
r1
QT
t=0
r2
P (at |st ; π) P (rt |st , at ) P (st+1 |st , at )
• In other domains (stochastic optimal control) it is also called
– world’s initial state distribution P (s0 ) – world’s transition probabilities P (st+1 | st , at ) – world’s reward probabilities P (rt | st , at ) – agent’s policy π(at | st ) = P (a0 |s0 ; π) at = π(st ))
Many algorithms can directly be derived from properties of the value function.
cost-to-go function (cost = −reward) 13:6
(or deterministic
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Recursive property of the value function
67
• Value Iteration:
(initialize Vk=0 (s) = 0)
h i X ∀s : Vk+1 (s) = max R(s, a) + γ P (s0 |s, a) Vk (s0 ) a
π
s0
2
V (s) = E{r0 + γr1 + γ r2 + · · · | s0 = s; π}
stopping criterion:
= E{r0 | s0 = s; π} + γE{r1 + γr2 + · · · | s0 = s; π} P = R(s, π(s)) + γ s0 P (s0 | s, π(s)) E{r1 + γr2 + · · · | s1 = s0 ; π} P = R(s, π(s)) + γ s0 P (s0 | s, π(s)) V π (s0 )
maxs |Vk+1 (s) − Vk (s)| ≤
• Note that V ∗ is a fixed point of value iteration! • Value Iteration converges to the optimal value function V ∗ (proof below) demo:
V π = Rπ + γP π V π
• We can write this in vector notation
test/mdp runVI 13:10
with vectors V πs = V π (s), Rπs = R(s, π(s)) and matrix P πs0 s = P (s0 | s, π(s))
State-action value function (Q-function)
• For stochastic π(a|s): P P V π (s) = a π(a|s)R(s, a) + γ s0 ,a π(a|s)P (s0 | s, a) V π (s0 )
• We repeat the last couple of slides for the Q-function... 13:7
• The state-action value function (or Q-function) is the expected discounted return when starting in state s and taking first action
Bellman optimality equation
a: • Recall the recursive property of the value function π
V (s) = R(s, π(s)) + γ
P
s0
0
Qπ (s, a) = Eπ {r0 + γr1 + γ 2 r2 + · · · | s0 = s, a0 = a} X = R(s, a) + γ P (s0 | s, a) Qπ (s0 , π(s0 ))
0
π
P (s | s, π(s)) V (s )
s0
(Note: V π (s) = Qπ (s, π(s)).)
• Bellman optimality equation i h P V ∗ (s) = maxa R(s, a) + γ s0 P (s0 | s, a) V ∗ (s0 ) h i P with π ∗ (s) = argmaxa R(s, a) + γ s0 P (s0 | s, a) V ∗ (s0 )
• Bellman optimality equation for the Q-function P Q∗ (s, a) = R(s, a) + γ s0 P (s0 | s, a) maxa0 Q∗ (s0 , a0 )
(Sketch of proof: If π would select another action than argmaxa [·], then π 0 which = π everywhere except π 0 (s) = argmaxa [·] would be better.)
with π ∗ (s) = argmaxa Q∗ (s, a) 13:11
Q-Iteration
• This is the principle of optimality in the stochastic case 13:8
• Recall the Bellman equation: Richard E. Bellman (1920—1984)
Bellman’s principle of optimality
Q∗ (s, a) = R(s, a) + γ • Q-Iteration:
B
P
s0
P (s0 | s, a) maxa0 Q∗ (s0 , a0 )
(initialize Qk=0 (s, a) = 0)
∀s,a : Qk+1 (s, a) = R(s, a) + γ
A
X
P (s0 |s, a) max Qk (s0 , a0 ) 0 a
s0
A opt ⇒ B opt
stopping criterion:
maxs,a |Qk+1 (s, a) − Qk (s, a)| ≤
• Note that Q∗ is a fixed point of Q-Iteration! h
i
V ∗ (s) = max R(s, a) + γ s0 P (s0 | s, a) V ∗ (s0 ) a h i P ∗ π (s) = argmax R(s, a) + γ s0 P (s0 | s, a) V ∗ (s0 ) P
• Q-Iteration converges to the optimal state-action value function Q∗
a
13:12 13:9
Proof of convergence
Value Iteration
• Let Dk = ||Q∗ − Qk ||∞ = maxs,a |Q∗ (s, a) − Qk (s, a)|
• How can we use this to compute V ∗ ? • Recall the Bellman optimality equation:
Qk+1 (s, a) = R(s, a) + γ
X
≤ R(s, a) + γ
X
s0 ∗
h
V (s) = maxa R(s, a) + γ
P
s0
0
∗
0
P (s | s, a) V (s )
i
s0
P (s0 |s, a) max Qk (s0 , a0 ) 0 a
h i ∗ 0 0 P (s0 |s, a) max Q (s , a ) + D k 0 a
68
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 h i X ∗ 0 0 = R(s, a) + γ P (s0 |s, a) max Q (s , a ) + γDk 0 a
s0
• Learning – Temporal Difference & Q-learning – Limitations of the model-free view – Model-based RL
= Q∗ (s, a) + γDk similarly: Qk ≥ Q∗ − Dk ⇒ Qk+1 ≥ Q∗ − γDk
• Exploration • Briefly
• The proof translates directly also to value iteration 13:13
– Imitation Learning & Inverse RL – Continuous states and actions (LSPI, Policy Gradients) 13:16
For completeness* Learning in MDPs • While interacting with the world, the agent collects data of the
• Policy Evaluation computes V π instead of V ∗ : Iterate:
form π ∀s : Vk+1 (s) = R(s, π(s)) + γ
P
s0
D = {(st , at , rt , st+1 )}Tt=1
P (s0 |s, π(s)) Vkπ (s0 )
(state, action, immediate reward, next state) Or use matrix inversion V π = (I −γP π )−1 Rπ , which is O(|S|3 ). What could we learn from that? • Policy Iteration uses V π to incrementally improve the policy:
• Model-based RL: learn to predict next state: estimate P (s0 |s, a)
1. Initialise π0 somehow (e.g. randomly)
learn to predict immediate reward: estimate P (r|s, a)
2. Iterate: – Policy Evaluation: compute V
πk
πk
or Q
– Policy Update: πk+1 (s) ← argmaxa Qπk (s, a)
• Model-free RL: learn to predict value: estimate V (s) or Q(s, a)
demo:
test/mdp runPI 13:14
• Policy search: e.g., estimate the “policy gradient”, or directly use black box
Towards Learning
(e.g. evolutionary) search 13:17
• From Sutton & Barto’s Reinforcement Learning book: The term dynamic programming (DP) refers to a collection of algorithms that can be used to compute optimal policies given a perfect model of the environment as a Markov decision process (MDP). Classical DP algorithms are of limited utility in reinforcement learning both because of their assumption of a perfect model and because of their great computational expense, but they are still important theoretically. DP provides an essential foundation for the understanding of the methods presented in the rest of this book. In fact, all of these methods can be viewed as attempts to achieve much the same effect as DP, only with less computation and without assuming a perfect model of the environment.
Let’s introduce basic model-free methods first. D = {(s, a, r, s0 )t }Tt=0
learn
→
V (s)
→
π(s) 13:18
• So far, we introduced basic notions of an MDP and value func-
Temporal difference (TD) learning with V
tions and methods to compute optimal policies assuming that • Recall the recursive property of V (s):
we know the world (know P (s0 |s, a) and R(s, a)) Value Iteration and Q-Iteration are instances of Dynamic Programming
V π (s) = R(s, π(s)) + γ
P
s0
P (s0 | s, π(s)) V π (s0 )
• TD learning: Given a new experience (s, a, r, s0 ) Vnew (s) = (1 − α) Vold (s) + α [r + γVold (s0 )]
• Reinforcement Learning?
= Vold (s) + α [r + γVold (s0 ) − Vold (s)] . 13:15
Outline • Markov Decision Processes as formal model – Definition – Value/Q-function – Planning as computing V /Q given a model
• Reinforcement: – more reward than expected (r > Vold (s) − γVold (s0 )) → increase V (s) – less reward than expected (r < Vold (s) − γVold (s0 )) → decrease V (s)
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 13:19
69
• Reinforcement: – more reward than expected (r > Qold (s, a)−γ maxa0 Qold (s0 , a0 ))
Temporal difference (TD) learning with Q
→ increase Q(s, a) – less reward than expected (r < Qold (s, a)−γ maxa0 Qold (s0 , a0 ))
• Recall the recursive property of Q(s, a):
→ decrease Q(s, a) π
Q (s, a) = R(s, a) + γ
P
s0
0
∗
0
0
P (s |s, a) Q (s , π(s ))
13:22
Q-learning = off-policy TD learning with Q∗
• TD learning: Given a new experience (s, a, r, s0 , a0 = π(s0 )) Qnew (s, a) = (1 − α) Qold (s, a) + α [r + γQold (s0 , a0 )]
• Off-policy: We estimate Q∗ while executing π
= Qold (s, a) + α [r + γQold (s0 , a0 ) − Qold (s, a)] • Q-learning: 1:
• Reinforcement:
2:
– more reward than expected (r > Qold (s, a) − γQold (s0 , a0 ))
3: 4:
→ increase Q(s, a)
5:
– less reward than expected (r < Qold (s, a) − γQold (s0 , a0 ))
6: 7:
→ decrease Q(s, a)
8:
13:20
9: 10:
Sarsa = on-policy TD learning with Q
Initialize Q(s, a) = 0 repeat // for each episode Initialize start state s repeat // for each step of episode Choose action a ≈ argmaxa Q(s, a) Take action a, observe r, s0 Q(s, a) ← Q(s, a) + α [r + γ maxa0 Qold (s0 , a0 ) − Qold (s, a)] s ← s0 , a ← a0 until end of episode until happy 13:23
• On-policy: We estimate Qπ while executing π
Q-learning convergence with prob 1 1: 2: 3: 4: 5:
• Sarsa:
6: 7: 8: 9: 10: 11:
Initialize Q(s, a) = 0 repeat // for each episode Initialize start state s Choose action a ≈ argmaxa Q(s, a) repeat // for each step of episode Take action a, observe r, s0 Choose action a0 ≈ argmaxa0 Q(s0 , a0 ) Q(s, a) ← Q(s, a)+α [r+γQold (s0 , a0 )−Qold (s, a)] s ← s0 , a ← a0 until end of episode until happy
• Q-learning is a stochastic approximation of Q-Iteration: Q-learning: Qnew (s, a) = (1 − α)Qold (s, a) + α[r + γ maxa0 Qold (s0 , a0 )] Q-Iteration: P ∀s,a : Qk+1 (s, a) = R(s, a) + γ s0 P (s0 |s, a) maxa0 Qk (s0 , a0 )
We’ve shown convergence of Q-Iteration to Q∗
• Convergence of Q-learning: Q-Iteration is a deterministic update: Qk+1 = T (Qk ) • -greedy action selection:
Q-learning is a stochastic version: Qk+1 = (1−α)Qk +α[T (Qk )+ ηk ]
a ≈ argmax Q(s, a)
⇐⇒
a=
a
random with prob. argmaxa Q(s, a) else
ηk is zero mean! 13:24
13:21
Q-learning impact
Q-learning
• Q-Learning was the first provably convergent direct adaptive op-
• Recall the Bellman optimality equation for the Q-function: Q∗ (s, a) = R(s, a) + γ
P
s0
timal control algorithm
P (s0 |s, a) maxa0 Q∗ (s0 , a0 ) 0
• Q-learning (Watkins, 1988) Given a new experience (s, a, r, s ) Qnew (s, a) = (1 − α) Qold (s, a) + α [r + γmax Qold (s0 , a0 )] 0 a
= Qold (s, a) + α [r − Qold (s, a) + γ max Qold (s0 , a0 )] 0 a
• Great impact on the field of Reinforcement Learning in 80/90ies – “Smaller representation than models” – “Automatically focuses attention to where it is needed,” i.e., no sweeps through state space – Can be made more efficient with eligibility traces 13:25
70
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Eligibility traces
TD-Gammon notes
• Temporal Difference: based on single experience (s0 , r0 , s1 )
• Choose features as raw position inputs (number of pieces at each place)
Vnew (s0 ) = Vold (s0 ) + α[r0 + γVold (s1 ) − Vold (s0 )]
→ as good as previous computer programs • Longer experience sequence, e.g.: (s0 , r0 , r1 , r2 , s3 )
• Using previous computer program’s expert features
Temporal credit assignment, think further backwards: receiv-
→ world-class player
ing r3 also tells us something about V (s0 ) Vnew (s0 ) = Vold (s0 ) + α[r0 + γr1 + γ 2 r2 + γ 3 Vold (s3 ) − Vold (s0 )] • TD(λ): remember where you’ve been recently (“eligibility trace”) and update those values as well:
• Kit Woolsey was world-class player back then: – TD-Gammon particularly good on vague positions – not so good on calculable/special positions – just the opposite to (old) chess programs
e(st ) ← e(st ) + 1 ∀s : Vnew (s) = Vold (s) + α e(s) [rt + γVold (st+1 ) − Vold (st )]
• See anotated matches: http://www.bkgm.com/matches/ woba.html
∀s : e(s) ← γλe(s) • Core topic of Sutton & Barto book
• Good example for – value function approximation
→ great improvement of basic RL algorithms 13:26
– game theory, self-play
TD(λ), Sarsa(λ), Q(λ)
13:29
Detour: Dopamine • TD(λ): ∀s : V (s) ← V (s) + α e(s) [rt + γVold (st+1 ) − Vold (st )]
• Sarsa(λ) ∀s,a : Q(s, a) ← Q(s, a) + α e(s, a) [r + γQold (s0 , a0 ) − Qold (s, a)]
• Q(λ) ∀s,a : Q(s, a) ← Q(s, a) + α e(s, a) [r + γ maxa0 Qold (s0 , a0 ) −
Montague, Dayan & Sejnowski: A Framework for Mesencephalic Dopamine Systems based on Predictive Hebbian Learning. Jour-
Qold (s, a)] 13:27
nal of Neuroscience, 16:1936-1947, 1996. 13:30
TD-Gammon, by Gerald Tesauro So what does that mean?
(See section 11.1 in Sutton & Barto’s book.)
• MLP to represent the value function V (s) – We derived an algorithm from a general framework – This algorithm involves a specific variable (reward residual) – We find a neural correlate of exactly this variable
Great! Devil’s advocate:
• Only reward given at end of game for win. • Self-play: use the current policy to sample moves on both sides!
– Does not proof that TD learning is going on Only that an expected reward is compared with a experienced reward
• random policies → games take up to thousands of steps. Skilled – Does not discriminate between model-based and modelfree (Both can induce an expected reward)
players ∼ 50 − 60 steps. • TD(λ) learning (gradient-based update of NN weights) 13:28
13:31
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Limitations of the model-free view
71
• Learning – Temporal Difference & Q-learning – Limitations of the model-free view – Model-based RL
• Given learnt values, behavior is a fixed SR (or state-action) mapping • If the “goal” changes: need to re-learn values for every state in
• Exploration
the world! all previous values are obsolete
• Briefly – Imitation Learning & Inverse RL – Continuous states and actions (LSPI, Policy Gradients)
• No general “knowledge”, only values • No anticipation of general outcomes (s0 ), only of value
13:35
• No “planning” 13:32
dynamic prog. V (s)
policy π(s)
learn policy π(s)
Model−free
policy π(s)
optimize policy π(s)
learn latent costs R(s, a)
dynamic prog. V (s)
Inverse RL
learn value fct. V (s)
Imitation Learning
learn model P (s0 |s, a) R(s, a)
demonstration data D = {(s0:T , a0:T )d }n d=1
Policy Search
experience data D = {(st , at , rt )}Tt=0
Model−based
By definition, goal-directed behavior is performed to obtain a desired goal. Although all instrumental behavior is instrumental in achieving its contingent goals, it is not necessarily purposively goal-directed. Dickinson and Balleine [1,11] proposed that behavior is goal-directed if: (i) it is sensitive to the contingency between action and outcome, and (ii) the outcome is desired. Based on the second condition, motivational manipulations have been used to distinguish between two systems of action control: if an instrumental outcome is no longer a valued goal (for instance, food for a sated animal) and the behavior persists, it must not be goaldirected. Indeed, after moderate amounts of training, outcome revaluation brings about an appropriate change in instrumental actions (e.g. leverpressing) [43,44], but this is no longer the case for extensively trained responses ([30,31], but see [45]). That extensive training can render an instrumental action independent of the value of its consequent outcome has been regarded as the experimental parallel of the folk psychology maxim that wellperformed actions become habitual [9] (see Figure I).
Five approaches to learning behavior
policy π(s)
13:36
Niv, Joel & Dayan: A normative perspective on motivation. TICS,
Imitation Learning
10:375-381, 2006. 13:33
D = {(s0:T , a0:T )d }n d=1
learn/copy
→
π(s)
Model-based RL D = {(s, a, r, s0 )t }Tt=0
learn
→
P (s0 |s, a)
DP
→
V (s)
→
π(s)
• Model learning: Given data D = {(st , at , rt , st+1 )}Tt=1 estimate
• Use ML to imitate demonstrated state trajectories x0:T
Literature:
0
P (s |s, a) and R(s, a)
Atkeson & Schaal: Robot learning from demonstration (ICML 1997) Schaal, Ijspeert & Billard: Computational approaches to motor learning by imitation (Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences 2003)
For instance:
– discrete state-action: Pˆ (s0 |s, a) =
#(s0 ,s,a) #(s,a)
– continuous state-action: Pˆ (s0 |s, a) = N(s0 | φ(s, a)>β, Σ) estimate parameters β (and perhaps Σ) as for regression (including non-linear features, regularization, cross-validation!)
Grimes, Chalodhorn & Rao: Dynamic Imitation in a Humanoid Robot through Nonparametric Probabilistic Inference. (RSS 2006) Rudiger Dillmann: Teaching and learning of robot tasks via observation ¨ of human performance (Robotics and Autonomous Systems, 2004) 13:37
• Planning, for instance: – discrete state-action: model
Value Iteration with the estimated
– continuous state-action: Least Squares Value Iteration Stochastic Optimal Control (Riccati, Differential Dynamic Prog.) 13:34
Imitation Learning • There a many ways to imitate/copy the oberved policy:
Learn a density model P (at | st )P (st ) (e.g., with mixture of Gaussians) from the observed data and use it as policy (Billard et al.)
Outline • Markov Decision Processes as formal model – Definition – Value/Q-function – Planning as computing V /Q given a model
Or trace observed trajectories by minimizing perturbation costs (Atkeson & Schaal 1997) 13:38
72
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Imitation Learning
(Abbeel & Ng, ICML 2004) 13:41
Atkeson & Schaal 13:42
13:39
Continuous state/actions in model-free RL
Inverse RL
• All of this is fine in small finite state & action spaces. D = {(s0:T , a0:T )d }n d=1
learn
→
R(s, a)
DP
→
V (s)
→
π(s)
Q(s, a) is a |S| × |A|-matrix of numbers. π(a|s) is a |S| × |A|-matrix of numbers.
• Use ML to “uncover” the latent reward function in observed behavior
• In the following: – optimize a parameterized π(a|s) (policy search)
Literature:
13:43
Pieter Abbeel & Andrew Ng: Apprenticeship learning via inverse reinforcement learning (ICML 2004) Andrew Ng & Stuart Russell: Algorithms for Inverse Reinforcement Learning (ICML 2000) Nikolay Jetchev & Marc Toussaint: Task Space Retrieval Using Inverse Feedback Control (ICML 2011).
Policy gradients • In continuous state/action case, represent the policy as linear in arbitrary state features:
13:40
π(s) =
k X
φj (s)βj = φ(s)>β
(deterministic)
j=1
Inverse RL (Apprenticeship Learning) • Given: demonstrations D = {xd0:T }n d=1
π(a | s) = N(a | φ(s)>β, Σ)
(stochastic)
with k features φj .
• Try to find a reward function that discriminates demonstrations from other policies
• Basically, given an episode ξ = (st , at , rt )H t=0 , we want to esti-
– Assume the reward function is linear in some features R(x) =
mate
w>φ(x) ∂V (β) ∂β
– Iterate: 1. Given a set of candidate policies {π0 , π1 , ..}
13:44
2. Find weights w that maximize the value margin between
Policy Gradients
teacher and all other candidates max ξ w,ξ
s.t. ∀πi :
w>hφiD | {z }
value of demonstrations
≥ w>hφiπi +ξ | {z } value of πi
2
||w|| ≤ 1 3. Compute a new candidate policy πi that optimizes R(x) = >
w φ(x) and add to candidate list.
• One approach is called REINFORCE: Z Z ∂V (β) ∂ ∂ = P (ξ|β) R(ξ) dξ = P (ξ|β) log P (ξ|β)R(ξ)dξ ∂β ∂β ∂β = Eξ|β {
H H X ∂ log π(at |st ) X t0 −t ∂ log P (ξ|β)R(ξ)} = Eξ|β { γt γ rt0 } ∂β ∂β t=0 t0 =t | {z } Qπ (st ,at ,t)
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 ∂V (β) ∂β
• Another is PoWER, which requires β←β+
Eξ|β {
PH
Eξ|β {
Pt=0 H
=0
13:48
t Qπ (st , at , t)}
t=0
73
Basic topics not covered
Qπ (st , at , t)}
• Partial Observability (POMDPs) See: Peters & Schaal (2008): Reinforcement learning of motor skills with policy gradients, Neural Networks. Kober & Peters: Policy Search for Motor Primitives in Robotics, NIPS 2008. Vlassis, Toussaint (2009): Learning Model-free Robot Control by a Monte Carlo EM Algorithm. Autonomous Robots 27, 123-130. 13:45
What if the agent does not observe the state st ? → The policy π(at | bt ) needs to build on an internal representation, called belief βt .
• Continuous state & action spaces, function approximation in RL
• Predictive State Representations, etc etc... 13:49
Kober & Peters: Policy Search for Motor Primitives in Robotics, NIPS 2008. 13:46
policy π(s)
Model−free
policy π(s)
learn policy π(s)
learn latent costs R(s, a)
dynamic prog. V (s)
Inverse RL
dynamic prog. V (s)
optimize policy π(s)
Imitation Learning
learn value fct. V (s)
Model−based
learn model P (s0 |s, a) R(s, a)
demonstration data D = {(s0:T , a0:T )d }n d=1
Policy Search
experience data D = {(st , at , rt )}Tt=0
policy π(s)
– Policy gradients are one form of policy search. – There are other, direct policy search methods (plain stochastic search, e.g. “Covariance Matrix Adaptation”) 13:47
Conclusions • Markov Decision Processes and RL provide a solid framework for describing behavioural learning & planning
• Little taxonomy:
policy π(s)
learn policy π(s)
learn latent costs R(s, a)
dynamic prog. V (s)
Inverse RL
policy π(s)
optimize policy π(s)
Imitation Learning
dynamic prog. V (s)
Model−free
learn value fct. V (s)
Model−based
learn model P (s0 |s, a) R(s, a)
demonstration data D = {(s0:T , a0:T )d }n d=1
Policy Search
experience data D = {(st , at , rt )}Tt=0
policy π(s)
74
16
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Reinforcement Learning – Exploration • Don’t forget the difference between learning and planning. Planning (solving an MDP / calculating optimal state and action val-
Exploration is fundamental intelligent behavior
ues) is the problem of adapting the behavior based on a given model 14:3
• Try to control the data you collect! – Make decisions that lead to interesting data
-greedy exploration in Q-learning 1: 2: 3:
– Reflect your own knowledge; know what you don’t know, what you could learn
4: 5:
– Go to situations where you might learn something • Curiousity, fun, intrinsic motivation, life-long learning
6: 7:
8: 9: 10:
Initialize Q(s, a) = 0 repeat[for each episode] Initialize start state s repeat[for each step of episode] Choose action ( random with prob.
a
argmaxa Q(s, a) else Take action a, observe r, s0 Qnew (s, a) ← Qold (s, a) γ maxa0 Qold (s0 , a0 ) − Qold (s, a)] s ← s0 until end of episode until happy
=
+
α
[r
+
14:1
• Estimate Q(s, a) converges to Q∗ (s, a) with infinite number of
Recall Markov decision processes a0
a1
a2
s0
s1
s2
r0
r1
samples for (s, a) and appropriate α (Watkins and Dayan, 1992) • Off-policy learning: – Q-learning estimates π ∗ in form of Q∗ (line 7) – However, Q-learning does not execute π ∗ (or its current estimate thereof), but -greedy to ensure to explore (line 5)
r2
14:4
P (s0:T , a0:T , r0:T ; π) = P (s0 )P (a0 |s0 ; π)P (r0 |a0 , s0 )
QT
t=1
P (st |at-1 , st-1 )P (at |st ; π)P (rt |at , st )
“Exploration-exploitation tradeoff” • Two different types of behavior: – exploration: act with the goal to learn as much as possible; perform actions with unknown rewards / outcomes / values
– world’s initial state distribution P (s0 ) – world’s transition probabilities P (st+1 | at , st ) – world’s reward probabilities P (rt | at , st )
– exploitation: act with the goal of getting as much reward as possible;
– discount parameter γ for future rewards
perform actions which are known to produce large reward / value
– agent’s policy π(at | st ) (or deterministic at = π(st )) • Exploration-exploitation tradeoff:
not part of the world model! – two different sources of uncertainty: the world itself (not con-
be sure not to miss states and actions with large rewards; but do not waste too much time in low-reward states and actions
trolled by the agent) vs. the policy (controlled by the agent)
14:5
14:2
Recall reinforcement learning P t • Agent wants to maximize its future rewards E[ ∞ t=0 γ rt | s0 ; π]
Sample Complexity • Let A be an RL algorithm which acts in an unknown MDP, resulting in s0 , a0 , r0 , s1 , a1 , r1 , . . .
• Agent starts without a world model → no P (st+1 | at , st ), no P (rt | at , st ), no V ∗ (s), no Q∗ (s, a)
• Agent needs to learn from experience s0 , a0 , r0 , s1 , a1 , r1 , . . . which actions lead to high rewards – model-based RL: learn world model and then plan – model-free RL: learn V and Q directly (Q-learning, TDlearning)
• How can we describe and judge the exploration efficiency of A in formal terms?
• Definition (Kakade, 2003): P s ∗ Let VtA = E[ ∞ s=0 γ rt+s | s0 , a0 , r0 . . . st−1 , at−1 , rt−1 , st ]. V is the value function of the optimal policy. Let > 0 be a prescribed accuracy. The sample complexity of A is the number of timesteps t such
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
75
that VtA (st ) < V ∗ (st ) −
14:9
This is the number of timesteps where the policy of A is more than worse than the optimal policy 14:6
E 3 sketch*
PAC-MDP efficiency • Let δ > 0 be an allowed probability of failure. A is called PAC-MDP efficient if with probability 1 − δ its sample complexity scales polynomially in δ, and quantities describing the MDP
• PAC: probably (δ) approximately () correct The PAC framework is fundamental to frequentist learning theory. For instance, it can be used to derive guarantees on the generalization performance of support vector machines
Input: State s Output: Action a 1: if s is known then 2: Plan in MDPknown // Sufficiently accurate model estimates 3: if resulting plan has value above some threshold then 4: return first action of plan // Exploitation 5: else 6: Plan in MDPunknown 7: return first action of plan // Planned exploration 8: end if 9: else 10: return action with the least observations in s// Direct exploration 11: end if 14:10
• Quantities describing the MDP: number of states, number of actions, discount factor γ, maximal reward Rmax > 0, parame-
E 3 example*
ters in the transition model P (s0 | s, a), . . . 14:7
PAC-MDP efficiency • -greedy is not PAC-MDP efficient. Its sample complexity is exponential in the number of states
(Whitehead, 1991)
S. Singh (Tutorial 2005) • Examples of PAC-MDP efficient approaches: – model-based: E 3 , R- MAX
14:11
– model-free: Delayed Q-learning 14:8
3
Explicit-Exploit-or-Explore (E ) algorithm*
E 3 example*
Kearns and Singh (2002)
• PAC-MDP efficient model-based RL algorithm • Based on two previously established key ideas: – counts c(s, a) for states and actions to quantify model confidence: s is known if all actions in s sufficiently often executed – optimism in the face of uncertainty: unknown states are assumed to give maximum reward (whose value is known) • E 3 uses two MDPs:
S. Singh (Tutorial 2005)
– MDPknown : known states with (approximately exact) estimates of P (st+1 | st , at ) and P (rt | st , at ) → captures what you know and drives exploitation – MDPunknown : MDPknown without reward + special state s0 where the agent receives maximum reward → drives exploration
14:12
E 3 example*
76
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 • Rmax: ( Rmax – R (s, a) = ˆ θrsa ∗
if #s,a < n , P ∗ (s0 |s, a) = otherwise
(
ds0 s∗ θˆs0 sa
if #s other
– Guarantees over-estimation of values, polynomial PAC results! – Read about “KWIK-Rmax”! (Li, Littman, Walsh, Strehl, 2011) • Bayesian Exploration Bonus (BEB), Kolter & Ng (ICML 2009) – Choose P ∗ (s0 |s, a) = P (s0 |s, a, b) integrating over the current belief b(θ) (non-over-confident) β – But choose R∗ (s, a) = θˆrsa + with a hyperparam1+α0 (s,a)
eter α0 (s, a), over-estimating return • Confidence intervals for V -/Q-function (Kealbling ’93, Dearden
Mˆ : estimated known state MDP
M : true known state MDP
et al. ’99) 14:16
S. Singh (Tutorial 2005) 14:13
More ideas about exploration
R- MAX
• Intrinsic rewards for learning progress – “fun”, “curiousity”
Brafman and Tennenholtz (2002) • similar to E 3 ; implicit instead of explicit exploration
– in addition to the external “standard” reward of the MDP
• based on reward function
– “Curious agents are interested in learnable but yet unknown regularities, and get bored by both predictable and inherently unpredictable things.” (J. Schmidhuber)
RR- MAX (s, a) =
– Use of a meta-learning system which learns to predict the error that the learning machine makes in its predictions; meta-predictions measure the potential interestingness of situations (Oudeyer et al.)
R(s, a) c(s, a) ≥ m (s, a known) Rmax c(s, a) < m (s, a unknown)
• Is PAC-MDP efficient • Optimism in the face of uncertainty 14:14
• Dimensionality reduction for model-based exploration in continuous spaces: low-dimensional representation of the transition
Bayesian RL
function; focus exploration on relevant dimensions (A. Nouri,
• There exists an optimal solution to the exploration-exploitation
M. Littman) 14:17
trade-off: belief planning (see my tutorial “Bandits, Global Optimization, Active Learning, and Bayesian RL – understanding the common ground”)
V π (b, s) = R(s, π(b, s)) +
Z
P (b0 , s0 | b, s, π(b, s)) V π (b0 , s0 ) b0 ,s0
– Agent maintains a distribution (belief) b(m) over MDP models m – typically, MDP structure is fixed; belief over the parameters – belief updated after each observation (s, a, r, s0 ): b → b0 – only tractable for very simple problems
• Bayes-optimal policy π ∗ = argmaxπ V π (b, s) – no other policy leads to more rewards in expectation w.r.t. prior distribution over MDPs
More ideas about exploration • Exploration to reduce uncertainty of a belief p(x) – In robotics, p(x) might be the belief about the robot position x – Entropy: a probabilistic measure of information R H(p(x)) = − p(x) log p(x)dx Hp (x) is maximal if p is uniform, and minimal if p is a point mass distribution – Information gain of action a: I(a) = H(p(x)) − Ez [H(p(x0 | z, a))] (z is the potential observation) expected change of entropy when executing an action – maximizing information gain = minimizing uncertainty in belief 14:18
– solves the exploration-exploitation tradeoff 14:15
Optimistic heuristics
Digression: Active Learning Cohn, Ghahramani, Jordan (1996)
• As with UCB, choose estimators for R∗ , P ∗ that are optimistic/over-
• active choice of learning examples in a supervised learning setting:
confident h
∗
Vt (s) = max R + a
P
s0
∗
0
0
P (s |s, a) Vt+1 (s )
i
learn mapping X → Y from training examples D = {(xi , yi )m i=1 } • active learning protocol:
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 – select a new input x ˜: x ˜ may be a query, experiment, action ... – observe resulting output y˜ – incorporate new example (˜ x, y˜) into D, relearn, and repeat;
77
• Ideas for driving exploration: random actions, optimism in the face of uncertainty, maximizing learning progress and information gain
• crucial question: how to choose x ˜? • Generalization over states and actions is crucial (→ relational
• heuristics: – where we don’t have data
RL)
– where we perform poorly – where we have low confidence
• Active learning selects statistically optimal training data for effi-
– where we expect it to change our model
cient supervised learning
– where we previously found data
14:22
• in the following: select x ˜ in a statistically “optimal” manner 14:19
Digression: Active Learning (continued) • Goal: minimize variance of prediction yˆ for given x σy2ˆ = ED [(ˆ y − ED [ˆ y ])2 ] changes with new training example (˜ x, y˜)
• Choose x ˜ which minimizes the expected predictive variances conditioned on having seen x ˜: ˜] hσy2ˆ i = ED∪(˜x,˜y) [σy2ˆ | x
References • Brafman, Tennenholtz (2002). R-max - a general polynomial time algorithm for near-optimal RL. JMLR. • Cohn, Ghahramani, Jordan (1996): Active learning with statistical models. JAIR. • Kakade (2003): On the sample complexity of RL. PhD thesis. • Kearns, Singh (2002): Near-optimal reinforcement learning in polynomial time. Machine Learning Journal. • Li (2009): A unifying framework for computational RL theory. PhD thesis. • Nouri, Littman (2010): Dimension reduction and its application to modelbased exploration in continuous spaces. Machine Learning Journal. • Oudeyer, Kaplan, Hafner (2007): Intrinsic motivation systems for autonomous mental development. IEEE Evolutionary Computation. • Schmidhuber (1991): Curious model-building control systems. In Int. Joint Conf. on Neural Networks. 14:23
• How can we compute
hσy2ˆ i?
Monte Carlo approximation (sam-
pling): evaluate at a set of reference points drawn from P (x) 14:20
Digression: Active Learning (continued) • Example: mixture of Gaussians – analytic solution for calculating expected predictive variances of the learner
Cohn, Ghahramani, Jordan (1996) 14:21
Conclusions • Exploration is fundamental intelligent behavior
• RL agents need to solve the exploration-exploitation tradeoff
• Sample complexity is a measure of the exploration efficiency of an RL algorithm
78
17
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
Exercises
17.1 17.1.1
Exercise 1 First Steps
You will hand in your exercises in groups of (up to) three. If you did not write Johannes your group members, please do so as soon as possible. (
[email protected]) All exercises will be in Python and handed in via Git. Make yourself familiar with both Python and Git by reading a few tutorials and examples. You can find links to some free good tutorials at the course website at https:
around with the data in it. The last file is run_tests.sh. It runs the tests, so that you can use the test to check whether you are doing right. Note that our test suite will be different from the one we hand to you. So just mocking each function with the desired output without actually computing it will not work. You can run the tests by executing: $ sh run_tests.sh
If you are done implementing the exercise simply commit your implementation and push it to our server. $ git add e01-graphsearch.py $ git commit $ git push
Task: Implement breadth-first search, uniform-cost search, limited-depth search, iterative deepening search and A14-ArtificialIntelligence/. star as described in the lecture (A-star will be topic of the next lecture). All methods get as an input a graph, Login in our GitLab system at https://sully.informatik. a start state, and a goal state. Your methods should uni-stuttgart.de/gitlab/ with the account sent return two things: the path from start to goal, and the to you. If you did not receive an account yet, please fringe at the moment when the goal state is found (that email Johannes. latter allows us to check correctness of the implemenCreate a SSH key (if you don’t already have one) and tation). The first return value should be the found Node upload it in your profile at “Profile Settings” and “SSH (which has the path implicitly included through the parKeys”. ent links) and a Queue (one of the following: Queue, LifoQueue, PriorityQueue and NodePriorityQueue) $ ssh-keygen object holding the fringe. You also have to fill in the pri$ cat ˜/.ssh/id_rsa.pub ority computation at the put() method of the NodePriorityQue //ipvs.informatik.uni-stuttgart.de/mlr/marc/teaching/
Clone your repository with:
Iterative Deepening and Depth-limited search are a bit different in that they do not explicetly have a fringe. You $ git clone
[email protected]:ai_lecture/group_[GROUP_NUMBER].git don’t have to return a fringe in those cases, of course. Depth-limited search additionally gets a depth limit as input. A-star gets a heuristic function as input, which you can call like this: 17.1.2 Tree Search
def a_star_search(graph, start, goal, heuristic In the repository you will find the directory e01-graphsearch # ... with a couple of files. First there is e01-graphsearch.py h = heuristic(node.state, goal) with the boilerplate code for the exercise. The com# ... ments in the code define what each function is supposed to do. Implement each function and you are done with the exercise. Tips: The second file you will find is tests.py. It consists of tests that check whether your function does what they should. You don’t have to care about this file, but you can have a look in it to understand the exercise better. The next file is data.py. It consists of a very small graph and the S-Bahn net of Stuttgart as graph structure. It will be used by the test. If you like you can play
– For those used to IDEs like Visual Studio or Eclipse: Install PyCharm (Community Edition). Start it in the git directory. Perhaps set the Keymap to ’Visual Studio’ (which sets exactly the same keys for running and stepping in the debugger). That’s helping a lot. – Use the data structure Node that is provided. It has exactly the attributes mentioned on slide 26. – Maybe you don’t have to implement the ’Tree-Search’ and ’Expand’ methods separately; you might want to put
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
• Breitensuche, Tiefensuche und Suche mit ein¨ einer Bestenheitlichen Kosten sind Spezialfalle suche.
them in one little routine.
17.2
79
• Suche mit einheitlichen Kosten ist ein Spezialfall der A∗ -Suche.
Exercise 2
¨ Prasenz ¨ Dieses Blatt enthalt ubungen, die teils am 6.11. ¨ ¨ in der Ubungsgruppe besprochen werden und auch zur Klausurvorbereitung dienen. Sie sind nicht abzugeben. ¨ Studenten werden zufallig gebeten, sich an den Aufgaben zu versuchen.
¨ • Es gibt Zustandsraume, in dem iterative Tiefen¨ ¨ als Tiefensuche eine hohere Laufzeit-Komplexitat 2 suche hat (O(n ) vs. O(n)).
17.2.3
¨ Prasenzaufgabe: Greedy-Bestensuche und A∗ -Suche
17.2.1
¨ Prasenzaufgabe:
Beispiel fur ¨ Baum-
¨ Betrachten Sie die Rumanien-Karte auf Folie 4 in 03-search.pd
suche Betrachten Sie den Zustandsraum, in dem der Startzustand mit der Nummer 1 bezeichnet wird und die Nach¨ folgerfunktion fur mit den Num¨ Zustand n die Zustande mern 4n − 2, 4n − 1, 4n und 4n + 1 zuruck ¨ gibt. Nehmen Sie an, dass die hier gegebene Reihenfolge auch genau die Reihenfolge ist, in der die Nachbarn in expand durchlaufen werden und in die LIFO fringe eingetragen werden.
• Zeichnen Sie den Teil des Zustandsraums, der ¨ die Zustande 1 bis 21 umfasst. • Geben Sie die Besuchsreihenfolge (Besuch=[ein Knoten wird aus der fringe genommen, goal-check, ¨ und expandiert]) fur Tiefensuche ¨ eine beschrankte mit Grenze 2 und fur ¨ eine iterative Tiefensuche, jeweils mit Zielknoten 4, an. Geben Sie nach jedem Besuch eines Knotens den dann aktuellen Inhalt der fringe an. Die initiale fringe ist [1]. Nutzen Sie fur ¨ jeden Besuch in etwa die Notation: besuchter Zustand: [fringe nach dem Besuch] • Fuhrt ein endlicher Zustandsraum immer zu einem ¨ endlichen Suchbaum? Begrunden Sie Ihre Antwort. ¨
17.2.2
¨ ¨ der SuchPrasenzaufgabe: Spezialfalle
• Die Luftlinien-Heuristik hLL bringt Probleme fur ¨ eine Greedy-Bestensuche, wenn wir von Iasi nach Faragas gehen wollen. In umgekehrter Richtung jedoch nicht. Finden Sie einen Fall, in dem GreedySuche mit hLL fur ¨ keine Richtung den kurzesten ¨ Weg findet. • Verfolgen Sie den Weg von Lugoj nach Bukarest mittels einer A∗ -Suche und verwenden Sie die Luftlinien-Distanz als Heuristik. Geben Sie alle Knoten an, die auf dem Weg berucksichtigt wer¨ den, und ermitteln Sie jeweils die Werte fur ¨ f, g und h. • Geben Sie den mittels der A∗ -Suche gefundenen kurzesten Weg an. ¨
17.3 17.3.1
Exercise 3 Install Numpy
For this exercise you will need numpy. Please install a recent version of it. Installation notes you can find http://www.scipy.org/install.html. Any reasonably recent version will do for this exercise. (So using a version from the Ubuntu repository for instance is fine.)
strategien Beweisen Sie die folgenden Aussagen: • Breitensuche ist ein Spezialfall der Suche mit einheitlichen Kosten.
17.3.2
Constrained Satisfaction Problems
Pull the current exercise from our server to your local repository:
80
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
$ git pull Task 1: Implement backtracking for the constrained satisfaction problem definition you find in csp.py. Make three different versions of it 1) without any heuristic 2) with minimal remaining value as heuristic but without tie-breaker (take the first best solution) 3) with minimal remaining value and the degree heuristic as tie-breaker.
17.4.1
¨ Prasenzaufgabe: CSP
Betrachten Sie folgenden Kartenausschnitt:
Optional: Implement AC-3 or any approximate form of constraint propagation and activate it if the according parameter is set. Task 2: Implement a method to convert a Sudoku into a csp.ConstrainedSatisfactionProblem. The sudoku is given as a numpy array. Every empty field is set to 0. The CSP you create should cover all rules of a Sudoku, which are (from http://en.wikipedia. org/wiki/Sudoku): Fill a 9 × 9 grid with digits so that each column, each row, and each of the nine 3 × 3 sub-grids that compose the grid (also called ’boxes’, ’blocks’, ’regions’, or ’subsquares’) contains all of the digits from 1 to 9.
Der Kartenausschnitt soll mit insgesamt 4 Farben so eingef”arbt werden, dass je zwei Nachbarl”ander verschiedene Farben besitzen.
(a) Mit welchem Land w”urde man am ehesten beginnen? (b) F”arben Sie das erste Land ein und wenden Sie durchgehend Constraint Propagation an.
In the lecture we mentioned the all different constraint for columns, rows, and blocks. As the csp.ConstrainedSatisfactionProblem only allows you to represent pair-wise unequal con¨ 17.4.2 Prasenzaufgabe: Generalized Arc Constraints (to facilitate constraint propagation) you need sistency to convert this. We have n variables xi , each with the (current) domain Di . Constraint propagation by establishing local constraint consistency (“arc consistency”) in general means the following: For a variable xi and an adjacent constraint Ck , we delete all values v from Di for which there exists no tuple τ ∈ DIk with τi = v that satisfies the constraint. Consider a simple example x1 , x2 ∈ {1, 2} ,
x3 , x4 ∈ {2, .., 6} ,
c = AllDiff(x1 , .., x4 )
(a) How does constraint propagation from c to x3 update the domain D3 ?
17.4
Exercise 4
¨ Prasenz ¨ Dieses Blatt enthalt ubungen, die am 20.11. in ¨ ¨ der Ubungsgruppe besprochen werden und auch zur Klausurvorbereitung dienen. Sie sind nicht abzugeben. ¨ Studenten werden zufallig gebeten, sich an den Aufgaben zu versuchen.
(b) On http://norvig.com/sudoku.html Norvig describes his Sudoku solver, using the following rules for constraint propagation:
(1) If a square has only one possible value, then eliminate that value from the square’s peers.
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 (2) If a unit (block, row or column) has only one possible place for a value, then put the value there.
Is this a general implementation of constraint propagation for the allDiff constraint?
17.5.2
81
Modelle enumerieren (Aussagenlogik)
Betrachten Sie die Aussagenlogik mit Symbolen A, B, C und D. Insgesamt existieren also 16 Modelle. In wievielen Modellen sind die folgenden S”atze erf”ullt? 1. (A ∧ B) ∨ (B ∧ C) 2. A ∨ B
Note: The generalized arc consistency is equivalent so-called message passing (or belief propagation) in probabilistic networks, except that the messages are domain sets instead of belief vectors.
3. A ⇔ (B ⇔ C)
17.5.3
¨ Unifikation (Pradikatenlogik)
See also www.lirmm.fr/˜bessiere/stock/TR06020. pdf Geben Sie f”ur jedes Paar von atomaren S”atzen den allgemeinsten Unifikator an, sofern er existiert. Standardisieren Sie nicht weiter. Geben Sie None zuruck, ¨ wenn kein Unifikator existiert. Ansonsten ein Dictioary, dass als Key die Variable und als Value die Konstante 17.5 Exercise 5 ¨ enthalt. ¨ Abgabetermin: 3. Dez, 24:00h. Die Losungen bitte als python-Datei (siehe Vorlage in der Email) mit dem Namen e05/e05_sol.py (in Verzeichnis e05) in Euer git account einloggen. Bei Unklarheiten bitte melden.
Fur ¨ P (A), P (x):
sol3z = {’x’: ’A’}
1. P (A, B, B), P (x, y, z). 2. Q(y, G(A, B)), Q(G(x, x), y). 3. Older(F ather(y), y), Older(F ather(x), John).
17.5.1
Erfullbarkeit ¨ und allgemeine Gultigkeit ¨
4. Knows(F ather(y), y), Knows(x, x).
(Aussagenlogik) Entscheiden Sie, ob die folgenden S”atze erf”ullbar (satisfiable), allgemein g”ultig (valid) oder keins von beidem (none) sind.
(a) Smoke ⇒ Smoke (b) Smoke ⇒ F ire (c) (Smoke ⇒ F ire) ⇒ (¬Smoke ⇒ ¬F ire) (d) Smoke ∨ F ire ∨ ¬F ire (e) ((Smoke∧Heat) ⇒ F ire) ⇔ ((Smoke ⇒ F ire)∨ (Heat ⇒ F ire)) (f) (Smoke ⇒ F ire) ⇒ ((Smoke ∧ Heat) ⇒ F ire) (g) Big ∨ Dumb ∨ (Big ⇒ Dumb) (h) (Big ∧ Dumb) ∨ ¬Dumb
17.5.4
¨ Prasenzaufgabe: Matching as Constraint Satisfaction Problem
Consider the Generalized Modus Ponens (slide 09:15) for inference (forward and backward chaining) in first order logic. Applying this inference rule requires to find a substitution θ such that p0i θ = pi θ for all i. Show constructively that the problem of finding a substitution θ (also called matching problem) is equivalent to a Constraint Satisfaction Problem. “Constructively” means, explicitly construct/define a CSP that is equivalent to the matching problem. Note: The PDDL language to describe agent planning problems (slide 08:24) is similar to a knowledge in Horn form. Checking whether the action preconditions hold in a given situation is exactly the matching problem; applying the Generalized Modus Ponens corresponds to the application of the action rule on the current situation.
82 17.5.5
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 ¨ Prasenzaufgabe
def inform_move(self, move): # after each move (also your own) this f # the player of the move played (which c # chose, if you chose a illegal one. pass
In the lecture we discussed the case “A first cousin is a child of a parent’s sibling”
def get_next_move(self, board, secs): ∀x, y F irstCousin(x, y) ⇐⇒ ∃p, z P arent(p, x)∧Sibling(z, p)∧P arent(z, y) # return a move you want to play on the # seconds pass A question was whether this is equivalent to
¨ Siearent(z, konnen die mitgelieferte python-chess Bibliothek nutzen. ∀x, y, p, z F irstCousin(x, y) ⇐⇒ P arent(p, x)∧Sibling(z, p)∧P y)
Let’s simplify: Show that the following two ∀x A(x) ⇐⇒ ∃y B(y, x)
(2)
∀x, y A(x) ⇐⇒ B(y, x)
(3)
are different. For this, bring both sentences in CNF as described on slides 09:21 and 09:22 of lecture 09FOLinference.
Dieses mal gibt es keine Unittests, sondern Ihr ChessPlayer sollte in der Lage sein gegen einen anderen ¨ Spieler zu spielen und sollte – wenn moglich – gegen ¨ ¨ einen zufallig spielenden Spieler gewinnen. Sie konnen Ihre implementierung testen mit $ python2 interface.py --human um als Mensch gegen Ihren Spieler zu spielen. Oder mit $ python2 interface.py --random
17.6
Exercise 6
¨ um einen zufallig spielenden Spieler gegen ihr Programm antreten zu lassen.
Abgabetermin: 7. Jan 2015, 24:00h.
17.6.1
Schach
Implementieren Sie ein Schach spielendes Programm. Der grundlegende Python code ist dafur ¨ in Ihren Repositories. Wir haben auch bereits die Grundstruktur fur ¨ den UCT Algorithmus implementiert, so dass Sie nur die einzelnen Funktionen implementieren mussen. Sie ¨ ¨ konnen dafur und z.B. die ¨ auch die Suchtiefe verkurzen ¨ mitgelieferte sehr einfache Evaluationsfunktion – oder eine eigene – nutzen. Es ist aber Ihnen uberlassen ¨ ¨ auch einen vollig anderen Algorithmus zu implementieren (z.B. Minimax), solange das folgende Interface eingehalten wird:
17.7
Exercise 7
¨ Prasenz ¨ Dieses Blatt enthalt ubungen, die am 12.01. in ¨ ¨ der Ubungsgruppe besprochen werden und auch zur Klausurvorbereitung dienen. Sie sind nicht abzugeben. ¨ Studenten werden zufallig gebeten, sich an den Aufgaben zu versuchen.
17.7.1
¨ Prasenzaufgabe: Bedingte Wahrscheinlichkeit
1. Die Wahrscheinlichkeit, an der bestimmten tropischen Krankheit zu erkranken, betr”agt 0,02%. Ein Test, der bestimmt, ob man erkrankt ist, ist in 99,995% der F”alle korrekt. Wie hoch ist die Wahrscheinclass ChessPlayer(object): lichkeit, tats”achlich an der Krankheit zu leiden, def __init__(self, game_board, player): wenn der Test positiv ausf”allt? # The game board is the board at the beginning, the optimization is 2. Eine andere seltene Krankheit # either chess.WHITE or chess.BLACK, depending on the player youbetrifft are.0,005% aller Menschen. Ein entsprechender Test ist in 99,99% pass
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 der F”alle korrekt. Mit welcher Wahrscheinlichkeit ist man bei positivem Testergebnis von der Krankheit betroffen? 3. Es gibt einen neuen Test f”ur die Krankheit aus b), der in 99,995% der F”alle korrekt ist. Wie hoch ist hier die Wahrscheinlichkeit, erkrankt zu sein, wenn der Test positiv ausf”allt?
17.7.2
p(x, c) = p(c)
• From bandit 1: 8 7 12 13 11 9
D Y
p(xi |c)
(4)
i=1
Nun kann man fur ¨ eine neue eintreffende Email die ¨ vorkommenden Worter x∗ analysieren und mit Hilfe von Bayes Formel die Wahrscheinlichkeit berechnen, ob diese Email Spam oder Ham ist.
p(c|x∗ ) =
¨ Prasenzaufgabe: Bandits
Assume you have 3 bandits. You have already tested them a few times and received returns
83
p(x∗ |c)p(c) p(x∗ |c)p(c) =P ∗ ∗ p(x ) c p(x |c)p(c)
(5)
Aufgabe: Implementieren Sie einen Naive Bayes Klassifikator fur ¨ die Spam Emails. Sie finden Trainingsdaten und Python-Code, der mit dieser umgehen kann, in Ihrem Repository. Ihre Implementierung sollte zwei Funktionen enthalten:
• From bandit 2: 8 12 • From bandit 3: 5 13
For the returns of each bandit separately, compute a) the mean return, the b) standard deviation of returns, and c) standard deviation of the mean estimator.
class NaiveBayes(object): def train(self, database): ’’’ Train the classificator with the giv pass
def spam_prob(self, email): ’’’ Compute the probability for the give return 0.
Which bandid would you choose next? (Distinguish cases: a) if you know this is the last chance to pull a bandit; b) if you will have many more trials thereafter.)
17.8
Exercise 8
Tip: David Barber gibt ein seinem Buch “Bayesian Reasoning and Machine Learning” eine sehr gute Einfuhrung ¨ in den Naive Bayes Klassifikator (Seite 243 ff., bzw. Seite 233 ff. in der kostenlosen Online Version des Buches, die man unter http://www.cs.ucl.ac.uk/ staff/d.barber/brml/ herunterladen kann).
Abgabetermin: Mi, 28.01.2015, 23:59 h
17.9 17.8.1
Exercise 9
Spamfilter mit Naive Bayes
¨ Prasenz ¨ Dieses Blatt enthalt ubungen, die am 26.01. in ¨ ¨ Sie haben in der Vorlesung grafische Modelle und Inder Ubungsgruppe besprochen werden und auch zur ferenz in ihnen kennengelernt. Auf dieser Grundlage Klausurvorbereitung dienen. Sie sind nicht abzugeben. basiert der viel verwendete Naive Bayes Klassifikator. ¨ Studenten werden zufallig gebeten, sich an den AufDer Bayes Klassifikator wird zum Beispiel dafur ¨ vergaben zu versuchen. wandet, Spam Emails automatisch zu erkennen. Dafur ¨ ¨ werden Trainings-Emails untersucht und die Worthaufigkeiten ¨ gezahlt. Daraus werden dann D Wahrscheinlichkeiten ¨ 17.9.1 Prasenzaufgabe: Hidden Markov Modp(xi |c) fur ¨ das Auftreten eines bestimmten Wortes, gegeben elle ¨ das eine Email Spam/Ham ist, geschatzt. Nun wird die Annahme getroffen, dass all diese Wahrscheinlichkeiten ¨ unabhangig sind, so dass die Joint Verteilung wie folgt Sie stehen bei Nacht auf einer Br”ucke ”uber der B14 berechnet werden kann: in Stuttgart und m”ochten z”ahlen, wieviele LKW, Busse
84
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015
und Kleintransporter in Richtung Bad Canstatt fahren. Da Sie mehrere Spuren gleichzeitig beobachten und es dunkel ist machen Sie folgende Fehler bei der Beobachtung des Verkehrs:
17.10.1
¨ Prasenzaufgabe: Value Iteration 7 6
1
5
2
• Einen LKW erkennen Sie in 30% der F”alle als Bus, in 10% der F”alle als Kleintransporter. • Einen Bus erkennen Sie in 40% der F”alle als LKW, in 10% der F”alle als Kleintransporter. • Einen Kleintransporter erkennen Sie in je 10% der F”alle als Bus bzw. LKW. Zudem nehmen Sie folgendes an: • Auf einen Bus folgt zu 10% ein Bus und zu 30% ein LKW, ansonsten ein Kleintransporter. • Auf einen LKW folgt zu 60% ein Kleintransporter und zu 30% ein Bus, ansonsten ein weiterer LKW. • Auf einen Kleintransporter folgt zu 80% ein Kleintransporter und zu je 10% ein Bus bzw. ein LKW. Sie wissen sicher, dass das erste beobachtete Fahrzeug ¨ tatsachlich ein Kleintransporter ist. a) Formulieren Sie das HMM dieses Szenarios. D.h., geben Sie explizit P (X1 ), P (Xt+1 |Xt ) und P (Yt |Xt ) an.
8
4
3
Consider the circle of states above, which depicts the 8 states of an MDP. The green state (#1) receives a reward of r = 4096 and is a terminal state, the red state (#2) is punished with r = −512. Consider a discounting of γ = 1/2. Description of P (s0 |s, a): • The agent can choose between two actions: going one step clock-wise or one step counter-clockwise. • With probability 3/4 the agent will transition to the desired state, with probability 1/4 to the state in opposite direction. • Exception: the green state (#1) is a terminal state: The MDP terminates after the agent reached this state and collected the reward.
Description of P (r|s, a):
¨ b) Pradiktion: Was ist die Marginal-Verteilung P (X3 ) uber das 3. Fahrzeug. ¨
• The agent will receive reward of r = 4096 upon reaching the green state (#1).
c) Filtern: Sie machten die Beobachtungen Y1:3 = (K, B, B). Was ist die Wahrscheinlichkeit P (X3 |Y1:3 ) des 3. Fahrzeugs gegeben diese Beobachtungen?
• The agent receives reward of r = −512 upon reaching the red state (#2).
¨ d) Glatten: Was ist die Wahrscheinlichkeit P (X2 |Y1:3 ) des 2. Fahrzeugs, gegeben die 3 Beobachtungen? e) Viterbi (wahrscheinlichste Folge): Was ist die wahrscheinlichste Folge argmaxX1:3 P (X1:3 |Y1:3 ) an Fahrzeugen, gegeben die 3 Beobachtungen?
17.10
Exercise 10
¨ Prasenz ¨ Dieses Blatt enthalt ubungen, die am 05.02. in ¨ ¨ der Ubungsgruppe besprochen werden und auch zur Klausurvorbereitung dienen. Sie sind nicht abzugeben. ¨ Studenten werden zufallig gebeten, sich an den Aufgaben zu versuchen.
1. Perform three steps of Value Iteration: Initialize Vk=0 (s) = 0, what is Vk=1 (s), Vk=2 (s), Vk=3 (s)? 2. How could you compute the optimal value function V π (s) for a GIVEN policy (e.g., always walk clock-wise) in closed form? Provide an explicit matrix equation. 3. Assume you are given V ∗ (s). How can you compute the optimal Q∗ (s, a) form this? And assume Q∗ (s, a) is given, how can you compute the optimal V ∗ (s) from this? Provide general equations. 4. What is Qk=3 (s, a) for the example above? What is the “optimal” policy given Qk=3 ?
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 17.10.2
¨ Prasenzaufgabe: TD-learning
Consider TD-learning. The initial value function V (s) = 0. Consider the following setup: The agent starts in state 4, then permanently chooses clock-wise action. As soon as it reaches the green terminal state (one way or another), it will be beamed back to the start state 4 and everything repeats over and over again.
1. Describe at what events plain TD-learning will update the value function, how it will update it. Guess roughly how many steps the agent will have taken when for the first time V (s4 ) becomes non-zero. How would this be different for eligibility traces?
85
Index Existential quantification (6:8), Explicit-Exploit-or-Explore* (14:9), Exploration, Exploitation (9:6),
n-queens as ILP (4:19), A∗ search (2:15), A∗ : Proof 1 of Optimality (2:22), A∗ : Proof 2 of Optimality (2:28),
Factor graph (11:30), Filtering, Smoothing, Prediction (12:3), FOL: Syntax (6:5), Forward Chaining (7:15), Forward chaining (5:41), Forward checking (3:21), Frame problem (6:22), Frequentist vs Bayesian (8:4),
Admissible heuristics (2:30), Alpha-Beta Pruning (10:6), Backtracking (3:10), Backward Chaining (5:52), Backward Chaining (7:17), Bayes’ Theorem (8:11), Bayesian Network (11:3), Bayesian RL (14:15), Belief propagation (11:36), Bellman optimality equation (13:8), Bernoulli and Binomial (8:14), Best-first Search (2:3), Beta (8:15), Breadth-first search (BFS) (1:29),
Gaussian (8:27), Generalized Modus Ponens (7:14), Genetic Algorithms (4:14), Gibbs sampling (11:24), Graph search and repeated states (1:65), Greedy Search (2:5), Hidden Markov Model (12:4), HMM inference (12:6), HMM: Inference (12:5), Horn Form (5:40),
Completeness of Forward Chaining (5:51), Complexity of BFS (1:37), Complexity of DFS (1:52), Complexity of Greedy Search (2:14), Complexity of Iterative Deepening Search (1:63), Complexity of A∗ (2:27), Conditional distribution (8:9), Conditional independence in a Bayes Net (11:7), Conditional random field* (11:46), Conjugate priors (8:23), Conjunctive Normal Form (5:64), Constraint propagation (3:25), Constraint satisfaction problems (CSPs): Definition (3:2),
Imitation Learning (13:37), Importance sampling (11:22), Inference (5:28), Inference in graphical models: overview (11:17), Inference: general meaning (11:12), Inverse RL (13:40), Iterated Local Search (4:9), Iterative deepening search (1:54), Joint distribution (8:9), Junction tree algorithm (11:41),
Conversion to CNF (5:65), Conversion to CNF (7:20), CSP as ILP (4:21),
Kalman filter (12:9), Knowledge base: Definition (5:2), Kullback-Leibler divergence (8:34),
Definitions based on sets (8:6), Depth-first search (DFS) (1:39), Dirac (8:26), Dirichlet (8:19),
Local optima, plateaus (4:8), Local Search (4:5), Logical equivalence (5:37), Logics: Definition, Syntax, Semantics (5:20), Loopy belief propagation (11:39), LP, QP, ILP, NLP (4:17),
Eligibility traces (13:26), Entailment (5:21), Entropy (8:33), Epsilon-greedy exploration in Q-learning (14:4), Evaluation functions (10:11), Example: Romania (1:2), Example: The 8-Puzzle (1:15), Example: Vacuum World (1:5),
Map-Coloring Problem (3:3), Marginal (8:9), Markov Decision Process (MDP) (13:3), Markov Process (12:2), Maximum a-posteriori (MAP) inference (11:45),
86
Introduction to Artificial Intelligence, Marc Toussaint—April 7, 2015 Memory-bounded A∗ (2:34), Message passing (11:36), Minimax (10:3), Model (5:22), Model-based RL (13:34), Modus Ponens (5:40), Monte Carlo (11:19), Monte Carlo Tree Search (MCTS) (9:14), Multi-armed Bandits (9:1), Multinomial (8:18), Multiple RVs, conditional independence (8:12), Optimistic heuristics (14:16), Optimization problem: Definition (4:2), PAC-MDP efficiency (14:7), Particle approximation of a distribution (8:29), Planning Domain Definition Language (PDDL) (6:24), Policy gradients (13:44), Probabilities as (subjective) information calculus (8:2), Probability distribution (8:8), Problem Definition: Deterministic, fully observable (1:9), Proof of convergence of Q-Iteration (13:13), Proof of convergence of Q-learning (13:24), Propositional logic: Semantics (5:31), Propositional logic: Syntax (5:29), Q-Function (13:11), Q-Iteration (13:12), Q-learning (13:22), R-Max (14:14), Random variables (8:7), Reduction to propositional inference (7:6), Resolution (5:64), Resolution (7:19), Sample Complexity (14:6), Sarsa (13:21), Satisfiability (5:38), Simulated Annealing (4:11), Situation Calculus (6:21), Slack Variables (4:19), Temporal difference (TD) (13:19), Travelling Salesman Problem (TSP) (4:6), Tree search implementation: states vs nodes (1:25), Tree Search: General Algorithm (1:26), Tree-structured CSPs (3:33), TSP as ILP (4:20), UCT for games (10:12),
Unification (7:9), Uniform-cost search (1:38), Universal quantification (6:7), Upper Confidence Bound (UCB) (9:8), Upper Confidence Tree (UCT) (9:19), Utilities and Decision Theory (8:32), Validity (5:38), Value Function (13:4), Value Iteration (13:10), Value order: Least constraining value (3:20), Variable elimination (11:27), Variable order: Degree heuristic (3:19), Variable order: Minimum remaining values (3:18), Wumpus World example (5:4),
87
