Algorithms and Complexity

The sphere of algorithmic problems E.G.M. Petrakis

Algorithms and Complexity

1

ƒ Traveling Salesman Problem (TSP):

find the shortest route that passes though each of the nodes in a graph exactly once and returns to the starting node Α starting point 5

2 3 1

Ε 4 2

2

2

D

C

B 3

6

Shortest route: ABDECA Length(ABDECA) = 2 + 2 + 2 + 2 + 3 = 11 E.G.M. Petrakis

Algorithms and Complexity

2

ƒ Exhaustive search: compute all paths and

select the the one with the minimum cost

ƒ Length(ABCDEA) = 2 + 3 + 6 + 2 + 5 = 18 ƒ Length(ACBDEA) = 3 + 3 + 2 + 2 + 5 = 15 …… ƒ Length(ABDECA) = 2 + 2 + 2 + 2 + 3 = 11Í BEST PATH ƒ Length(AEBCDA) = 5 + 1 + 3 + 6 + 4 = 19 5 Ε 2

2 D

E.G.M. Petrakis

starting point

Α 4

3 1

2

2 6

B 3

(n-1)! paths Î Complexity: Ω(2n)

C

Algorithms and Complexity

3

ƒ Hard Problems: an exponential algorithm

that solves the problem is known to exist ƒ E.g., TSP ƒ Is there a better algorithm? ƒ Until when do we try to find a better algorithm? ƒ Prove that the problem as at least as hard as another hard problem for which no better solution has even been found ƒ Then, stop searching for a better solution for the first problem E.G.M. Petrakis

Algorithms and Complexity

4

ƒ NP Complete problems (NPC): hard

problems for which only exponential algorithms are known to exist ƒ Polynomial solutions might exist but none has found yet! ƒ Examples: ƒ Traveling Salesman Problem (TSP) ƒ Hamiltonian Path Problem ƒ 0-1 Knapsack Problem ƒ Properties of NP Completeness: ƒ Non deterministic polynomial ƒ Completeness

E.G.M. Petrakis

Algorithms and Complexity

5

ƒ Non-deterministic polynomial:a polynomial algorithm doesn’t guarantee optimality ƒ The polynomial algorithm is non

deterministic: tries to find a solution using heuristics ƒ E.g., TSP: the next node is the one closer to the current node Α

5 Ε 2 E.G.M. Petrakis

2 D

4

starting point 3 1

2 B

2 6

C

3

Length(ABEDCA) = 2 + 1 + 2 + 6 + 3 = 14 Non optimal

Algorithms and Complexity

6

ƒ Completeness: if an optimal algorithm

exists for one of the them, then a polynomial algorithm can be found for all ƒ It is possible to transform each problem

to another using a polynomial algorithm ƒ The complexity for solving a different problem is the complexity of transforming the problem to the original one (takes polynomial time) plus the complexity of solving the original problem (take polynomial time again)!

E.G.M. Petrakis

Algorithms and Complexity

7

ƒ Hamiltonian Path (HP) problem: is

there a path that visits each node of a graph exactly once? ƒ The exhaustive search algorithm checks

n! paths ƒ HP is NP Complete ƒ It is easy to transform HP to TSP in polynomial time ƒ Create a full graph G’ having cost 1 in edges that exist in G and cost 2 in edges that don’t belong to G

E.G.M. Petrakis

Algorithms and Complexity

8

G

2 G’ 1 1 1 1 1 2 1 2 2

ƒ The transformation of HP to TSP

takes polynomial time ƒ Cost of creating G + cost of adding extra edges: O(n)

ƒ HP: equivalent to searching for a TSP path on G’ with length n + 1

E.G.M. Petrakis

Algorithms and Complexity

9

Algorithms ƒ Types of Heuristic algorithms ƒ Greedy ƒ Local search ƒ Types of Optimal algorithms ƒ Exhaustive Search (ES) ƒ Divide and Conquer (D&C) ƒ Branch and Bound (B&B) ƒ Dynamic Programming (DP) E.G.M. Petrakis

Algorithms and Complexity

10

ƒ Greedy Algorithms: whenever they

make a selection they choose to increase the cost by the minimum amount ƒ E.g., TSP: the next node is the one closer to the current node Α

5 Ε 2

2 D

4

3 1

2

2 6

starting B point 3

Greedy algorithm: Length(BECADB) = 1 + 2 + 3 + 4 + 2 = 12 Non optimal

C

Optimal: Length(BACEDB) = 2 + 3 + 2 + 2 + 2 = 11 E.G.M. Petrakis

Algorithms and Complexity

11

ƒ In some cases, greedy algorithms find the optimal solution ƒ Knapsack problem: choosing among n objects

with different values and equal size, fill a knapsack of capacity C with objects so that the value in the knapsack is maximum ƒ Fill the knapsack with the most valuable objects ƒ Job-scheduling: which is the service order of n customers that minimizes the average waiting time? ƒ Serve customers with the less service times first ƒ Minimum cost spanning tree: next transparency

E.G.M. Petrakis

Algorithms and Complexity

12

Minimum Cost Spanning Tree (MCST) ƒ In an undirected graph G with costs, find the set of edges that has minimum total cost and keeps all nodes connected ƒ Application: connect cities by telephone in a

way that requires the minimum amount of wire ƒ MCST contains no cycles (it’s a tree) ƒ Exhaustive search takes exponential time (choose among nn-2 trees or among n! edges) ƒ Two standard algorithms (next transparency)

E.G.M. Petrakis

Algorithms and Complexity

13

Α

5 Ε 2

2 D

4

2 3 1 2 6

Α B 3

MST |V|=5

1

Ε

2 B

2

Cost=7

2

C

D

C

ƒ Prim’s algorithm: at each step take

the minimum cost edge and connect it to the tree ƒ Kruskal’s algorithm: at each step take the minimum cost edge that creates no cycles in the tree. ƒ Both algorithms are optimal!! E.G.M. Petrakis

Algorithms and Complexity

14

Prim’s algorithm: graph G=(V,E),V={1,2,…,n} function Prim(G:graph, MST: set of edges) U: set of edges; u, v: vertices; { T = 0; U = {1}; while (U != V) { (u,v) = min. cost edge: u in U, v in V T = T + {(u,v)}; U = U + {v}; } } Complexity: O(n2) why?? The tree contains n – 1 edges E.G.M. Petrakis

Algorithms and Complexity

15

1 5 1 2 5 3 5 4 4 2 3 6 5 6 6 6

G = (V,E)

1 Step 2 1 3

1 Step 1 1 3

1 1 Step 3 3

4 6

2

1 1 5 3

Step 4 4 4 6

E.G.M. Petrakis

2

2 3

1 1 5 3 5

Algorithms and Complexity

4 4

6 Step 5 4 4 6

2

2 16

Local Search ƒ Heuristic algorithms that improve a non-optimal solution ƒ Local transformation that improves a solution ƒ E.g., a solution obtained by a greedy algorithm ƒ Apply many times and as long as the solution improves ƒ Apply in difficult (NP problems) like TSP ƒ E.g., apply a greedy algorithm to obtain a solution, improve this solution using local search

ƒ The complexity of the transformation must be polynomial

E.G.M. Petrakis

Algorithms and Complexity

17

Local search on TSP: Α

5 Ε 2

greedy

2 3 1

64

2

2

D

C

6

Α

⇒

B

cost=12

starting point

3

4

Α 2 1

Ε

⇒

B Α

D local cost = 5 Ε

2 D

2

B

⇒

D local cost = 4

2 B

3

2

2

E.G.M. Petrakis

C

D

Α Ε

B B

Ε

C

cost = 11: optimal

Algorithms and Complexity

18

Divide and Conquer ƒ The problem is split into smaller sub-problems ƒ Solve the sub-problems ƒ Combine their solutions to obtain the solution of the original problem ƒ The smaller a sub-problem is, the easier it is to solve it ƒ Try to get sub-problems of equal size ƒ D&C is often expressed by recursive algorithms ƒ E.g. Mergesort

E.G.M. Petrakis

Algorithms and Complexity

19

Merge sort list mergesort(list L, int n) { if (n == 1) return (L); L1 = lower half of L; L2 = upper half of L; return merge (mergesort(L1,n/2), mergesort(L2,n/2) ); } n: size of array L (assume L some power of 2) merge: merges the sorted L1, L2 in a sorted array E.G.M. Petrakis

Algorithms and Complexity

20

Complexity: O(nlogn)

E.G.M. Petrakis

Algorithms and Complexity

21

Dynamic Programming ƒ The original problem is split into smaller

sub-problems ƒ Solve the sub-problems ƒ Store their solutions ƒ Reuse these solutions several times when the

same partial result is needed more than once in solving the main problem ƒ DP is often based on a recursive formula for solving larger problems in terms of smaller ƒ Similar to Divide and Conquer but recursion in D&C doesn’t reuse partial solutions

E.G.M. Petrakis

Algorithms and Complexity

22

ƒ DP example (1) Fibonacci numbers F ( 0) = 0

F (n)

F (1) = 0 F (n) = F (n − 1) + F (n − 2) if n ≥ 2

ƒ Using recursion the complexity is F ( n ) ∈ Θ(φ n )

ƒ Using a DP table the complexity is O(n) F(0)

E.G.M. Petrakis

F(1)

…

F(n-1)

F(n)

Algorithms and Complexity

23

0-1 Knapsack ƒ 0-1 knapsack problem: given a set of

objects that vary in size and value, what is maximum value that can be carried in a knapsack of capacity C ?? ƒ How should we fill-up the capacity in order to achieve the maximum value?

E.G.M. Petrakis

Algorithms and Complexity

24

Exhaustive (1) ƒ The obvious method to solve the 0-1

knapsack problem is by trying all possible combinations of n objects

ƒ ƒ ƒ ƒ

0

1

2

0

1

1

…

n

0

Each object corresponds to a cell If it is included its cell becomes 1 If it is left out its cell becomes 0 There are 2n different combinations

E.G.M. Petrakis

Algorithms and Complexity

25

Notation n objects s1, s2, s3, … sn: capacities v1, v2, v3, …. vn: values C: knapsack capacity Let 0