Finite Automata and Theory of Computation CSci 423/523, TTh 11:00-12:20, Jones 302

Professor Mao, [email protected], 1-3472, McGl 114

General Information I

Instructor: Weizhen Mao, [email protected], 1-3472, McGl 114

I

Office Hours: TTh 10:00 – 10:45 and W 1:00 – 2:00

I

Grader: Zeyi Tao, [email protected], T 1:00 – 2:00, McGl 139

I

Textbook: Intro to the theory of computation (any edition), Michael Sipser

I

Prerequisites/background: Linear algebra, Data structures and algorithms, and Discrete math

Use of Blackboard I

Announcements

I

Homework assignments

I

Homework solutions

I

Grades

I

Checked at least weekly

Lecture Basics I

Lectures: Slides + board

I

Lecture slides: http://www.cs.wm.edu/∼ wm/mao slides.pdf

I

Not all taught in class are in the lecture notes/slides, e.g., some example problems. So take your own notes in class

I

The use of laptops and phones is not allowed in the classroom so that you can pay attention on the lectrue

Course Organization I

Automata theory: 50%

I

Computability theory: 25%

I

Complexity theory: 25%

Grading I

10 homework assignments: 45%

I

5 in-class random quizzes: 10%

I

Midterm: 15%

I

Final: 30%

Grading Policy (will be curved) I

[90, 100]: A or A-

I

[80, 90): B+, B, or B-

I

[70, 80): C+, C, or C-

I

[60, 70): D+, D, or D-

I

[0, 60): F

Homework Submission Policy I

Hardcopy (stapled and typeset in LaTex) to be submitted at the beginning of class on the due date

I

Extensions may be permitted for illness, family emergency, and travel to interviews/conferences. Requests must be made prior to the due date

Homework Policy I

Homework must be typeset in LaTex. Nothing should be handwritten, including diagrams and tables

I

Empty-hand policy when you discuss homework problems with your classmates

I

List your collaborators for every homework problem

I

Cite all references that help you solve a problem

I

In no case you should copy verbatim from other people’s work without proper attribution, as this is considered plagiarism

Exam/Quiz Policy I

Quizzes: Closed format, 10 minutes

I

Midterm: Take-home, half-day, any source of help except from a human

I

Final Exam: Closed except a cheat sheet (A4 size, both sides), 3 hours

Writing Assignments fro CSci423W I

A paper of 3-4 pages long about a person pioneering the field of ToC and one of his/her major contributions, written in the IEEE standard format for CS conferences. The title of the paper should be in the format of “name and result”, e.g., Alan Turing and the Enigma, or Stephen Cook and the Satisfiability Problem, or Rivest, Shamir, and Adleman and the RSA Cryptosystem. More details later.

I

The class will be divided into ten groups, each being responsible for the production of one homework solution

Three areas of ToC (Sipser 0.1) Theory of Computation is to study the fundamental capabilities and limitations of computers. It contains three areas. I

Automata theory: Models of computation. Seeking a precise but concise definition of a computer.

I

Computability theory: What can and cannot a computer do? Computationally unsolvable versus computationally solvable problems. Determining whether a problem is unsolvable by computer algorithms.

I

Complexity theory: What can a computer do efficiently? Computationally hard versus computationally easy problems. For example, factorization versus sorting. Cryptography needs hard problems such as factorization to ensure security.

Languages (Sipser 0.2) Languages are central concepts in automata theory. I

Alphabet Σ: Finite and nonempty, e.g., Σ = {0, 1} and Σ = {a, b, . . . , z}.

I

String (or word), e.g., w = 01110: The empty string ε; length of a string, |w|; concatenation of two strings w1 w2 ; reverse of a string w R , and substring of a string. / Σ, Language: A language is a set of strings, e.g., {ε}, 0, A = {w | w has an equal number of 0s and 1s}, and B = {0n 1n |n ≥ 1}.

I

I

Regular operators: Let A and B be two languages. I I I

I

Union: A ∪ B = {x | x ∈ A or x ∈ B}. Concatenation: AB = {xy | x ∈ A and y ∈ B}. Star: A∗ = {x1 x2 · · · xk | all k ≥ 0 and each xi ∈ A for i = 1, 2, . . . , k }. Note ε ∈ A∗

More operators I I

I I

Intersection: A ∩ B = {x | x ∈ A and x ∈ B} Complement: A is the set of all elements under consideration that are not in A. Difference: A − B = A ∩ B. Usually, A = Σ∗ − A. Power: Ak = {x1 x2 · · · xk | xi ∈ A for i = 1, 2, · · · , k }. Note A0 = {ε} and A∗ = A0 ∪ A1 ∪ A2 ∪ · · · .

Proof techniques (Sipser 0.4) I

By definition: Convert terms in the hypothesis to their definitions.

I

By construction: To prove the existence of X , construct it.

I

By counterexample: Used to disprove something seemingly true but actually not true. Example: All primes are odd. Example: There is no integers a and b such that a mod b = b mod a.

I

By contradiction: H → C is equivalent to ¬C → ¬H or ¬C ∧ H → ¬T , where T is an axiom, a known truth, or a proven fact. Example: √ The number of primes is infinite. Example: 2 is irrational.

I

By induction: Used to prove a statement S(n) about an integer n ≥ c. There is one form of induction: Simple integer induction: S(n) for n ≥ c iff S(c) ∧ ∀k (S(k) → S(k + 1)). (What are the basis step, inductive hypothesis, and inductive step?) Example: For n ≥ 1, ∑ni=1 i 2 = 16 n(n + 1)(2n + 1).

What is a Finite Automaton?(Sipser 1.1 (pp. 31-34)) I

Finite automata are the simplest computational models for computers with an extremely limited amount of memory.

I

Use of automata theory in software applications includes: study of the behavior of digital circuits, lexical analyzer in compilers, text pattern matching, and verification of finite-state systems.

I

They are designed to accept some strings, therefore to recognize a language, which is the set of accepted strings.

I

Given an input string, a finite automaton reads the string, one symbol at a time, from left to right. Using a transition function, the FA changes from one state to another based on what symbol is being read. Eventually, it reads all symbols in the string. If at this time the FA enters a final state, then the FA halts and the string is accepted.

Example of FA I

Alphabet: Σ = {0, 1}.

I

Four states: q0 , q1 , q2 , q3 , in which q0 is the start state and q3 is a final state. 0 1 δ → q0 q0 q1 Transition function δ: q1 q0 q2 q2 q0 q3 ∗q3 q3 q3 Try w1 = 100111010 and w2 = 0011010001

I

I I

The language recognized/accepted is {w ∈ {0, 1}∗ | w contains substring 111}

DFA (Sipser 1.1 (pp. 35-44)) I

DFA M = (Q, Σ, δ, q0 , F ), where I I I I I

Q is a finite set of states, Σ is an alphabet, q0 ∈ Q is the start state, F ⊆ Q is a set of accept/final states, and δ : Q × Σ → Q is a transition function, where δ(q, a) = p is the next state of M if the current state is q and the current symbol is a.

I

Extending δ to ˆδ : Q × Σ∗ → Q: For any q ∈ Q and any w = xa ∈ Σ∗ , define ˆδ(q, x) recursively as below: I I

I

ˆδ(q, ε) = q and ˆδ(q, w) = δ(ˆδ(q, x), a)

Language recognized by DFA M: L(M) = {w|ˆδ(q0 , w) ∈ F }.

I

A language is called a regular language if some DFA recognizes it.

I

How does a DFA accept a string? A path in the diagram that starts from q0 and ends at qf ∈ F such that the concatenation of the symbols on the path matches the input string.

Example: Let A = {w ∈ {0, 1}∗ | w has even numbers of 0s and 1s} Construct DFA M such that L(M) = A. Example (Sipser p. 36): A DFA M is given below: δ → q0 ∗q1 q2

0 q0 q2 q1

1 q1 q1 q1

Describe L(M). Reminder: More examples of DFAs in Sisper pp. 37-44. Special attention to ”Designing Finite Automata” pp. 41-44.

NFA (Sipser 1.2 (pp. 47-54)) I

NFA N = (Q, Σ, δ, q0 , F ), where I I I I I

Q is a finite set of states, Σ is an alphabet, q0 ∈ Q is the start state, F ⊆ Q is a set of accept/final states, and δ : Q × (Σ ∪ {ε}) → 2Q is a transition function, where δ(q, a) = P is the set of states that N may enter if the current state is q and the current symbol is a. In the case of δ(q, ε) = P, N ignores the current symbol and goes from q to any state in P without reading any symbol.

I

I

ε-closure: For any P ⊆ Q, E(P) is the set of all states reachable from any state in P via ≥ 0 ε-transitions. Extending δ to ˆδ : Q × Σ∗ → 2Q : For any q ∈ Q and any w = xa ∈ Σ∗ , define I I

I I

ˆδ(q, ε) = E({q}) and ˆδ(q, w) = E(∪k δ(pi , a)) if w = xa and i =1 ˆδ(q, x) = {p1 , . . . , pk }.

/ Language of NFA N: L(N) = {w|ˆδ(q0 , w) ∩ F 6= 0}. How does an NFA accept a string? Among all paths from q0 to qf ∈ F , there is some path such that the concatenation of symbols on the path matches the string.

Example: A = {w ∈ {0, 1}∗ |w has 101 or 11 as substrings} Example: B = {w ∈ {0, 1}∗ |w has a 1 in the 3rd position from the right end} Example: An NFA that accepts decimal numbers (a number that may have + or − preceding it, but must have a decimal point).

DFAs ⇔ NFAs (Sipser 1.2 (pp.54-58)) Subset construction method: Given NFA N = (Q, Σ, δ, q0 , F ), construct a DFA M = (Q 0 , Σ, δ0 , q00 , F 0 ) such that L(M) = L(N). I

Q 0 = 2Q (power set), i.e., Q 0 contains all subsets of Q. Note that if |Q| = n then |Q 0 | = 2n . This is just the worst case. Since many states in M are inaccessible or dead-end states and thus may be thrown away, so in practice, |Q 0 | may be much less than 2n .

I

q00 = E({q0 }).

I

/ F 0 = {R ∈ Q 0 |R ∩ F 6= 0}.

I

For each R ∈ Q 0 and each a ∈ Σ, δ0 (R, a) = E(∪p∈R δ(p, a)).

Theorem: The equivalence of DFAs, NFAs, and RLs.

Example: A DFA that accepts decimal numbers (converted from the previous NFA). Example (Sipser p. 51): A four-state NFA and an eight-state DFA that recognize the language consisting of strings over {0, 1} with a 1 in the third position from the end. Example: A bad case for the subset construction: |QN | = n + 1 and |QD | = 2n .

Closure Properties of RL’s (Sisper 1.1 (pp. 44-47) and 1.2 (pp. 58-63)) I

Union: If A and B are regular, so is A ∪ B.

I

Concatenation: If A and B are regular, so is AB.

I

Star: If A is regular, so is A∗ . (Need a new start state.)

I

Complementation: If A is regular, so is A (which is Σ∗ − A).

I

Intersection: If A and B are regular, so is A ∩ B.

I

Difference: If A and B are regular, so is A − B.

I

Reverse: If A is regular, so is AR .

I

Homomorphism: If A is regular, so is h(A) (which is {h(w)|w ∈ A} for a homomorphism h : Σ → (Σ0 )∗ ).

I

Inverse homomorphism: If A is regular, so is h−1 (A) (where h−1 (A) = {w|h(w) ∈ A}).

Example: Prove that A = {w ∈ {a, b}∗ |w is of odd length and contains an even number of a’s} is regular. (A is the intersection of two regular languages.)

Definition of REs (Sipser 1.3 (pp. 63-66)) Regular expressions (REs) are to represent regular languages. Let L(R) be the language that regular expression R represents. A recursive definition is given below: I Basis: ε and 0 / are REs, and L(ε) = {ε} and L(0) / = 0. / For any a ∈ Σ, a is an RE and L(a) = {a}. I Induction: If R1 and R2 are REs, then I I I I

R1 ∪ R2 is an RE, with L(R1 ∪ R2 ) = L(R1 ) ∪ L(R2 ), R1 R2 is an RE, with L(R1 R2 ) = L(R1 )L(R2 ), R1∗ is an RE, with L(R1∗ ) = (L(R1 ))∗ , and (R1 ) is an RE, with L((R1 )) = L(R1 ).

Remark: I

Precedence order for regular-expression operators: Star, concatenation, and finally union. () may override this order.

I

Use of R + and R k . Algebraic laws:

I

I

I

I

I

R1 ∪ R2 = R2 ∪ R1 , (R1 ∪ R2 ) ∪ R3 = R1 ∪ (R2 ∪ R3 ), and (R1 R2 )R3 = R1 (R2 R3 ). / = R 0/ = 0, / and 0/ ∪ R = R ∪ 0/ = R, εR = Rε = R, 0R R ∪ R = R. R1 (R2 ∪ R3 ) = R1 R2 ∪ R1 R3 and (R1 ∪ R2 )R3 = R1 R3 ∪ R2 R3 . (R ∗ )∗ = R ∗ , 0/ ∗ = ε, R + = RR ∗ = R ∗ R, and R ∗ = R + ∪ ε.

Examples: 1. A regular expression for the language of strings that consist of alternating 0s and 1s: (01)∗ ∪ (10)∗ ∪ 0(10)∗ ∪ 1(01)∗ . 2. A regular expression for the language of strings with a 1 in the third position from the end: (0 + 1)∗ 1(0 + 1)(0 + 1).

RE ⇒ NFA (Sipser 1.3 (pp. 67-69)) Since REs are defined recursively, it is suitable to construct the equivalent NFAs recursively. I

/ Basis: The finite automata for regular expressions ε, 0, and a for a ∈ Σ.

I

Induction: Given the NFAs for REs R1 and R2 , what are the NFAs for R1 ∪ R2 , R1 R2 , and R1∗ ?

Example: Convert regular expression (0 ∪ 1)∗ 1(0 ∪ 1) to a finite automaton.

DFA ⇒ RE (Sipser 1.3 (pp. 69-76)) A generalized NFA (GNFA) is an NFA with REs (not symbols) on its transition arcs. Assume that the given finite automaton is a DFA M = (Q, Σ, δ, q0 , F ). We first convert the DFA to a GNFA by 1. adding a new start state s that goes to the old start state q0 via an ε-transition, 2. adding a new accept state a to which there is an ε-transition from each old accept state in F , and 3. converting symbols to regular expressions on all arcs.

Then this GNFA with |Q| + 2 states will be converted to an equivalent GNFA with |Q| + 1 states by eliminating a state that is neither s nor a. This state elimination step will be applied a total of |Q| times until there are only states s and a left in the resulting GNFA. The regular expression on the arc from s to a is the regular expression for the original DFA. Given a GNFA with k states, how can one convert it to an equivalent GNFA with k − 1 states by eliminating a state that neither s nor a? (Sipser Figure 1.63 on p.72)

Example (Siper p. 76): A three-state DFA to be converted to a regular expression. Example: A language of all strings over {0, 1} with one 1 either two or three positions from the end. Theorem: The equivalence of RLs, REs, and FAs.

Regular versus nonregular languages I

A = {0∗ 1∗ }

I

B = {0n 1n |n ≥ 0}

I

C = {w ∈ {0, 1}∗ |w has an equal number of 0s and 1s}

I

D = {w ∈ {0, 1}∗ |w has an equal # of substrings 01 and 10}

Proving nonregularity by pumping lemma (Sisper 1.4 (pp. 77-82)) Theorem (The pumping lemma for regular languages): Let A be a regular language. Then there exists a constant p (the pumping length which depends on A) such that ∀s ∈ A with |s| ≥ p, we can break s into three substrings s = xyz such that 1. |y | > 0; 2. |xy| ≤ p; and 3. ∀i ≥ 0, string xy i z ∈ A.

Proof: Let A = L(M) for some DFA M with p states. Consider any s ∈ A with s = s1 s2 · · · sn , where n ≥ p and si ∈ Σ for i = 1, 2, . . . , n. Assume that ˆδ(q0 , s1 · · · si ) = qi . On the path q0 → q1 → · · · → qn that accepts s, there are n + 1 ≥ p + 1 states. By the pigeonhole principle, there are at least two identical states on the path. Let qi = qj for some 0 ≤ i < j ≤ n be the first such pair on the path. Now we can break s = xyz as follows: I

x = s1 · · · si ;

I

y = si+1 · · · sj ; and

I

z = sj+1 · · · sn .

We can then easily verify that this partition of s satisfies all three requirements stated in the theorem, i.e., |y | > 0, |xy| ≤ p, and xy i z ∈ A for any i ≥ 0. This completes the proof.

How to use the pumping lemma to prove that a language is not regular: I

Assume that A was regular by contradiction. Then the pumping lemma applies to A. Let p be the constant in the pumping lemma.

I

Select s ∈ A with |s| = f (p) ≥ p.

I

By the pumping lemma, ∃x, y, z such that s = xyz with |y | > 0, |xy| ≤ p and xy i z ∈ A for any i ≥ 0.

I

For any x, y, z such that s = xyz, |y| > 0, and |xy| ≤ p, find i ≥ 0 such that xy i z 6∈ A. A contradiction!

Example (Sipser p. 80): Prove that B = {0n 1n |n ≥ 0} is not regular. Example: Prove that A = {1r |r is a prime} is not regular. Example (Sipser p. 81): Prove that A = {ww|w ∈ {0, 1}∗ } is not regular. 2 Example: Prove that L = {1n |n ≥ 1} is not regular. Example: Prove that A = {10n 1n | n ≥ 0} is not regular. Example: Prove that A = {(01)a 0b |a > b ≥ 0} is not regular. Example: Prove that A = {0m 1n |m 6= n} is not regular.

Prove nonregularity by closure properties To prove that A is not regular, assume it was. Find a regular language B and a language operator that preserves regularity, and then apply the operator on A and B to get a regular language C. If C is known to be nonregular, a contradiction is found. Example: Prove that C = {w ∈ {0, 1}∗ |w has an equal # of 0s and 1s} is not regular. Example: Prove that A = {0m 1n | m 6= n} is not regular. Example: Prove that A = {am bn c m+n | m, n ≥ 0} is not regular.

Context-free grammars (Sipser 2.1 (pp. 100-105)) I

CFG G = (V , Σ, R, S), where V is the set of variables, Σ is the set of terminals (alphabet), R is the set of rules in the form of V → (V ∪ Σ)∗ (head→body), and S ∈ V is the start variable.

I

The CFG that generates all palindromes (strings that read the same forward and backward) over {0, 1} is G = ({S}, {0, 1}, R, S), where R contains S → ε|0|1|0S0|1S1.

I

I

Let u, v , w be strings in (V ∪ Σ)∗ . If A → w is a rule, then uAv yields uwv , written uAv =⇒ uwv . We say u derives v , ∗ written u ⇒ v , if ∃u1 , . . . , uk ∈ (V ∪ Σ)∗ such that u =⇒ u1 =⇒ · · · =⇒ uk =⇒ v . Here, =⇒ means one step ∗ and ⇒ means zero or more steps. ∗ ∗ Leftmost and rightmost derivations: ⇒ , ⇒ , ⇒ , ⇒ . lm

I

The language of a CFG G, L(G) = {w ∈ is said to be a CFL.

I

Why called “context-free”?

rm lm ∗ ∗ Σ |S ⇒ w}.

rm

L(G)

Example: Is L = {0n 1n |n ≥ 0} a context-free language? Yes since it is the language of the context-free grammar S → 0S1|ε. Example: What is the language for grammar G with S → AA and A → AAA|bA|Ab|a? L(G) = {w ∈ {a, b}∗ |w has an even (nonzero) number of as}. Example: A CFG for simple expressions in programming languages: E → E + E|E ∗ E|(E)|I and I → Ia|Ib|I0|I1|a|b.

Parse trees I

A parse tree is a tree representation for a derivation, in which each interior node is a variable, each leaf node is either a variable, a terminal, or ε, and if an interior node is a variable A and its children are X1 , . . . , Xk , then there must be a rule A → X1 · · · Xk .

I

Yield of a parse tree: Concatenation of the leaf nodes in a parse tree rooted at the start variable. Four equivalent notions:

I

∗

1. A ⇒ w; ∗ 2. A ⇒ w; lm ∗

3. A ⇒ w; and rm 4. A parse tree with root A and yield w.

Regular grammars I

A regular grammar (RG) is a special CFG where the body of each rule contains at most one variable which, if present, must be the last symbol in the body.

I

For each regular language L there is a RG G such that L = L(G).

I

Given RG G = (V , Σ, R, S), construct an equivalent NFA N = (Q, Σ, δ, q0 , F ), where Q = V ∪ {f }, q0 = S, F = {f }, and for each rule A → wB, δ(A, w) = B and for each rule A → w, δ(A, w) = f . Example: Given RG: S → aS|bA|aB, A → aS|ε, and B → bS|ε, what is its NFA?

I

Given DFA A = (Q, Σ, δ, q0 , F ), define an equivalent RG G = (V , Σ, R, S), where V = Q, S = q0 , and for each transition δ(q, a) = p, q → ap and for each final state q ∈ F , q → ε.

Example: Given a four-state DFA for language aa∗ bb∗ a, what is its regular grammar?

Ambiguity in grammars and languages (Sipser 2.1 (pp. 105-106)) I

A CFG G = (V , Σ, R, S) is ambiguous if there is w ∈ Σ∗ for which there are at least two parse trees (or leftmost derivations).

I

Grammar G: E → E + E|E ∗ E|(E)|I and I → Ia|Ib|I0|I1|a|b is ambiguous since a+b*a has two parse trees.

I

Some ambiguous grammars have an equivalent unambiguous grammar. For example, an unambiguous grammar for the simple expressions is G0 : E → E + T |T , T → T ∗ F |F , F → (E)|I, and I → Ia|Ib|I0|I1|a|b.

I

A context-free language is said to be inherently ambiguous if all its grammars are ambiguous. For example, {an bn c m d m |m, n ≥ 1} ∪ {an bm c m d n |m, n ≥ 1}. Its grammar is S → S1 |S2 , S1 → AB, A → aAb|ab, B → cBd|cd, S2 → aS2 d|aCd, and C → bCc|bc. The grammar is ambiguous (considering abcd). There is a not so easy proof that all grammars for the language are ambiguous, thus it is inherently ambiguous.

I

There is no algorithm to determine whether a given CFG is ambiguous. There is no algorithm to remove ambiguity from an ambiguous CFG. There is no algorithm to determine whether a given CFL is inherently ambiguous.

Chomsky normal form (Sipser 2.1 (pp. 106-109)) There are several steps to rewrite a CFG. I

Eliminating ε rules for L(G) − {ε} by finding nullable ∗ variables (A ⇒ ε). For example, S → AB, A → aAA|ε, and B → bBB|ε

I

Eliminating unit rules. For example, E → T |E + T , T → F |T ∗ F , F → I|(E), and I → Ia|Ib|I0|I1|a|b

I

Eliminating useless variables (and thus associated rules) by finding nongenerating and unreachable variables. For example, S → AB|a and A → b

The Chomsky Normal Form (CNF): Any nonempty CFL without ε has a CFG G in which all rules are in one of the following two forms: A → BC and A → a, where A, B, C are variables, and a is a terminal. Note that one of the uses of CNF is to turn parse trees into binary trees. To convert a CFG to a grammar in CNF: I

Add a new start variable S0 in the case when the old start variable S appears in the body of some rules.

I

Simplify the grammar by removing ε rules, unit rules, and useless variables.

I

Convert the rules in the simplified grammar into the proper forms of CNF by adding additional variables and rules.

Example (Sipser p. 108): Given a CFG G, construct a CNF G0 such that L(G) − {ε} = L(G0 ).

6.1 PDAs (Sipser 2.2 (pp. 102-114)) I

I

PDA = NFA + Stack (still with limited memory but more than that in FAs) PDA M = (Q, Σ, Γ, δ, q0 , F ), where I I I I

I I

Q: A finite set of states Σ: A finite set of input symbols (input alphabet) Γ: A finite set of stack symbols (stack alphabet) δ: The transition function from Q × (Σ ∪ {ε}) × (Γ ∪ {ε}) to 2Q×(Γ∪{ε}) q0 : The start state F : The set of final states

I

What does δ(q, a, X ) = {(p, Y )} mean? If the current state is q, the current input symbol is a, and the stack symbol at the top of the stack is X , then the automaton changes to state p and replace X by Y . What if ε replaces a, or X , or Y?

I

The state diagram of PDAs: For transition δ(q, a, X ) = {(p, Y )}, draw an arc from state q to state p labeled with a, X → Y .

I

Instantaneous description (ID) of a PDA: (q, w, γ) represents the configuration of a PDA in the state of q with the remaining input of w yet to be read and the stack content of γ. (The convention is that the leftmost symbol in γ is at the top of the stack.)

I

Binary relation ` on ID’s: (q, aw, X β) ` (p, w, Y β) if δ(q, a, X ) contains (p, Y ). ` represents one move of the ∗

PDA, and ` represents zero or more moves of the PDA. I

Language of a PDA M (or language recognized by M) is ∗

L(M) = {w|(q0 , w, ε) ` (f , ε, γ) for f ∈ F }. I

How does a PDA check the stack is empty? At the beginning any computation, it always pushes a special symbol $ to the initially empty stack by having transition δ(q0 , ε, ε) = {(q, $)}.

Example (Sipser p. 112): A PDF that recognizes {0n 1n |n ≥ 0}. Example (Sipser p. 114): A PDA that recognized for {ai bj c k |i, j, k ≥ 0, i = j or i = k }. Example: How the PDA in the above example accepts input aabbcc.

6.2 Equivalence of PDAs and CFGs(Sipser 2.2 (pp. 115-112)) I

From CFG to PDA: Let G be the CFG for CFL A. Construct a PDA M such that L(M) = L(G) = A. Given a CFG G = (V , Σ, R, S). Define a PDA M = (Q, Σ, Γ, δ, q0 , F ) to simulate leftmost derivations in CFGs, where I

I I I

Q = {q0 , q1 , q2 , q3 } ∪ Q 0 , where Q 0 is a set of additional states that may be added later on. F = {q3 } Γ = V ∪ Σ ∪ {$} ∪ {ε} The transition function is defined as follows: I I I

I I I

δ(q0 , ε, ε) = {(q1 , $)} δ(q1 , ε, ε) = {(q2 , S)} For each rule X → Xk · · · X1 in R with k ≥ 1, δ(q2 , ε, X ) = {(q21 , X1 )}, δ(q21 , ε, ε) = {(q22 , X2 )}, . . ., δ(q2k −1 , ε, ε) = {(q2 , Xk )} For eachh rule X → ε in R, δ(q2 , ε, X ) = {(q2 , ε)} For each a ∈ Σ, δ(q2 , a, a) = {(q2 , ε)}. δ(q2 , ε, $) = {(q3 , ε)}

Example (Sipser p. 118): S → aTb|b, T → Ta|ε.

I

From PDA to CFG: Let M be a PDA, where there is only one final state, the stack is emptied upon acceptance, and any transition either pushes or pops (but not both or neither). Construct a CFG G such that L(M) = L(G). Given a PDA M = (Q, Σ, Γ, δ, q0 , {qf }). Define CFG G = (V , Σ, R, S), where V = {Apq |p, q ∈ Q}, S = Aq0 qf , and R has the following rules: I I

I

For each p ∈ Q, App → ε. (other rules for App later) For each p, q, r ∈ Q, Apq → Apr Arq . (Stack becomes empty at least once between states p and q.) For each p, q, r , s ∈ Q, t ∈ Γ, and a, b ∈ Σ ∪ {ε}, if (r , t) ∈ δ(p, a, ε) and (q, ε) ∈ δ(s, b, t), then Apq → aArs b. (Stack never becomes empty between states p and q.)

Note: Variable Apq is intended to generate all strings that can take M from state p with an empty stack to state q with an empty stack. This is why we set S = Aq0 qf . Theorem: The equivalence of PDAs, CFGs, and CFLs.

7.1 Proving non-CFLs by pumping lemma (Sipser 2.3 (pp. 123-127)) Lemma: Consider a parse tree according to a CNF grammar with a yield of w ∈ Σ∗ . If the height of the tree is h then |w| ≤ 2h−1 . Theorem (The pumping lemma for context-free languages): Let A be a CFL. Then there exists a constant p (the pumping length) such that ∀s ∈ A with |s| ≥ p, we can write s = uvxyz such that 1. |vy| > 0; 2. |vxy | ≤ p; and 3. ∀i ≥ 0, string uv i xy i z ∈ A.

Proof: Given a CFL A, there is a CNF G = (V , Σ, R, S) such that L(G) = A − {ε}. Let m = |V | and choose p = 2m . Suppose s ∈ L(G) with |s| ≥ p. Any parse tree for s must have a height of at least m + 1, otherwise |s| ≤ 2m−1 = p/2 by Lemma. Therefore, there must be a path A0 (S), A1 , . . . , Al , a with at least m + 1 variables, i.e., l ≥ m, in the parse tree for s. Since there are only m different variables in the grammar, by the pigeonhole principle, there must be at least two identical variables on the path. Choose the identical pair closest to the leaf, i.e., Ai = Aj with l − m ≤ i < j ≤ l.

Then it is possible to divide the parse tree such that x is the yield of the subtree rooted at Aj , vxy is the yield of the subtree rooted at Ai , and s = uvxyz is the yield of the entire parse tree. Next we examine whether this partition satisfies all three conditions stated in the pumping lemma. First, since there is no unit production in CNF, v and y can not both be ε. So |vy| > 0. Second, since the height of the subtree rooted at Ai is at most m + 1, its yield vxy has length at most 2m = p. So |vwx| ≤ p. ∗ ∗ Third, since Ai = Aj = T , we have T ⇒ vTy and T ⇒ x from the ∗ parse tree. So for any i ≥ 0, T ⇒ v i xy i , thus, ∗ ∗ S ⇒ uTz ⇒ uv i xy i z. So uv i xy i z ∈ A for i ≥ 0. This completes the proof of the pumping lemma.

How to use the pumping lemma to prove that a language is not context-free: I

Assume that A was context-free by contradiction. Then the pumping lemma applies to A. Let p be the constant in the pumping lemma.

I

Select s ∈ A with |s| = f (p) ≥ p.

I

By the pumping lemma, s = uvxyz with |vy| > 0, |vxy| ≤ p, and uv i xy i z ∈ A ∀i ≥ 0.

I

For any u, v , x, y , z such that s = uvxyz, |vy| > 0, and |vxy | ≤ p, find i ≥ 0 such that uv i xy i z 6∈ A. A contradiction!

Example (Sipser p. 126): Prove that B = {an bn c n |n ≥ 0} is not context-free. Example (Sipser p. 127): Prove that D = {ww|w ∈ {0, 1}∗ } is not context-free. Example: Prove that {0i 1j |j = i 2 } is not context free.

7.2 Proving non-CFLs by closure properties I

Closed under union: If A and B are context-free, so is A ∪ B.

I

Closed under concatenation: If A and B are context-free, so is AB.

I

Closed under star: If A is context-free, so is A∗ .

I

Closed under reverse: If A is context-free, so is AR .

I

Not closed under intersection: Consider A = {an bn c m } and B = {am bn c n }.

I

Not closed under complementation: Note that A ∩ B = A ∪ B.

I

Not closed under difference: Note that A = Σ∗ − A.

I

Intersect with a regular language: If A is context-free and B is regular, then A ∩ B is context-free.

I

Difference from a regular language: If A is context-free and B is regular, then A − B is context-free. Note that A − B = A ∩ B.

The closure properties of CFLs may be used to prove or disprove that a given language is context free. Example: A = {w ∈ {a, b, c}∗ |w has equal numbers of as, bs and cs} is not a CFL.

Computability Theory: The question of what computers can do/solve, or equivalently, what languages can be defined/recognized by any computational device whatsoever.

8.1 Unsolvable problems I

Two types of problems: “Solve this” and “decide this”.

I

Decision problems (the “decide this” type) have a yes/no solution. They are just as hard as their “solve this” version in the sense of dealing with important questions in complexity theory.

I

A problem is said to be unsolvable/undecidable if it cannot be solved/decided using a computer.

I

Recall that a decision problem is really membership of a string in a language. For example, the problem of primality testing is actually the language of all prime numbers in binary representation.

I

I

The number of problems/languages over an alphabet with more than one symbol is uncountably infinite. However, the number of programs that a computer may use to solve problems is countably infinite. Therefore, there are more problems than there are programs. Thus, there must be some undecidable problems. An undecidable problem (the halting problem): I I

Input: Any program P and any input I to the program; Output: “Yes” if P terminates on I and “No” otherwise.

8.2 Turing machine (Sipser 3.1 (pp. 137-147)) I

I

A Turing machine includes a control unit, a read-write head, and a one-way infinite tape. TM M = (Q, Σ, Γ, δ, q0 , qaccept , qreject ), where I I

I I

I I I

Q: The finite set of states for the control unit. Σ: An alphabet of input symbols, not containing the “blank symbol” t (or B). Γ: The complete set of tape symbols. Σ ∪ {t} ⊂ Γ. δ: The transition function from Q × Γ to Q × Γ × D, where D = {L, R}. q0 : The start state. qaccept : The accept state. qreject : The reject state.

I

Configuration: X1 · · · Xi−1 qXi · · · Xn is a configuration (snapshot) of the TM in which q is the current state, the tape content is X1 · · · Xn , and the head is scanning Xi . I

I

If δ(q, Xi ) = (p, Y , L), then X1 · · · Xi−1 qXi · · · Xn ` X1 · · · Xi−2 pXi−1 YXi +1 · · · Xn . If δ(q, Xi ) = (p, Y , R), then X1 · · · Xi−1 qXi · · · Xn ` X1 · · · Xi−1 YpXi +1 · · · Xn .

I

Starting configuration q0 w, accepting configuration uqaccept v , and rejecting configuration uqreject v , where the latter two are called the halting configurations.

I

Language of a Turing machine M (or language recognized ∗

by M) is L(M) = {w ∈ Σ∗ |q0 w ` αqaccept β for α, β ∈ Γ∗ }.

I

For any given input, a TM has three possible outcomes: accept, reject, and loop. Accept and reject mean that the TM halts on the given input, but loop means that the TM does not halt on the input.

I

A language A is Turing-recognizable (or recursively enumerable) if there is a TM M such that A = L(M). In other words, ∀w ∈ A, M accepts w by entering qaccept . However, ∀w 6∈ A, M may reject or loop.

I

A language A is Turing-decidable (or decidable, or recursive) if there is a TM M such A = L(M) and M halts on all inputs. In other words, ∀w ∈ A, M accepts w and ∀w 6∈ A, M rejects w. Such TM’s are a good model for algorithms.

Example: Give a TM M with L(M) = {0n 1n |n ≥ 0}. Example (Sipser p. 143): Give a TM M that decides n A = {02 |n ≥ 0}.

8.3 Properties of TDLs and TRLs Theorem: A Turing-decidable language is also Turing-recognizable, but not vice versa. Theorem: A and A: I

If A is Turing-decidable, so is A.

I

If A and A are both Turing-recognizable, then A is Turing-decidable.

I

For any A and A. we have one of the following possibilities: (1) Both are Turing-decidable; (2) Neither is Turing-recognizable; (3) One is Turing-recognizable but not decidable, the other is not Turing-recognizable.

Some additional closure properties: TRLs and TDLs both are closed under I

Union

I

Intersection

I

Concatenation

I

Star

8.4 Variations of TMs (Sipser 3.2 (pp. 148-159)) I

TM with multi-tapes (and multi-cursors) (δ : Q × Γk → Q × Γk × {L, R}k ).

I

TM with multi-strings (and multi-cursors).

I

TM with multi-cursors.

I

TM with multi-tracks.

I

TM with two-way infinite tape.

I

TM with multi-dimensional tape.

I

Nondeterministic TM’s (δ : Q × Γ → 2Q×Γ×D ).

Theorem: The equivalent computing power of the above TM’s: For any language L, if L = L(M1 ) for some TM M1 with multi-tapes, multi-strings, multi-cursors, multi-tracks, two-way infinite tape, multi-dimensional tape, or nondeterminism, then L = L(M2 ) for some basic TM M2 .

Theorem: The equivalent computing speed of the above TM’s except for nondeterministic TM’s: For any language L, if L = L(M1 ) for some TM M1 with multi-tapes, multi-strings, multi-cursors, multi-tracks, two-way infinite tape, or multi-dimensional tape in a polynomial number of steps, then L = L(M2 ) for some basic TM M2 in a polynomial number of steps (with a higher degree). Or in other words, all reasonable models of computation can simulate each other with only a polynomial loss of efficiency. Note: The speed-up of a nondeterministic TM vs. a basic TM is exponential.

The Church-Turing Thesis: Any reasonable attempt to model mathematically algorithms and their time performance is bound to end up with a model of computation and associated time cost that is equivalent to Turing machines within a polynomial. (The power of TM.)

9.1 Hilbert’s tenth problem I

The Hilbert’s tenth problem (proposed in 1900 among a list of 23 open problems for the new century): Devise a process with a finite number of operations that tests whether a polynomial has an integral root. What Hilbert meant by “a process with a finite number of operations” is an algorithm.

I

Formulating Hilbert’s problem with today’s terminology: Is there an algorithm to test whether a polynomial has an integral root? (If yes, give the algorithm.) Or, define a language D = {p|p is a polynomial with integral root}. Is there a Turing machine to decide D? Here p, although a polynomial, is treated as a string. Note: In 1970, it was proved that D is not Turing-decidable (or undecidable).

9.2 A binary encoding scheme for TMs I

TM ⇔ binary number. Q = {q1 , q2 , . . . , q|Q| } with q1 to be the start state, q2 to be the accept state, and q3 to be the reject state. Γ = {X1 , X2 , . . . , X|Γ| }. D = {D1 , D2 } with D1 to be L and D2 to be R. A transition δ(qi , Xj ) = (qk , Xl , Dm ) is coded as 0i 10j 10k 10l 10m . A TM is coded as C1 11C2 11 · · · 11Cn , where each C is the code for a transition.

I

TM M with input w is represented by < M, w > and coded as M111w.

I

Using similar schemes, we can encode DFA, NFA, PDA, RE, and CFG.

9.3 Decidable languages (Sipser 4.1 (pp. 166-173)) The following languages are decidable by TMs. I

ADFA = {< B, w > |B is a DFA that accepts string w}.

I

ANFA = {< B, w > |B is an NFA that accepts string w}.

I

AREX = {< R, w > |R is a regular expression that generates string w}. / EDFA = {< B > |B is a DFA and L(B) = 0}.

I I

EQDFA = {< B1 , B2 > |B1 and B2 are DFAs and L(B1 ) = L(B2 )}.

I I

ACFG = {< G, w > |G is a CFG that generates string w}. / ECFG = {< G > |G is a CFG and L(G) = 0}.

I

Every CFL is decidable.

9.4 Diagonalization (Sipser 4.2 (pp. 174-179)) I

The size of an infinite set: Countably infinite and uncountably infinite.

I

Diagonalization to prove a set to be uncountably infinite. Example (Sipser p. 175): Q, the set of positive rational numbers, is countably infinite. Example (Sipser p. 177): R, the set of real numbers, is uncountably infinite.

I

Some languages are not Turing-recognizable. (Or equivalently, there are more languages than Turing machines. Since the number of Turing machines is countable, we wish to prove that the number of languages over an alphabet is uncountable.)

9.4 A non-TRL I

Enumerating binary strings: ε, 0, 1, 00, 01, 10, 11, · · · . The ith string, wi , is the ith string in the above lexicographic ordering.

I

Let the ith TM, Mi , be the TM whose code is wi , the ith binary string. If wi is not a valid TM code, then let Mi be the / TM that immediately rejects any input, i.e., L(Mi ) = 0.

I

Define the diagonalization language AD = {wi |wi 6∈ L(Mi )}. A boolean table where the (i, j) entry indicates whether TM Mi accepts string wj . Language AD is made by complementing the diagonal.

I

AD is not Turing-recognizable. Proof: Suppose, by contradiction, there is a TM M such that AD = L(M). Then M = Mi with code wi for some i. wi ∈ AD iff wi 6∈ L(Mi ) by definition of AD . wi ∈ AD iff wi ∈ L(Mi ) by AD = L(Mi ). A contradiction.

9.5 A TRL but non-TDL (Sipser 4.2 (pp. 173-174 and 179-182)) I

A universal TM: I

I

I

Each TM (among those discussed) can only solve a single problem, however, a computer can run arbitrary algorithms. Can we design a general-purposed TM that can solve a wide variety of problems? Theorem: There is a universal TM U which simulates an arbitrary TM M with input w and produces the same output. TM U is an abstract model for computers just as TM M is a formal notion for algorithms.

I

Let ATM = {< M, w > |M is a TM and M accepts string w} ATM is TR since it can be recognized by TM U. ATM is called the universal language.

I

ATM is non-TD. Proof: Assume that ATM is decided by TM T , Then on input < M, w >, T accepts iff M accepts w. Define TM D, which on input < M >, runs T on input < M, < M >> and accepts iff T rejects < M, < M >>. Feed < D > to D. We see that D accepts < D > iff T rejects < D, < D >> iff D does not accept < D >. A contradiction. Diagonalization is used in this proof. Why?

10.1 Reducibility (Sipser 5 (pp. 187-188)) We say that problem A reduces (or is reducible) to problem B, if we can use a solution to B to solve A (i.e., if B is decidable/solvable, so is A.). We may use reducibility to prove undecidability as follows: Assume we wish to prove problem B to be undecidable and we know a problem A that has already been proved undecidable. We use contradiction. Assume B is decidable. Then there exists a TM MB that decides B. If we can use MB as a sub-routine to construct a TM MA that decides A, we have a contradiction. The construction of TM MA using TM MB establishes that A is reducible to B.

10.2 Another proof of the non-TD ATM Recall ATM = {< M, w > |w ∈ L(M)}. Thus ATM = {< M, w > |w 6∈ L(M)}. Recall AD = {wi |wi 6∈ L(Mi )}. Proof: Assume that ATM is decidable. Then ATM must be decidable. Let M be the TM that decides ATM . We will construct a TM MD that would decide AD , an undecidable language. MD works as follows: For input wi (the ith binary string in the lexicographic sequence of all binary strings), it first makes a string wi 111wi and then feed it to M. We notice that wi 111wi ∈ L(M) iff wi 111wi ∈ ATM iff wi 111wi 6∈ ATM iff wi 6∈ L(Mi ) (recall that Mi is the TM with code wi ) iff wi ∈ L(MD ). So MD accepts wi iff M accepts wi 111wi . We just proved that AD is reducible to ATM .

10.3 The halting problem (Sipser 5.1 (pp. 188-189)) Let HALTTM = {< M, w > |M is a TM and M halts on string w}. HALTTM is TR since it can be recognized by TM U. HALTTM is non-TD. Proof: We will reduce ATM to HALTTM . Assume TM R decides HALTTM . We construct TM S that decides ATM as follows: On input < M, w > where M is a TM and w is a string, S first run TM R on < M, w >, if R rejects, rejects. If R accepts, simulate M on w until it halts. If M accepts, accept; if M rejects, reject.

10.4 Other non-TD problems (Sipser 5.1 (pp. 189-192)) The following problems about Turing machines are non-TD: I Whether L(M) = 0 / for any TM M. (See proofs below.) I

Whether L(M1 ) = L(M2 ) for any two TMs M1 and M2 .

I

Whether L(M) is finite for any TM M

I

Whether ε ∈ L(M) for any TM M.

I

Whether L(M) = Σ∗ for any TM M.

/ is non-TD. ETM = {< M > |M is a TM and L(M) = 0} Proof: Reduce ATM to ETM . Assume that ETM is decidable. Let R be the TM that decides ETM . We use R to construct TM S that decides ATM as follows: On input < M, w >, I

Construct TM M1 which on input x, rejects if x 6= w and simulates M on w if x = w.

I

Run R on < M1 >.

I

If R accepts, reject and if R rejects, accept.

/ is TR but non-TD. NETM = {< M > |M is a TM and L(M) 6= 0} Proof: To prove that NETM is TR, we design a TM MNE to recognize NETM . On input < M >, I

MNE systematically generates strings w: ε, 0, 1, 00, 01, . . . and use the universal TM U to test whether M accepts w. (What if M never halts on w? Run M on w1 , . . . , wi for i steps for i = 1, . . ..)

I

If M accepts some w, then MNE accepts its own input M.

We next prove that NETM is non-TD. Assume that there is a TM MNE that decides NETM , i.e., TM MNE determines whether L(M) 6= 0/ for any TM M. We will use MNE to construct a TM Mu that would decides the undecidable ATM . On input < M, w >, I

Mu constructs a new TM M 0 , which rejects if its input is not w and mimics M if its input is w.

I

M 0 is then fed to MNE .

I

MNE accepts its input M 0 iff L(M 0 ) 6= 0/ iff M accepts w.

Note: ETM is non-TR. Rice’s Theorem: Every nontrivial property of the TRLs is undecidable.

10.5 More non-TD problems (Sipser 5.2 (pp. 199-205)) Post’s correspondence problem (PCP) is non-TD. INSTANCE: P = { bt11 , bt22 , . . . , btkk }, where t1 , t2 , . . . , tk and b1 , b2 , . . . , bk are strings over alphabet Σ. (P can be regarded as a collection of dominos, each containing two strings, with one stacked on top of the other.) QUESTION: Does P contain a match, i.e., i1 , i2 , . . . , il ∈ {1, 2, . . . , k } with l ≥ 1 such that ti1 ti2 · · · til = bi1 bi2 · · · bil ? Equivalently, defined as a language, we have LPCP = {< P > |P is an instance of PCP with a match}. b a ca abc , ab , a , c }, sequence 2, 1, 3, 2, 4 For example, for P1 = { ca ca acc indicates a match. For P2 = { abc ab , a , ba }, there is no match.

Any nontrivial property that involves what a program does is undecidable. For example, whether a program prints a certain message, whether it terminates, or whether it calls a certain function. It is undecidable whether a CFG is ambiguous. Let G1 and G2 be CFG’s and let R be a regular expression. It is undecidable whether I L(G1 ) ∩ L(G2 ) = 0. / I

L(G1 ) = L(G2 ).

I

L(G1 ) = L(R).

I

L(G1 ) = Σ∗ .

I

L(G1 ) ⊆ L(G2 ).

I

L(R) ⊆ L(G1 ).

Complexity Theory: I

Theory of computability is the study of what can or cannot be computed by a TM/computer, among all problems.

I

Theory of complexity is the study of what can or cannot be computed efficiently by a TM/computer, among all decidable problems.

11.1 The class of P (Sipser 7.2) I

Definition: P is the class of problems solvable in polynomial time (number of steps) by deterministic TMs. O(nc ), where c is a constant. Tractable (not so hard).

I

Why use polynomial as the criterion? – If a problem is not in P, it will be extremely expensive and probably impossible to solve in practice for large sizes. – Polynomials have nice closure properties: +, −, ∗, and composition. – P is independent of models of computation: Basic TM, TMs with multi-tracks, multi-tapes, multi-cursors, multi-strings, multi-dimensional tape. (All variations of TM except nondeterministic TM.)

I

Examples of problems in P: Sorting, Searching, Selecting, Minimum Sapnning Tree, Shortest Path, etc..

11.2 The class of NP (Sipser 7.3) I

An NTM is an unrealistic (unreasonable) model of computation which can be simulated by other models with an exponential loss of efficiency. It is a useful concept that has had great impact on the theory of computation.

I

NTM N = (Q, Σ, Γ, δ, q0 , qaccept , qreject ), where δ : Q × Γ → 2P for P = Q × Γ × {L, R}.

I

δ(q, X ) is a set a moves. Which one to choose? This is nondeterminism. The computation can be illustrated by a tree, with each node representing a configuration.

I

Nondeterminism can be viewed as a kind of parallel computation wherein multiple independent processes or threads can be running concurrently. When a nondeterministic machine splits into several choices, that corresponds to a process forking into several children, each then proceeding separately. If at least one process accepts, then the entire computation accepts.

I

Time complexity of nondeterministic TMs (NTMs): Let N be an NTM that is a decider (where all computation paths halt in the tree for any input). The time complexity of N, f (n), is the maximum number of steps that N uses on any computation path for any input of length n. In other words, f (n) is the maximum height of all computation trees for all input of length n.

I

An unreasonable model of computation: Theorem; Every T (n)-time multi-tape TM has an equivalent O(T 2 (n))-time single-tape TM. Theorem: Every T (n)-time single-tape NTM has an equivalent O(2O(T (n)) )-time single-tape DTM.

I

Definition: NP is the class of problems solvable in polynomial time by nondeterministic TMs.

I

Another definition of nondeterministic TMs (algorithms): I I

Guessing phase: Guess a solution (always on target). Verifying phase: Verify the solution.

Example: TSP (DEC) is in NP. INSTANCE: An edge-weighted, undirected, and complete graph G(V , E, w) and a bound B ≥ 0. (The weight on an edge is allowed to be +∞.) QUESTION: Is there a tour (a cycle that passes through each node exactly once) in G with total weight no more than B? A NTM N that decides TSP in polynomial time can be defined as follows: Assume that a coded instance of TSP is shown on the tape as input. N first generates an arbitrary permutation of the n nodes and then checks whether the permutation represents a tour with total weight no larger than the bound B. If so, N accepts the input. If not. N rejects the input, perhaps hoping that another choice of permutations may end up accepting. Note that the permutation generated can be any of the n! possible permutations. The nondeterminism of the machine guarantees that if the instance is a yes-instance, the right permutation will always be selected.

Note: The acceptance of an input by a nondeterministic machine is determined by whether there is an accepting computation among all, possibly exponentially many, computations. In the above proof, if there is a solution, it will always be generated by the Turing machine. This is like that a nondeterministic machine has a guessing power. A tour can only be found by a deterministic machine in exponential time, however, it can be found by a nondeterministic machine in just one step. So any nondeterministic proof should always contain two stages: Guessing and verifying.

Example: Graph Coloring is in NP. Consider the decision problem: For a given graph, is there a coloring scheme of the nodes that uses no more than B colors such that no two nodes connected by an edge are given the same color? A nondeterministic TM first guesses a coloring scheme (in polynomial time) and then verifies if the coloring uses no more than B colors and if any two adjacent nodes have different colors (in polynomial time).

I

Theorem: P⊆NP. Any deterministic TM is a special case of a nondeterministic TM.

I

Theorem: Any Π ∈NP can be solved by a deterministic TM in time O(c p(n) ) for some c > 0 and polynomial p(n).

I

Open problem: NP⊆P? or P=NP?

11.3 Polynomial reduction (Sipser 7.4) I

Definition: Let Π1 and Π2 be two decision problems, and {I1 } and {I2 } be sets of instances for Π1 and Π2 , respectively. We say there is a polynomial reduction from Π1 to Π2 , or Π1 ∝p Π2 if there is f : {I1 } → subset of {I2 } such that (1) f can be computed in polynomial time and (2) I1 has a “yes” solution if and only if f (I1 ) has a “yes” solution.

I

Theorem: If Π1 ∝p Π2 , then Π2 ∈P implies Π1 ∈P.

I

Theorem: If Π1 ∝p Π2 and Π2 ∝p Π3 , then Π1 ∝p Π3 .

I

Remark: ∝p means “no harder than”.

11.4 The class of NPC (Sipser 7.4) I

Definition: NPC (NP-complete) is the class of the hardest problems in NP, or equivalently, Π ∈NPC if Π ∈NP and ∀Π0 ∈NP, Π0 ∝p Π.

I

Theorem: If Π1 ∈NPC, Π2 ∈NP, and Π1 ∝p Π2 , then Π2 ∈NPC.

I

Theorem: If ∃Π ∈NPC such that Π ∈P, then P=NP.

I

Theorem: If ∃Π ∈NPC such that Π 6∈ P, then P6=NP.

Satisfiability (SAT): INSTANCE: A boolean formula α in CNF with variables x1 , . . . , xn (e.g., α = (x1 ∨ x2 ) ∧ (¬x2 ∨ x3 )). QUESTION: Is α satisfiable? (Is there a truth assignment to x1 , . . . , xn such that α is true?) Cook’s Theorem: SAT∈NPC. (Need to prove (1) SAT∈NP and (2) ∀Π ∈NP, Π ∝p SAT.)

Proof of Cook’s Theorem: SAT is clearly in NP since a NTM exists that guesses a truth assignment and verifies its correctness in polynomial time. Now we wish to prove ∀Π ∈NP, Π ∝p SAT, or equivalently, for any polynomial-time NTM M, L(M) ∝p LSAT . For any NTM M, assume Q = {q0 , q1 (accept), q2 (reject), . . . , qr } and Γ = {s0 , s1 , s2 , . . . , sv }. Also assume that the time is bounded by p(n), where n is the length of the input. We wish to prove that there is a function fM : Σ∗ → {instances of SAT} such that ∀x ∈ Σ∗ , x ∈ L(M) iff fM (x) is satisfiable. In other words, we wish to use a Boolean expression fM (x) to describe the computation of M on x. Variables in fM (x): — State: Q[i, k]. M is in qk after the ith step of computation (at time i). — Head: H[i, j]. Head points to tape square j at time i. — Symbol: S[i, j, l]. Tape square j contains sl at time i. (Assume the tape is one-way infinite and the leftmost square is labeled with 0.) For example, initially i = 0. Assume the configuration is q abba.

11.5 NP-complete problems How to prove Π is NP-complete: I

Show that Π ∈NP.

I

Choose a known NP-complete Π0 .

I

Construct a reduction f from Π0 to Π.

I

Prove that f is a polynomial reduction by showing that (1) f can be computed in polynomial time and (2) ∀I 0 for Π0 , I 0 is a yes-instance for Π0 if and only if f (I 0 ) is a yes-instance for Π.

Seven basic NP-complete problems. I

3-Satisfiability (3SAT): INSTANCE: A formula α in CNF with each clause having three literals. QUESTION: Is α satisfiable?

I

3-Dimensional Matching (3DM): INSTANCE: M ⊆ X × Y × Z , where X , Y , Z are disjoint and of the same size. QUESTION: Does M contain a matching, which is M 0 ⊆ M with |M 0 | = |X | such that no two triples in M 0 agree in any coordinate?

I

PARTITION: INSTANCE: A finite set A of numbers. QUESTION: Is there A0 ⊆ A such that ∑a∈A0 a = ∑a∈A−A0 a?

I

Vertex Cover (VC): INSTANCE: A graph G = (V , E) and 0 ≤ k ≤ |V |. QUESTION: Is there a vertex cover of size ≤ k , where a vertex cover is V 0 ⊆ V such that ∀(u, v ) ∈ E, either u ∈ V 0 or v ∈ V 0 ?

I

Hamiltonian Circuit (HC): INSTANCE: A graph G = (V , E). QUESTION: Does G have a Hamiltonian circuit, i.e., a tour that passes through each vertex exactly once?

I

CLIQUE: INSTANCE: A graph G = (V , E) and 0 ≤ k ≤ |V |. QUESTION: Does G contain a clique (complete subgraph) of size ≥ k ?

A proof that 3SAT is NP-complete: First, 3SAT is obvious in NP. Next, we show that SAT∝p 3SAT. Given any instance of SAT, f (x1 , . . . , xn ) = c1 ∧ · · · ∧ cm , where ci is a disjunction of literals. To construct an instance for 3SAT, we need to convert any ci to an equivalent ci0 , a conjunction of clauses with exactly 3 literals. Case 1. If ci = z1 (one literal), define yi1 and yi2 . Let ci0 = (z1 ∨ yi1 ∨ yi2 ) ∧ (z1 ∨ yi1 ∨ ¬yi2 ) ∧ (z1 ∨ ¬yi1 ∨ yi2 ) ∧ (z1 ∨ ¬yi1 ∨ ¬yi2 ). Case 2. If ci = z1 ∨ z2 (two literals), define yi1 . Let ci0 = (z1 ∨ z2 ∨ yi1 ) ∧ (z1 ∨ z2 ∨ ¬yi1 ). Case 3. If ci = z1 ∨ z2 ∨ z3 (three literals), let ci0 = ci .

Case 4. If ci = z1 ∨ z2 ∨ · · · ∨ zk (k > 3), define yi1 , yi2 , . . . , yik−3 . Let ci0 = (z1 ∨ z2 ∨ yi1 ) ∧ (¬yi1 ∨ z3 ∨ yi2 ) ∧ (¬yi2 ∨ z4 ∨ yi3 ) ∧ · · · ∧ (¬yik−3 ∨ zk−1 ∨ zk ). If ci is satisfiable, then there is a literal zl = T in ci . If l = 1, 2, let yi1 , . . . , yik−3 = F . If l = k − 1, k, let yi1 , . . . , yik−3 = T . If 3 ≤ l ≤ k − 2, let yi1 , . . . , yil−2 = T and yil−1 , . . . , yik−3 = F . So ci0 is satisfiable. If ci0 is satisfiable, assume zl = F for all l = 1, . . . , k . Then yi1 , . . . , yik−3 = T . So the last clause (¬yik−3 ∨ zk−1 ∨ zk ) = F . Therefore, ci0 is not satisfiable. Contradiction. 0 , The instance of 3SAT is therefore f 0 (x1 , . . . , xn , . . .) = c10 ∧ · · · ∧ cm 0 and f is satisfiable if and only if f is satisfiable.

Cook’s Theorem: SAT is NP-complete. Proof. SAT is clearly in NP since a NTM exists that guesses a truth assignment and verifies its correctness in polynomial time. Now we wish to prove ∀Π ∈NP, Π ∝p SAT, or equivalently, for any polynomial-time NTM M, L(M) ∝p LSAT . For any NTM M, assume Q = {q0 , q1 (accept), q2 (reject), . . . , qr } and Γ = {s0 , s1 , s2 , . . . , sv }. Also assume that the time is bounded by p(n), where n is the length of the input. We wish to prove that there is a function fM : Σ∗ → {instances of SAT} such that ∀x ∈ Σ∗ , x ∈ L(M) iff fM (x) is satisfiable. In other words, we wish to use a Boolean expression fM (x) to describe the computation of M on x.

Variables in fM (x): — State: Q[i, k]. M is in qk after the ith step of computation (at time i). — Head: H[i, j]. Head points to tape square j at time i. — Symbol: S[i, j, l]. Tape square j contains sl at time i. (Assume the tape is one-way infinite and the leftmost square is labeled with 0.) For example, initially i = 0. Assume the configuration is q0 abba. Let s0 = B, s1 = a, and s2 = b. Therefore, we set the following Boolean variables to be true: Q[0, 0], H[0, 0], S[0, 0, 1], S[0, 1, 2], S[0, 2, 2], S[0, 3, 1] and S[0, j, 0] for j = 4, 5, . . .. A configuration defines a truth assignment, but not vice versa.

Clauses in fM (x): — At any time i, M is in exactly one state. Q[i, 0] ∨ · · · ∨ Q[i, r ] for 0 ≤ i ≤ p(n). ¬Q[i, k] ∨ ¬Q[i, k 0 ] for 0 ≤ i ≤ p(n) and 0 ≤ k < k 0 ≤ r . — At any time i, head is scanning exactly one square. H[i, 0] ∨ · · · ∨ H[i, p(n)] for 0 ≤ i ≤ p(n). ¬H[i, j] ∨ ¬H[i, j 0 ] for 0 ≤ i ≤ p(n) and 0 ≤ j < j 0 ≤ p(n). — At any time i, each square contains exactly one symbol. S[i, j, 0] ∨ · · · ∨ S[i, j, v ] for 0 ≤ i ≤ p(n) and 0 ≤ j ≤ p(n). ¬S[i, j, l] ∨ ¬S[i, j, l 0 ] for 0 ≤ i ≤ p(n), 0 ≤ j ≤ p(n) and 0 ≤ l < l0 ≤ v.

— At time 0, M is in its initial configuration. Assume x = sl1 · · · sln . Q[0, 0]. H[0, 0]. S[0, 0, l1 ], . . . , S[0, n − 1, ln ]. S[0, j, 0] for n ≤ j ≤ p(n). — By time p(n), M has entered q1 (accept)). (If M halts in less than p(n) steps, additional moves can be included in the transition function.) Q[p(n), 1].

— Configuration at time i → configuration at time i + 1. Assume δ(qk , sl ) = (qk 0 , sl 0 , D), where D = −1, 1. If the head does not point to square j, symbol on j is not changed from time i to time i + 1. H[i, j] ∨ ¬S[i, j, l] ∨ S[i + 1, j, l] for 0 ≤ i ≤ p(n), 0 ≤ j ≤ p(n), and 0 ≤ l ≤ v . If the current state is qk , the head points to square j which contains symbol sl , then changes are made accordingly. ¬H[i, j] ∨ ¬Q[i, k] ∨ ¬S[i, j, l] ∨ H[i + 1, j + D], ¬H[i, j] ∨ ¬Q[i, k] ∨ ¬S[i, j, l] ∨ Q[i + 1, k 0 ], and ¬H[i, j] ∨ ¬Q[i, k] ∨ ¬S[i, j, l] ∨ S[i + 1, j, l 0 ], for 0 ≤ i ≤ p(n), 0 ≤ j ≤ p(n), 0 ≤ k ≤ r , and 0 ≤ l ≤ v .

Let fM (x) be the conjunction of all the clauses defined above. Then x ∈ L(M) iff there is an accepting computation of M on x iff fM (x) is satisfiable. fM can be computed in polynomial time since |fM (x)| ≤ (number of clauses) ∗( number of variables)= O(p(n)2 ) ∗ O(p(n)2 ) = O(p(n)4 ). So there is a polynomial reduction from any language in NP to SAT. So SAT is NP-complete.