Supervisor:

Dr. W. B¨ ohm

Working title: Matrix Multiplication – The Clever Way To Do It Keywords:

matrix algebra, combinatorial optimization, Catalan numbers, dynamic programming

Description. The intention of this topic is to introduce you into fascinating area of combinatorial optimization and dynamic programming. Combinatorial optimization deals with objective functions whose variables are usually bound to be integer valued. Furthermore, the variables have to satisfy constraints mostly of the form of inequalities. Applications of combinatorial optimization abound in operations research. Just to mention a few: vehicle routing, scheduling, facility location planning and flows in networks. Standard methods of optimization based on differential calculus are generally not applicable. Another distinguishing feature of combinatorial optimization problems is that in most cases of practical interest the number of feasible solutions, i.e. solutions that satisfy simultaneously all constraints, is finite in number. Thus, in principle it is possible to create a list of all feasible solutions, evaluate the objective function for each one and simply select among them the best solution. However, this is usually pretty inpractable since the number of solutions is generally extremely large. Therefore special and sophisticated methods of optimization are required. A very effective method is dynamic programming. Its basic idea is to recursively split a complex problem into smaller subproblems that are easier to solve, thereby maintaining a table of intermediate results which is finally used to construct the optimal solution. Matrix chain multiplication is a very good example of a combinatorial optimization problem which can be effectively solved by dynamic programming. You certainly remember your basic mathematics course (chapter 7) where you have been taught some basic facts about matrix multiplication. As you recall, this procedure is somewhat special: the product A · B = C can be calculated only, if A and B are compatible in the sense that the number of columns of A equals the number of rows of B. The result C has as many rows as A and as many columns as B. Suppose that A is of order m × n and B is of order n × p. Therefore C will be of order m × p and the component cij in row i and column j of C is determined by multiplying row i of A by column j of B in the sense that the sum of products of components is formed, i.e.,

(A)

cij =

n X k=1 1

aik bkj .

2

Matrix multiplication is in general not commutative but it is associative, i.e., (B)

ABC = A(BC) = (AB)C

This means: if you have to multiply more than two matrices in a chain, you may put parentheses around subproducts to set computational priority, the result will be always the same regardless the way you set parentheses. Therefore from an algebra point of view you can omitt parentheses completely. But what about the amount of computational work involved? Matrix multiplication is a rather simple task as it involves only addition and multiplication of real numbers, but it is quite tedious. Did you ever count the number of operations? (A) requires n multiplications and n−1 additions. Since A has m rows and B p columns, we get for the number of operations to calculate AB: mnp number of multiplications m(n − 1)p number of additions You may expect that even on a modern computer multiplication of real numbers is more time consuming than addition (see Bonus Problem below), so what really counts is the number of multiplications. Now an interesting question comes up: does the way we set parentheses in a product of matrices influence the amount of computational work to be done, if we measure this by the number of multiplications of real numbers? Let’s look at a simple example. Suppose we have to calculate ABCD, where A is of order 50 × 20, B of order 20 × 1, C of order 1 × 10 and D has order 10 × 100. Consider the following three possibilities to put parentheses, A((BC)D),

(A(BC))D,

(AB)(CD)

You can easily check that A((BC)D) requires 120 200 multiplications (A(BC))D requires 60 200 multiplications (AB)(CD) requires 7 000 multiplications So you can see, it makes really a big difference how we put parentheses! Now, two questions arise quite naturally: • In how many different syntactically correct ways can we put parentheses in a chain product of n factors? • Which of these requires the smallest number of multiplications? Remark. Syntactically correct means that parentheses must be balanced, i.e., expressions like (AB(, are not allowed.

or

A((BC)

3

The answer to the first question is Cn , the n-th Catalan number, which is given by   2n 1 Cn = n+1 n These numbers are ubiquitious in combinatorics, computer science and applied probability (you should elaborate on this point in your thesis), and they are growing very fast as n increases: C1 C2 C3 C4

=1 =2 =5 = 14 ... C10 = 16 796 C20 = 6 564 120 420

etc.

Note that in our example we have checked only 3 out of 14 possibilities. Can you find the other eleven? The Catalan numbers are growing exponentially fast. Indeed, one can show that for large n the following approximation holds: 4n Cn ∼ 3/2 √ . n π This gives us also a clue (in a negative sense) to answer question 2. It is fairly impractical to list all possible ways of putting parentheses, calculate the number of required multiplications of each and simply select the best one. A more sophisticated approach is certainly needed. This is the point where dynamic programming comes into play! Your Job. • Give a discussion of Catalan numbers, their properties and applications. • Devise an effective algorithm for optimal matrix chain multiplication based on dynamic programming. Are there alternative approaches to solve this problem? • Implement your algorithm in a computing environment of your choice, R, matlab, etc. • Bonus Problem: in a computer numbers are internally represented in binary format. Discuss and compare the amount of computational work of adding and multiplying two numbers. Note. Your own ideas and creativity are always welcome!

4

References. [1] Denardo E. V. (2003), Dynamic Programming, Models And Applications, Dover Publishing. [2] Dasgupta S., Papadimitriou C., Vazirani U. (2008), Algorithms, McGraw Hill Comments on References. • Denardo’s book provides you with a very readable and easily accessible introduction to dynamic programming. It puts some emphasis on network problems which is not surprising since dynamic programming is very closely related to the classical problem of finding shortest or longest paths in acyclic oriented graphs. • It is chapter 6 in [2] which covers dynamic programming and matrix chain multiplication. However, in the algorithm given there, something important is missing. Find out what is missing!

The Procedere. • If you are interested in this topic please contact be so that we can have an interview. • If, after the interview, you are still interested, the topic will be assigned to you. From that time on I expect within 4-5 weeks an expos´e of a few pages giving a clear representation to the structure of your thesis and the references you want to use. • If the expos´e is ok, you may start writing your thesis. Plan to finish it within about 5-6 months. This timeline is negotiable in particular cases. • Having finished a draft of your thesis, please send it to me in pdfformat. Within 7 days you will get my feedback, comments and notes on a hardcopy of your thesis. • Within 4 weeks you should take care of my comments (or not) and finish the final version of the thesis. This will be the basis of assessment. Prerequisites. • Nothing special, the basic course in mathematics is sufficient. • Honest interest in mathematics. Assessment. The result of assessment will be: • Sehr Gut, if – the thesis is comprehensive, – the thesis is methodologically correct, – the material is presented nicely, – the presentation is original and interesting to read, – you have elaborated and included own nontrivial ideas.

5

• Gut, if – the thesis is methodologically correct, – the material is presented nicely, – the presentation is interesting to read. • Befriedigend, if – the thesis is methodologically correct, but there are some errors in the draft which have been taken care of in the final version, – the presentation is somewhat sloppy and not very original. • Gen¨ ugend, if – there are severe errors in the draft which have been taken care of in the final version, – the theses missed the major points of the topic, – the presentation is sloppy and boring. • Nicht Gen¨ ugend in all other cases. Technical Note. The manuscript may be written in english or german language, but it must be written in LATEX.