Distributive Program Parallelization Using a Suggestion Language Bryan Jacobs, Tongxin Bai, and Chen Ding {jacobs,kelsey,cding}@cs.rochester.edu The University of Rochester Computer Science Department Rochester, NY 14627 Technical Report TR-952 December 2009

Abstract Most computing users today have access to clusters of multi-core computers. To fully utilize a cluster, a programmer must combine two levels of parallelism: shared-memory parallelism within a machine and distributed memory parallelism across machines. Such programming is difficult. Either a user has to mix two programming languages in a single program and use fixed computation and data partitioning between the two, or the user has to rewrite a program from scratch. Even after careful programming, a program may still have hidden concurrency bugs. Users who are accustomed to sequential programming do not find the same level of debugging and performance analysis support especially for a distributed environment. The paper presents a suggestion-based language that enables a user to parallelize a sequential program for distributed execution by inserting hints. The hints are safe against any type of misuse and expressive enough to specify independent, pipelined, and speculative parallel execution on a cluster of multi-core computers.

The research is supported by the National Science Foundation (Contract No. CNS-0720796, CNS0834566), IBM CAS Faculty Fellowship, and a gift from Microsoft Research. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding organizations.

1

Introduction

Computer users are increasingly interested in parallel programming because they want to utilize clusters of multi-core processors, which is capable, in theory, of performance tens or hundreds of times of a single personal computer. Often they have a program that take hours or days to execute, with coarse-grained parallelism that is easy to identify. The difficulty, however, is safe parallelization. Large computation tasks may execute tens of thousands of lines of code and make extensive use of bit-level operations, unrestricted pointers, exception handling, custom memory management, and third-party libraries. Although compiler techniques are effective in exploiting loop parallelism in scientific code written in Fortran [2, 8, 21, 47], they are not a sufficient solution for C/C++ programs or programs with input-dependent behavior where both the degree and the granularity of parallelism are not guaranteed or even predictable. Manual parallel programming is becoming easier. There are ready-to-use parallel constructs in mainstream languages such as Fortran, C, C++ and Java, in threading libraries such as Microsoft .NET, Intel Thread Building Blocks, in domain-specific languages such as Google’s Map Reduce and Intel Concurrent Collection for C++. Still, writing parallel code is considerably harder than sequential programming because of non-determinism. A program may run fine in one execution but generate incorrect results or run into a deadlock in the next execution because of a different thread interleaving. It may acquire new data races when ported to a machine with a different hardware memory consistency model. In addition, important types of computations such as mesh refinement, clustering, image segmentation, and SAT approximation cannot be parallelized without speculation since conflicts are not known until the computation finishes [26, 27]. This paper extends the suggestion language BOP [16,51,52]. BOP lets a user parallelize general types of sequential programs by inserting suggestions or hints. In this paper, the BOP language is extended to have four types of hints as follows. Based on these hints, the BOP system support divides a sequential program into tasks that can run on a computer cluster and ensures the correctness of parallel execution. • Parallelism hints let a user mark possibly parallel regions (PPR ) in a program. BOP then tries to execute PPR tasks in parallel in a distributed environment using explicit message passing instead of shared memory. • Data-checking hints let a user specify private data or use value-based checking to avoid false dependences in parallel execution. • Dependence hints include speculative post-wait primitives. They express possible dependences between PPR tasks. BOP implementation uses speculative synchronization and communication to satisfy these dependences. Since all BOP tasks logically share a single address space, it is easy for them to share (and build) dynamically allocated data structures such as lists and trees. • Computation reordering hints enable BOP to increase the granularity of parallelism by allowing successive PPR tasks be joined into larger units of execution. 1

BOP is a parallelization language. Unlike parallel constructs in a parallel language, BOP primitives exist purely as hints. The output of an execution with suggestions is always identical to that of the original sequential execution without suggestions, independent of task scheduling and execution interleaving. A user never needs parallel debugging. For more effective parallelization, a user may need to restructure a program to expose parallelism and remove conflicts, but the restructuring is done on sequential code using sequential programming tools such as conventional profilers or debuggers [16]. BOP uses software speculation and guarantees sequential semantics through speculative

execution, run-time monitoring, and dependence checking. Speculation, if it fails, wastes system resources including electric power. It may even delay useful computations by causing unnecessary contention. There are inherent overheads that limit the scalability of a speculative system. However, speculation is valuable and maybe necessary for parallelizing existing sequential code and for exploiting speculative parallelism (which is unknown until computation finishes). When speaking of programming language design, C++ inventor Bjarnes Stroustrup said “legacy code is code that actually works” and “legacy code breeds more legacy code.” As a parallelization language, BOP allows legacy sequential code be parallelized without explicit parallel programming. As programming hints, it offers new types of expressive power. As a result, BOP may enable ordinary users to more easily utilize a group of shared-use computers to improve the performance of existing, sequential software and of new software that reuses an existing code base. In the rest of the report, Section 2 gives an overview of the language design. Section 3 presents the parallelism hint and the system for distributed speculation. Section 4 and Section 5 describe dependence and computation-reordering hints. The dependence hint is an improvement over our previous design [52]. In Section 4, we describe an improved design, complete semantics, and a simpler implementation. In Section 6, we use BOP suggestions to build other speculation constructs. As a non-trivial use of the language, we show a parallelization technique called time skewing in Section 7. Finally, we discuss related work and summarize.

2

Overview of the Parallelization Language

A parallelization language must not change the result of a program, so BOP hints must guarantee correctness. But this is not sufficient. We design the BOP language interface with three additional goals. • Incremental. BOP supports parallelization with partial information, either a user examining part of a program or a profiling tool analyzing a few executions. The language should allow a user or a tool to gradually discover and supply needed information. • Succinct. For parallelization, a user should not have to mark all dependences in a program or to mark every dependence individually. The language should enable the user to express only relevant dependences with a minimal number of hints.

2

• Efficient. The system support should be efficient and robust in the presence of incorrect, incomplete, or out-of-date hints. Since BOP guarantees sequential semantics, it cannot use concurrency constructs such as critical sections, monitors, atomic region, or transactions. In their place, BOP uses speculative, serial constructs. Speculation enables similar out-of-order execution as concurrency constructs, at the cost of run-time monitoring and in-order commit. BOP tries to tolerate these costs with coarse-grained parallelism. The BOP language aims for both shared-memory and distributed-memory parallelization. Usually distributed programming is more onerous—a user must specify all data sharing to implement it with message passing. As a speculation system, BOP has to monitor all data access, so it can detect data sharing automatically and remove this distinction between shared-memory and distributed-memory programming. There is no language-level difference whether a user programs for one multi-core computer or ten. The BOP language has four types of hints shown in Figure 1(a). A parallelism hint suggests a parallel task. A dependence hint suggests coordination between parallel tasks. A data-checking hint helps run-time monitoring. A computation-reordering hint allows to runtime system to create larger tasks. Next we describe these hints, using the parallelization shown in Figure 1(b,c) as a running example.

3

Parallelism Hints

A bop parallel block suggests that a lexical block is possibly parallel with code after the block. An instance of a bop parallel block is the basic unit of BOP parallelism. We call each instance a possibly parallel (PPR ) task [16]. PPR tasks may have an iterative structure if they are spawned one after another, nested if one is spawned inside another, or a mixture of both. To see a use of the parallelism hint, consider the code of a generic processing loop in Figure 1(b). Each iteration de-queues an element from a queue called inputs, processes the element, and adds the result to a queue called results. In addition, the program displays the inputs queue before the loop and results after the loop. There is not enough information for static parallelization. The size of inputs is unknown. The process function may be too complex to analyze at compile time. It may use an externally compiled library with no source code available. Still, the code may be parallelizable. This possibility is expressed in two bop parallel blocks. The first suggests the function parallelism between displaying the inputs queue and processing it. The second suggests the loop parallelism between processing different elements. PPR is similar to the future construct in Multilisp. A future expresses the parallelism between the future and its continuation [22]. Other future-style constructs include spawn in Cilk [17], future in Java, and async in X10 [10] and in the proposed C++200X standard. PPR expresses possible rather than definite parallelism [16]. Two similar constructs have been studied, safe future in Java [46] and ordered transactions in C/C++ [45].

3

suggestion type

syntax

parallelism hint bop_parallel { (and implicit //code block dependence hint) }

meaning mark a possibly parallel region (ppr) in code, may be parallelized by spawning a speculative process to execute from the end of the code block, also called spawn_n_skip (sas). each ppr has my_ppr_index, incremented for each ppr.

bop_fill(channel, var)

inform the channel to post var at the time of bop_post

bop_post(channel)

post all modified data and mark the channel ready

bop_wait(channel)

wait for channel until it is ready and retrieve its data

computation reordering hint

bop_deferrable { //code block }

construct a closure for execution later

correctness checking hints

bop_private(var)

mark variable var possibly private

bop_check(var)

mark variable var for value-based checking

explicit dependence hints

(a) BOP language has four types of hints for marking possible parallelism and dependence

queue inputs, results inputs.display while ( ! inputs.empty ) { i = inputs.dequeue t = process( i ) results.enqueue( t ) } results.display

(b) A processing loop with two queues, one holds the inputs and the other the results. The queues are displayed before and after the loop.

bop_parallel { inputs.display } first_iter_id = my_ppr_id while ( ! inputs.empty ) i = inputs.dequeue bop_parallel { t = process( i ) if ( my_ppr_id > first_iter_id ) bop_wait( my_ppr_index - 1 ) results.enqueue( t ) bop_fill( my_ppr_id, &results.tail, sizeof(void*) ) bop_fill( my_ppr_id, results.tail, sizeof(qnode) ) bop_post( my_ppr_id ) } } results.display

(c) BOP parallelization. Two bop_parallel blocks mark function and loop parallelism. Post-wait ensures correct construction of the dynamically allocated results queue.

Figure 1: The parallelization language in (a), illustrated by an example in (b,c) A future represents a fork in fork-join parallelism. Each future should be paired with a join. A join point is often explicitly coded by the user, for example, using sync in Cilk, finish in X10, get in Java and in C++200X. Because of its speculation support, BOP does not require that a programmer specify a join point. There are three benefits. The first is safety. It allows partial-knowledge parallelization when the point of a join is not completely known. The second is speculative parallelization in cases when a join point is known only after future execution or when future execution ignores some infrequent conflicts. The third is flexibility. A user can suggest a task with or without a join. To suggest the join point, a user can add an explicit dependence hint, which we describe in Section 4.2.

4

3.1

Background on Software Speculative Parallelization

A speculation system deals with three basic problems. • monitoring data accesses and detecting conflicts — recording shared data accesses by parallel tasks and checking for dependence violations • managing speculative states — buffering and maintaining multiple versions of shared data and directing data accesses to the right version • recovering from erroneous speculation — preserving a safe execution state and reexecuting the program when speculation fails A speculation task can be implemented by a thread or a process. Most thread-based methods target loop parallelism and rely on a compiler or a user to identify shared data and re-direct shared data access when needed. We use a process-based solution [7, 16, 24, 25, 53]. A process can be used to start a speculation task anywhere in the code, allowing general types of fork-join parallelism. By using the page protection mechanism, a run-time system can monitor data access without instrumenting a program. Processes are well isolated from each other’s actions. Modern OS performs copy-on-write and replicates a page on demand. In a speculation system, such replication removes false dependence conflicts and makes error recovery trivial to do. We can simply discard an erroneous speculative task by terminating the process. Process-based software speculation has been shown effective in safe parallelization of large tasks in legacy code [16, 52], library calls [16], and map-reduce code [7], in debugging and race-free implementation of multi-threaded code [7], and in adaptive control of speculation [24]. It supports continuous speculation, which starts a speculation task whenever a processor becomes available, to tolerate load imbalance caused by irregular task sizes and uneven hardware speed [53]. It supports parallel memory allocation and deallocation during speculation [7, 52]. It supports threaded speculation in loops using two techniques from software transactional memory: logging and undo buffer [40]. It also supports speculative program optimization and its use in parallel program profiling and memory-access checking [25]. Next we extend the process-based design to enable program parallelization in a distributed environment.

3.2

Distributed Speculation

The BOP run-time system has three types of processes. • A control process manages shared system information and serves as the central point of communication. There is one control task in each BOP execution. We refer to it as the controller. • A host management process manages parallel execution within a host and coordinates other hosts through the control process. There is one management process on each host. We refer to it as a (host) manager. 5

• A work process runs one or a series of PPR tasks on a single processor. Work processes are dynamically created and terminated and may perform computation speculatively. We refer to it as a worker or a risky worker if its job is speculative. The first two types are common in a distributed system such as the one used by MPI and Erlang. The unique problems in BOP design are how to support speculation including the ability to run a PPR task on an available machine, checking correctness and maintaining progress in a distributed environment. To do so, the three types of processes divide the work and coordinate with each other as follows. • The control process distributes a group of PPR instances (or jobs) to the management process of each host. • On a host, the management process starts one or more work processes based on the resources available on the host. • After finishing its assigned PPR instances, a work process reports its results for verification. • If a finished PPR instance has a conflict, the control process redistributes the PPR instance and its successors for re-execution. If all finished PPR instances are verified correct, the control process continues distributing subsequent PPR instances until the program finishes. Figure 2 shows the distributed execution of an example program on two hosts. The program has four PPR tasks separated by gaps. The controller distributes PPR tasks to hosts. A host manager forks a worker process to execute a PPR task. Inter-PPR gaps are executed by the control and every manager. To start a worker, BOP needs to create a process and initialize it to have the right starting state. There are two basic ways of creating an existing state: one is copying from the existing state, the other is re-computing it from the beginning. On the same host, copying can be easily done using Unix fork. Across hosts, we use a hybrid scheme we call skeleton re-execution. Each host manager executes all inter-PPR gaps, which is the “skeleton.” When it reaches a PPR task, the manager waits for its successful completion (by some worker), copies the data changes, and skips to continue the skeleton execution at the next inter-PPR gap. With skeleton re-execution, a manager maintains a local copy of program state and use it to start worker tasks through fork. An alternative to re-execution is to use remote checkpointing, for example, to use a system like Condor [30] to implement a remote fork. Simple checkpointing would transfer the entire program state, which is unnecessary. Incremental checkpointing may alleviate the problem. Checkpointing is a more general solution than re-execution because it handles code that cannot be re-executed. In our current prototype, we allow only CPU, memory, and basic file I/O operations, where re-execution can be supported at the application level. Checkpointing support can eliminate these restrictions.

6

ppr task t.1 gap g.1

ppr task t.2

g.2

gap g.3

(a) A program execution consists of two PPR tasks, t.1 and t.2, separated by gaps g.1, g.2, and g.3. job t.1

Manager A run g.1 spawn A.1 wait for t.1 add t.1 result run g.2 wait for t.2

Controller

job t.2

Manager B run g.1 spawn B.1 wait for t.1

Host 1

Worker A.1 run t.1 return t.1 result

add t.1 result run g.2 wait for t.2

add t.2 result run g.3

Host 2 Worker B.1 skip t.1 run g.2 run t.2 return t.2 result

add t.2 result run g.3

(b) Distributed parallel execution of t.1 on Host 1 and t.2 on Host 2. Both hosts run gaps g.1, g.2 and g.3.

Figure 2: An example execution by BOP . The input program is a series of PPR tasks separated by gaps. The controller distributes PPR tasks to hosts. A host manager forks a worker process to execute a PPR task. Inter-PPR gaps are executed by the control and every manager.

3.3

Correctness Checking and Data Update

There are two basic problems in speculation support: correctness checking and data update. Each problem has two basic solution choices. • Correctness checking can be done by centralized validation, where checking is centralized in one process (the controller), or distributed validation, where checking work is divided among hosts. • Data update can be done by eager update, where changes made by one host are copied to all other hosts, or lazy update, where changes are communicated only when they are needed by a remote host.

7

The problem of data updates is similar to the ones in software distributed shared memory (S-DSM), while the problem of correctness checking is unique to a speculative parallelization system. The above choices can be combined to produce four basic designs. • centralized validation, eager update. This design is similar to shared-memory speculation. With eager update, a worker incurs no network-related delays because all its data is available on the local host. However, the controller may become a bottleneck, since it must send all data updates to all hosts. • distributed validation, eager update. With distributed validation, each host sends its data updates to all other hosts, which avoids the bottleneck in centralized validation but still retains the benefit of eager update. However, correctness checking is repeated on all hosts (in parallel), which increases the total checking and communication cost due to speculation. • distributed validation, lazy update. Lazy update incurs less network traffic because it transfers data to a host only if it is requested by one of its workers. The reduction of traffic comes at an increase of latency. Instead of accessing data locally, a worker must wait for data to be fetched from the network, although the latency may be tolerated by creating more workers to maintain full progress when some workers wait for data. With distributed validation, the global memory state is distributed instead of centralized and replicated. • centralized validation, lazy update. This scheme maintains a centralized global state. As the sole server of data requests, the controller should inevitably become a bottleneck as the number of lazy-update workers increases. This combination is likely not a competitive option. There are hybrids among these basic combinations. For example, we may combine centralized and distributed validation by checking correctness in the controller but sending data updates from each host, in a way similar to the “migrate-on-read-miss” protocol in a distributed, cache coherent system [12]. We may also divide program data and use eager update in one set and lazy update in another.

3.4

Centralized Validation and Eager Data Update

To verify correctness, the controller and manager processes maintain a progress frontier, which separates the verified part and the speculative part of an execution. A worker is started by a manager from its progress frontier and given a target PPR instance. A worker usually executes a series of gaps and a PPR instance. We use the following symbols when describing the checking algorithm. • f : progress frontier. fg is the (global) progress frontier maintained by the controller. fi is the progress frontier known by the manager at host i. • gap(f, i), ppr(f, i): the two parts, gap and PPR , of the execution of PPR instance i started from progress frontier f . 8

• R(x), W (x), RW (x): the read, write, and read-write access maps of a gap or a PPR instance x. • P (x): the content of modified data (pages) by a gap or PPR instance x. • U (i): the effect of PPR instance i, stored as the set of pages modified in i and their content. In centralized validation, each worker sends meta-data and data to the controller, which verifies correctness of PPR instances in order. Below are the steps of correctness checking. 1. A worker executes PPR instance i from progress frontier f . It executes the inter-PPR gaps from gap f + 1 to gap i − 1 and then starts recording read and write maps separately for gap i first and PPR i next as it executes them. 2. When a worker finishes a speculative PPR execution, it sends the result, consisting of RW (gap(f, i)), RW (ppr(f, i)), and P (ppr(f, i)) to the controller. 3. The controller validates worker results in the order of increasing gap and PPR index. The result of gap(f, i) and ppr(f, i) (i > f +1) is correct if gap(f 0 , i−1) and ppr(f 0 , i− 1) have been verified correct for some f 0 and: • there has been no failure between f and i, that is, the last failure index (j) is less than f in the controller’s blacklist. • the read and write sets of gap(f, i) and ppr(f, i) do not overlap with any update set U (j) after f . In symbolic terms we require \ [RW (gap(f, i)) ∪ RW (ppr(f, i))] Σi−1 j=f +1 U (j) = φ 4. If gap(f, i), ppr(f, i) is correct, the controller advances the progress frontier fg to i, stores the update set of i, U (i) = P (ppr(f, i)), and sends to all managers the new fg and U (i); otherwise, the controller blacklists index i and re-issues the job (fg , i) and its subsequent jobs for execution. A special case is i = f + 1, when the execution is non-speculative. A worker executes only PPR i (no gap) and returns P (ppr(f, f + 1)), W (ppr(f, f + 1)) for the controller to update its progress frontier and U (f + 1). This case is used for BOP to guarantee progress and base performance, as discussed in Section 3.6. Correctness checking in BOP differs from the checking in shared-memory speculative systems. In particular, BOP does not check for gap-to-gap dependences and does not need gap updates. This difference is due to the gap execution being repeated on every host. Because of the asymmetry in gap and PPR executions, they are treated differently in data collection and access checking. The incremental procedure in the algorithm for correctness checking guarantees two transitive properties. Speculation succeeds for PPR instances from 1 to n, if and only if there is no dependence between (1) gap(j) and any predecessor ppr(i) and (2) ppr(j) and any predecessor ppr(i). 9

3.5

Correctness Checking Hints

By default BOP monitors all global and heap data. Local variables are considered task private. Compiler analysis can be used to identify whether a local variable (stack data) is entire private. If a compiler cannot statically determine all accesses, it can allocate it in the heap for monitoring. Our previous system uses compiler support to identify global variables for monitoring. BOP provides a function for a program to explicitly register global data for monitoring, which allows BOP to be used without compiler support. For correctness checking, BOP classifies monitored data into three categories—possibly private, shared, and value checked—as explained in detail in [16]. The checking is primarily dependence based. There are three types of dependences. BOP checks for flow and output dependence violations. Anti-dependence violation poses no problem because of data replication [16, 52]. In our running example in Figure 1, the input queue can displayed in a PPR task while its elements are being removed by the next PPR tasks, because the queue is replicated by the later tasks. Flow and output dependences would normally cause speculation to fail. In two cases, however, BOP permits these dependences when it can prove that they are harmless. These two cases are suggested by bop private and bop check. A variable is marked by bop private if in a PPR task, its value is assigned before it is used. Because the first access is a write, the variable does not inherit value from prior tasks. Verifying the suggestion requires capturing the first access to a variable, which can be costly if the variable is an array or a structure. For efficiency we use a compromise. We insert code at the start of the PPR to write a constant value in all variables that are marked bop private. If the suggestion is correct, the additional write adds a small extra cost but does not change the program semantics. If the suggestion is wrong, the program may not execute correctly, but the sequential version has the same error, and the error can be identified using conventional debugging tools. Under this implementation, bop private is a directive rather than a hint, unlike other BOP primitives. The hint bop check suggests that a variable holds the same value before and after a PPR task. The variable may take other values in between, but the intermediate values are not visible outside and do not affect correctness of later PPR tasks. In implementation, BOP records the value of checked variables at the start of a PPR instance and at the end

compares the starting value with the ending value. If the two versions are identical, the variable has the same effect on parallel execution as a read-only variable and does not cause any dependence violation.

3.6

Progress and Performance Guarantee

Speculative execution may be too slow or produce incorrect results. Our previous systems reexecute speculative work in a sequential non-speculative task called the understudy, which guarantees progress and base performance [16, 52]. In the distributed case, we use the controller for this purpose. At each progress frontier, the controller executes the next PPR instance non-speculatively if it has not been finished correctly by a speculative worker. When the controller executes 10

a PPR instance, it records the write set for use in correctness checking and record keeping. Except for write access monitoring, the controller execution is the same as a sequential execution. Therefore, it guarantees that BOP execution takes at most the time required for the controller to finish the entire execution.

4

Dependence Hints

4.1

Implicit Dependence Hints

In BOP , a program execution becomes a tree of tasks similar to a function invocation tree. Tasks in every tree path are sequentially spawned. Since in between two PPR s the code is executed sequentially, dependences among inter-PPR code regions are automatically enforced. Inter-PPR code is most often used to implement serial work needed to start parallel tasks. For example in Figure 1(c), the input queue is accessed outside the loop PPR so queue elements are extracted sequentially. Another example is that in loops, the loop control code should be placed outside a PPR . Hence a PPR marks not only parallelism but also dependences. It suggests serial dependence in inter-PPR code, dependence from inter-PPR code to PPR code, and the absence of dependences from PPR code to inter-PPR code. An example of a PPR to inter-PPR dependence can be found in Figure 1(b), between the computation of the result t and its entry into the queue. To express this dependence, we need explicit dependence hints as we next describe.

4.2

Explicit Dependence hints

Post-wait was created by Cytron to express dependences in a do-across loop [13, 34]. We extend post-wait to mark dependences between PPR tasks. In BOP post-wait, dependent operations communicate data through a single-use channel. Channels are created on-demand and have the initial status not posted. Communicated data is identified by variable name and, if it is an array, optionally a range. BOP post-wait has three primitives. • bop fill(channel id, start addr, num pages) adds to a local, not yet communicated channel, with the identifier channel id. A bop fill call is ignored, if the channel has been posted. Only the location of the data is recorded at this point. The same memory locations may be added by multiple bop fill operations. The contents of the pages are taken only when the channel is actually posted. • bop post(channel id) releases the locally constructed channel data for consumption and stops the channel from accepting new data by changing the status of the channel, both globally and locally, from not posted to posted. It communicates only the data that have been modified by the time of the bop post. • bop wait(channel id) stalls if the channel status is not posted. If and when the channel status changes, it receives data into the waiting task. 11

For the same channel, bop post operations are atomic and logically does not overlap with each other. Bop wait operations for the same channel may retrieve data in parallel with each other. Since bop fill is local, two PPR tasks may create two channels with the same identifier. However, for the same channel identifier, only one bop post operation may succeed. Our running example shows a use of post-wait, where PPR tasks assemble their results into a linked list. In the code in Figure 1(c), post-wait is used to coordinate parallel PPR instances to enqueue their results sequentially. The first PPR modifies the queue and posts the tail. Each subsequent PPR waits for the previous PPR ’s post, modifies the queue, and posts the new tail. Since PPR tasks have an identical address space, the sender determines the placement of communicated data in the receiver. This property is indispensable for our example, where one PPR allocates a new queue node and passes it on to the next PPR . The example shows that BOP post-wait makes it easy for parallel construction of dynamic data structures. Not only can a waiting task receive data allocated before it was started, but it can also receive data allocated in other tasks after the receiver was started.

4.3

Expressiveness and Efficiency

Post-wait is a form of explicit communication similar to send-receive, with the posting PPR as the sender and the waiting PPR as the receiver. Compared to existing communication primitives, BOP post-wait has several distinct features. Deterministic behavior In the base design, a channel accepts a single post. After a channel is posted, its status is changed and all subsequent bop post and bop fill operations are ignored. This design is to ensure deterministic behavior. If we were to allow multiple posts, it would be uncertain how many of the posts had happened at the time of a bop wait. In addition, we make bop fill a task-local operation, which precludes a channel from being filled by multiple tasks. The design has two benefits. First, a local fill is efficient since it incurs no network communication. Second, it again avoids non-determinism. If two PPR tasks could fill a channel, a single post cannot guarantee that both tasks have finished placed data in the channel. If a task depends on two predecessor tasks, the right solution is to use two channels, one for each predecessor task, and let the dependent task wait for both channels. Uncertainty in data identity and access time If we know the exact dependence relation between two tasks, that is, we know data d needed by task j is produced by task i, then post-wait can easily express the dependence. Let’s relax the requirement. Suppose we know the time of access but not the location of data, as in our example where a PPR task creates a node and passes to the next task. With BOP post-wait, a waiting task can retrieve the unknown data using a channel identifier. A related case is when we know the identity of data d and its last write in i but not its first read in j. Current BOP post-wait cannot express this dependence but can be extended to express it by allowing waiting on

12

a data address instead of a channel identifier. The flow keyword in ordered transactions expresses one type of such dependence, where there is only one write in i [45]. If we relax the requirement further and suppose the last modification time is unknown. To preclude rollback, we have to put the post operation at the end of task i. Imperfect post-wait matching Post-wait is suggestion. Problems such as useless, redundant, or mismatched calls do not break a program. In the example in Figure 1(c), although the post in the last iteration has no matching wait, it does not affect correct (parallel) execution. The nature of hints makes their use in coding simpler. Incomplete knowledge of a program can cause three types of errors in specification: under-specification, where a dependence happens but a user does not mark it; over-specification, where a user specifies a dependence that does not happen in execution; or incorrect specification, where the location of the source or the target of a dependence is specified wrong or left unspecified. The speculative implementation tolerates these errors. It ignores an error if it does not affect parallelism or efficiency; otherwise, it can report it to the user to fix an error. Selective communication Not all communication has to use post-wait. In fact, most often the bulk of data changes in BOP are communicated implicitly when PPR tasks commit. The difference between post-wait and PPR commit is similar to the difference between synchronous and asynchronous communication. Post-wait is used when parallel execution requires direct synchronization. In the example in Figure 1(c), the next PPR must acquire the tail of the queue before it can append to it (assuming the queue is implemented as a singly linked list). There are a few interesting subtleties in the example. If more than three PPR tasks are run in parallel, the third PPR receives only the new node created by the second PPR but not the first. Appending to a linked list involves modifying the last two nodes, but only one node is posted. Clearly post-wait does not communicate all queue updates to all PPR s. Still, the solution is correct because other (non-essential) pieces will be combined at commit time. During parallel execution, each PPR works on one link of the queue. After it, BOP automatically merges the results and obtains the complete queue, one identical to the result obtained from a sequential execution. Aggregate communication Post-wait lets a user to aggregate communication. In the example, a PPR must communicate two pieces of information to the next PPR : the new tail pointer, and the new tail node. It adds them in two bop fill calls and communicates them in one post. Aggregation may also happen due to page-based data protection. If both data are on the same page, the second bop fill call is ignored. On the other hand, false sharing may cause unexpected conflict, for example, if the PPR modifies a queue node next to the tail after the post. Such cases will cause parallel execution to roll back. To enable full parallelism, a user has to either communicate at the end of PPR (as in this example) or allocate dependent data on separate memory pages. Progress frontier A posted channel may be consumed by any number of wait operations. Logically these wait operations all receive the same data. Communication may not

13

be needed if the waiting task already has posted data from earlier commits. If a user accidentally inserts a post without a matching wait or vice versa, the unmatched operations are eventually dropped when the progress frontier moves past. The system stores the identifier of all channels that have been posted in case a wait operation is later called on one of these channels. The storage cost is small since the system stores only channel identifiers not channel data. Extension to k-post Suppose in a more general case, we know a set of u possible program points that may be the source of a dependence and a set of v possible program points that may be the sink of the dependence. We would need to synchronize between the last executed point in u possible sources and the first executed point in v possible sinks. First, if some of the u possible sources happen in a single PPR , we find the latest point (the one that post-dominates the others in either program or profiling analysis) and reduce the possible sources to a set of up points in up PPR s. Similarly we reduce the set of possible sinks to vp points in vp PPR s. With single-post design, we need to insert at each source a post operation with a different channel and insert at each sink up wait operations so it waits for all channels, for a total of up post operations and up vp wait operations. We can extend the design to assign a parameter to a channel as the num posts, which means that the channel is posted after it has received exactly num posts. In this example, we create a single channel with num posts equals to u, insert a post operation at each source, and a wait operation at each sink. The total number of post-wait operations is u + v.

4.4

Implementation of Speculative Post-wait

Incorrect use of BOP post and wait may cause a task to consume a stale value of some data. BOP detects all stale-value accesses as conflicts. There are three types of conflicts. First, a sender conflict happens if a PPR instance modifies some data that it has posted earlier. This conflict causes a transmission of stale data. Second, a receiver conflict happens when a PPR instance accesses some data and then later receives the data through a bop wait. This conflict causes a consumption of stale data. Finally, a pairing conflict happens when some data is received not from the last modifying PPR instance but from an earlier one. In the first two types, the error is caused by one of the two ends of the communication. In the third, the error is caused by the actions of a third party. BOP detects and handles these conflicts as follows. Sender conflict The simple solution is that if PPR i incurs a sender conflict, we abort and re-start all speculation of PPR j for j > i. With sufficient bookkeeping, we can identify and rollback only those PPR instances that are affected by the offending send. The sender can remove the data from the earlier posted channels and sends the newest value in the latest post. To avoid recurring conflicts, the BOP runtime can delay any post operation that has caused a sender conflict.

14

task 1 write x fill(1, x)

task 2

wait(1)

task 3 wait(1) read x

post(1)

write x

Figure 3: Illustration of a pairing conflict when using post-wait. The read in task 3 is incorrect because the received value is not the newest one in sequential semantics. A pairing conflict. Task k is aborted because it

Receiver conflict and multiple receives uses a stale version of A dataPPR x. instance must be aborted if it incurs a receiver conflict. A related case is when a PPR receives x more than once from different senders. We rank the recentness of received values by the PPR index of their sender. A larger index means a later sender and a newer value. There are three cases. First, x is first accessed before the first receive, which means a receiver conflict. Second, x is first accessed between two receives. If the value from the first receive is less recent than that of the second receive, a receiver conflict is triggered. If the value from the first receive is newer, the second receive can be silently dropped without raising a conflict. In the third case, x is first accessed after two or more receives. BOP keeps the newest value of x and continues. We call the last two extensions silent drop and reordered receive. Pairing conflict A pairing conflict is caused when some data x is communicated incorrectly. The 3-task example in Figure 3 shows such a conflict. Data x is modified in tasks 1 and 2 and read in task 3. The value of x sent by task 1 is the newest for task 2 but not so for task 3 since it can get a newer value from task 2. In terms of data dependences, the dependence from task 2 to 3 is violated. To ensure correctness, we need to combine dependence and post-wait checking. A pairing conflict is detected at the time of commit, when all post-wait and data-access information is known and stored in access maps. An access map marks four types of data access: r, wr, p(i), wa(i), representing read, written, posted, and received (through a wait). The last two types include the PPR index of the sender. Let ∆i be the access map of PPR i. If PPR i posted data x, we add p(i) to ∆i (x) (and remove wr since the write is implied). If PPR i received data x from PPR s, we add wa(s) to ∆i (x). When checking ppr(f, i), we compare its access with each predecessor PPR from f + 1 to i − 1, as in the algorithm in Section 3.4. To handle post-wait, we augment the Step 3 of the algorithm as follows. For ease of checking, it builds a cumulative write map Φ representing the set of write accesses before PPR i. There are four cases, with different correctness conditions for i and updates to the cumulative map Φ. 1. x is not accessed in PPR i. If ∆i (x) = φ, the check passes; otherwise, continue to the next case. 2. x is not modified until PPR i. If Φ(x) = φ, set Φ(x) = ∆i (x) − r (Φ is a write map), and the check passes; otherwise, continue to one of the next two cases.

15

3. x is modified before i but not posted. If wr ∈ Φ(x) and ∆i (x) 6= φ, the check fails. 4. x is modified before i and posted. There are two cases depending on whether i received x from this sender s. If not, p(s) ∈ Φ(x) but wa(s) ∈ / ∆i (x), and the check fails. If so, p(s) ∈ Φ(x) and wa(s) ∈ ∆i (x), and the check passes but we need to set Φ(x) carefully. If x is only read in i, leave Φ(x) unchanged. If x is modified and posted in i, set Φ(x) = {p(i)}. If x is modified but not posted in i, set Φ(x) = {wr}. As an example, consider the steps of checking task 3 in the earlier example (Figure 3). At the end of task 1 and 2, Case 2 and 4 are invoked respectively, and the cumulative map Φ(x) becomes first p(1) and then w. Finally, the check at task 3 invokes Case 3 and detects a conflict. This solution differs from the previous solution, which was based on the concept of communication paths [52]. Inter-PPR post-wait handling The skeleton execution described in Section 3.2 permits more precise post-wait checking than the previous solution for shared-memory speculation [52] in that it can ignore post-wait constructs in inter-PPR code. A post in gap i is unnecessary since its data changes are visible to all subsequent execution. A wait in gap i is unnecessary either, if the matching post comes from another gap, or implies a dependence violation, if the matching post comes from a PPR . Therefore, bop fill and bop wait are ignored, and bop post sends its channel without any content. Outdated post As mentioned earlier, data updates in BOP are propagated in two ways, post-wait and PPR commit. If a post-wait pair spans many PPR instances, it is possible that the receiver is started after the commit of the sender so the receiver already possesses the sent data. The frontier parameter can be used to check this case precisely. If ppr(f, j) receives a post from PPR i that happens before f (i ≤ f ), bop wait succeeds but it will nullify the content of the channel and accept no new data. We call the special wait a nullified receive. Progress guarantee Usually blocking communication implies the risk of entangling a set of tasks into a deadlock. BOP post-wait is deadlock free because communication flows in one direction in the increasing order of PPR index. A wait without a matching, earlier post would stall the receiving task. However, it does not stall the advance of the progress frontier as explained in Section 3.6. When the progress frontier moves past, the stalling task is canceled.

5

Computation Reordering and Run-time Task Coarsening

At run-time, BOP may group a series of PPR instances to create a large parallel task. We call the transformation run-time task coarsening and the aggregate task a BOP task. By choosing different group sizes, we can create parallel tasks of a desirable length, which is especially useful in distributive parallelization. 16

for i in 1 ... n bop_parallel { t = data[ i ].analyze bop_deferrable ( t ) { if i > 1 bop_wait(i - 1) histogram.add( t ) bop_fill(...) bop_post( i ) } } end for

for i in 1 ... n bop_parallel { // call to a large function t = data[ i ].analyze if i > 1 bop_wait(i - 1) histogram.add( t ) bop_fill(...) bop_post( i ) } end for bop task 1 // ppr 1 data[ 1 ].analyze histogram.add bop_post( 1 ) // ppr 2 data[ 2 ].analyze bop_wait( 1 ) histogram.add bop_post( 2 )

bop task 2 // ppr 3 data[ 3 ].analyze bop_wait( 2 ) histogram.add bop_post( 3 ) // ppr 4 data[ 4 ].analyze bop_wait( 3 ) histogram.add bop_post( 4 )

(a) An example loop (upper graph) and parallel execution of the first 4 pprs in 2 tasks. Limited parallelism due to postwait.

bop task 1 // ppr 1 data[ 1 ].analyze // ppr 2 data[ 2 ].analyze // delayed blocks histogram.add bop_post( 1 ) bop_wait( 1 ) histogram.add bop_post( 2 )

bop task 2 // ppr 3 data[ 3 ].analyze // ppr 4 data[ 4 ].analyze // delayed blocks bop_wait( 2 ) histogram.add bop_post( 3 ) bop_wait( 3 ) histogram.add bop_post( 4 )

(b) An example deferrable block (upper graph) and parallel execution of the first 4 pprs in 2 tasks. More parallelism due to delayed post-wait.

Figure 4: An example showing the use of a deferrable block in exposing parallelism between “coarsened” BOP tasks that contain multiple PPR instances. Computation reordering is necessary to enable this type of automatic task coarsening. Consider PPR blocking for the example loop in Figure 4(a). Each iteration produces a result and adds it to a histogram. The suggested parallelization is to run each iteration in parallel but serialize its update to the histogram using post-wait. An example PPR grouping is shown below the loop. There are two BOP tasks each executing 2 PPR s. Because of postwait, PPR 3 must wait for PPR 2 (shown by the arrow in the figure), and BOP task 2 must stop after the first PPR to wait for task 1 to finish. Post-wait is necessary to guarantee sequential semantics but it causes PPR grouping to lose most parallelism. The purpose of computation reordering is to regain parallelism while preserving sequential semantics. The computation-reordering hint, bop deferrable, suggests that a block of code can be executed later. Figure 4(b) shows its use in our example code, suggesting that the histogram update can be delayed. The two BOP tasks, shown in the figure below the code, can now run in parallel. The histogram updates are still sequentially done, but they are delayed to the end of BOP tasks and do not stop task 2 in the middle of the execution. With delayed updates, PPR blocking can increase the amount of synchronization-free, parallel work by the group size. By selecting group size at run time, BOP can control the granularity 17

of parallelism and de-couple program specification from machine implementation: a user focuses on identifying parallelism, and BOP tailors the parallelism for a machine. As a form of dependence hint, deferrable block has an identical meaning as a PPR block. Both mean an absence of dependence between the code block and its succeeding computation. The use of the two types of blocks is so different that we give them different names in BOP . The implementation issues, however, are similar in many aspects especially in correctness checking. The main difference is interaction with the succeeding block. A PPR block is in a separate process from its succeeding block, but a deferrable block runs in the same process as its succeeding block. Instead of creating a process, a deferrable block creates a closure to include the code and all its variable bindings. The closure is invoked as late as possible in a BOP task. Most parallel languages provides an atomic-section construct for use by a parallel loop. The loop may then be partitioned and executed in blocks of iterations. However, atomic sections do not guarantee correctness (i.e. same result as sequential execution) or deterministic output. Computation-reordering hint in BOP aims to provide the same flexibility while guaranteeing sequential semantics.

6

Language Extensions

BOP hints are basic building blocks that can be combined to build other constructs. As

a result, we can add new suggestions without changing language implementation. We demonstrate the potential for language extensions by building two loop-based and one data-based suggestions. When parallelizing a loop, a direct way is to mark a parallel loop. Consider the example in Figure 5(a). Each iteration has two steps: computing a value and appending it to an object. Assume that the first step is parallel, and the second step has to be serial. To parallelize the loop, we mark the loop parallel and place the second step in a serial region. A second construct is a shared variable declaration as used by Jade [37]. A compiler is used to identify and manage shared data accesses. A third loop-based construct is the pipelined loop designed by Thies et al [42]. To parallelize the example, a user adds two pipeline stages and marks the first stage parallel (by specifying the number of processors p), as in Figure 5(c). Serial region One can declare a parallel loop and mark dependent operations in a serial region, as in Figure 5(e). The term “serial region” was used by an early version of the OpenMP language to mean code that should be executed by one thread. We generalize the concept to add a label at each region. Code of the same named regions must be serialized in the sequential order, while code from different named regions may execute in parallel. The generalized serial region can be implemented by BOP post-wait. Figure 5(f) shows the code translation for a generic serial region r1. The scheme is akin to token passing, where a “token” is a channel number indexed by the identity of the host task, and it is passed by post-wait. To determine data communication, we may send all data changed in the region or find ways to identify and send only dependent data. Since BOP serial region is a hint, it 18

parallel for i in 1 ... n r = compute( i ) serial r1 { s.append( r ) } end for

for i in 1 ... n r = compute( i ) s.append( r ) end for

(a) a loop whose body consists of a parallel step and a sequential step

parallel for i in 1 ... n // parallel work serial r1 { // sequential work } // parallel work end for

(e) general form of serial region

shared s parallel for i in 1 ... n r = compute( i ) s.append( r ) end for

(b) parallelization of (a) using a parallel loop and a serial region

(c) parallelization using a parallel loop and shared variable declaration

bop_post( r1_ids[0] ) for i in 1 ... n bop_parallel { // parallel work bop_wait( r1_ids[my_ppr -1] ) // sequential work bop_post( r1_ids[my_ppr] ) // parallel work } end for

(f) general implementation of serial region using BOP

for i in 1 ... n begin_pipelined_loop pipeline( p ) r = compute( i ) pipeline s.append( r ) end_pipelined_loop end for

(d) parallelization using a pipelined loop

for i in 1 ... n begin_pipelined_loop // stage 1 pipeline( p ) // stage 2 pipeline // stage 3 end_pipelined_loop end for //pipeline(p) means that next stage is parallel

for i in 1 ... n bop_parallel { bop_wait( ) // stage 1 bop_post( ) bop_wait( ) // stage 2 bop_post( ) bop_wait( ) // stage 3 bop_post( ) } end for

(g) general form of pipelined loop

(h) implementation by BOP

Figure 5: Using BOP as the base language to implement composite constructs similar to the serial region and pipelined-loop is possible in an execution that an iteration executes a serial region multiple times or none at all. Shared variable Shared variable declaration enables a compiler to identify shared data accesses [37]. Such a compiler can generate parallel code using BOP . For each PPR , the compiler identifies the earliest and latest accesses of shared data and places the wait and post operations as in the implementation of the serial region. If a PPR has no shared data accesses, it is equivalent to having the first and last accesses at the start of PPR . If the accesses in a PPR are not completely known, the conservative solution is assuming that the first access is at the beginning and the last access is at the end. To improve parallelism, the compiler may use a different serial region for each shared variable. Since the correctness is guaranteed by BOP run time, a compiler can generate more aggressive, albeit unsafe code. Pipelined loop With a pipelined loop, most of the work done by a user is just to divide a loop body into pipeline stages [42]. By default a stage is sequential in that its execution in different iterations does not overlap. If the work can run in parallel, a user marks the stage parallel by specifying the number of processors p. Figure 5(g) shows a 3-stage example, where stage 1 and 3 are sequential but stage 2 is parallel. Although simple, pipelined-loop shows clearly the composition of cross-stage and intra-stage parallelism. Its implementation is also simple and effective, with one process running a sequential stage and p processes running a parallel stage as specified. All processes are fully active in the steady state. 19

The same interface can be implemented by BOP hints using the following code transformation. First, we turn a pipelined loop body into a PPR . Then we create a series of channel numbers indexed by the PPR index and the stage index. For each sequential stage, we insert at the front a wait for the same stage from the preceding PPR and at the back a post for itself (this stage and this PPR ). The conversion of a parallel stage is the same except that the wait is for the preceding stage in the preceding PPR . Figure 5(h) shows this transformation. The BOP code exploits the same, staged parallelism but the implementation is very different. One process is used per iteration not per stage. To fully utilize p processors, BOP may need to execute more than p processes in parallel and rely on the multi-tasking support of the operating system. Potential benefits As the three examples show, BOP can serve as a basis for building other parallel constructs. Compared to the original constructs, BOP -based emulators benefit from uniformity and correctness guarantees of BOP . Take pipelined loop as an example. The original construct does not guarantee the same result in parallel executions, but the BOP version adds this guarantee. The original primitive cannot be used within a nested loop or called function. A target loop cannot have break and continue statements (need to first convert them to if statements). The BOP version removes these restrictions. The original implementation copies all data changes synchronously at the end of a pipelined loop. BOP copies data changes incrementally and asynchronously in parallel with loop or post-loop execution. In the original design, a user must remove all conflicts. With BOP , conflicts do not affect correctness, and occasional unknown conflicts may not affect parallel performance if most of the execution is conflict free. Another benefit is clarity—the meaning of an extension is completely defined by its construction. This removes ambiguity in semantics especially when a construct is used amid complex control flow. Take the example of the serial region. The implementation in Figure 5(f) assumes that a serial region is executed once and only once in each PPR . The assumption holds for the implementation of pipelined-loop in Figure 5(h). However, in a general case, one cannot guarantee how often a serial region is executed or whether it is executed at all. We call it the unknown occurrence problem. It includes the multi-occurrence problem, which may cause out of order region execution, and the missing-occurrence problem, which may cause indefinite waiting. To solve the multi-occurrence problem, we should move the post operation to the end of PPR . To solve the missing-occurrence problem, we add a post operation at the end of PPR . In all these cases, the behavior and properties of a serial region are defined according to how we choose to implement it with BOP primitives. There are implementation limitations to BOP -based extensions. BOP may not be the most efficient solution for example for pipelined loop. However, the previous solutions were not designed for software speculative execution. BOP implementation may still be the right choice in a speculative context. In addition, by building different constructs from a single basis, a user can combine and mix them, for example post-wait and pipelined loop, when parallelizing a single program.

20

7

A Demonstration

We demonstrate a use of BOP in parallelizing iterative computations. An iterative algorithm is commonly used in solving equations that have no direct solutions. Examples can be found in many types of numerical analysis and in general computational problems that require there is node to be clusteredfixed-point and equilibrium solutions. Figure 6 shows a typical program structure with a 2-nested loop. The inner loop computes on a data array, and the outer loop repeats this = new ... // init next cluster choice computation until the result is good enough. Informally the loop can be said to have two B) { // b-loop to find largest cluster dimensions with the inner loop being the space dimension and the outer loop being the st; // init block choice time dimension. The inner loop is usually parallel, but the outer loop is not because the i ++) { // i-loop to find largest in block same data is used and updated in every time step. In addition, the convergence check, continue; in line C in Figure 6, must be done after one time step and before the next. Without e; support, a parallel version must use a barrier to ensure that the next iteration ); // cluster centered at nodespeculation i +) { wait for the current iteration to finish completely. With BOP , however, the next iteration et_dis(nodes[ i ], nodes[ j ]); can start before the current one finishes. At the end when the converged variable is set, !eliminated[ j ] ) s[ current.size++ ] = j; the write access causes a dependence conflict, BOP detects the conflict and aborts the speculation beyond the convergence point. Before the convergence, different time iterations e( &block_largest, ¤t ) may overlap. The transformation is known as time skewing [39, 48]. current // largest cluster in this block intra-block search

while ( not converged ) for i is 1 to N bop_parallel { r = compute( data[ i ] ) bop_deferrable { if ( i > 1 ) bop_wait( my_ppr - 1 ); s = add_result( r ) bop_fill( my_ppr, cur ) B: bop_post( my_ppr ) } // end bop_deferrable } // end bop_parallel } // end of i-loop

- 1);

in all blocks global_largest, &block_largest ) = block_largest

while ( not converged ) for i is 1 to N

ppr, global_largest);

newly found cluster al_largest.size; nn++) { _largest.nodes[ nn ] ] = 1

and exit if clustering is done erged = true

r = compute( data[ i ] ) s = add_result( r ) } // end of i-loop if good_enough?( s ) C: converged = true

if good_enough?( s ) C: converged = true }

}

Figure 6: Example iterative computation and its parallelization by BOP . There is no barrier at the end of the while-loop, which permits speculative parallelism. The parallelism from time-skewing is data dependent. Time skewing is likely successful if the beginning part of the next iteration accesses different data from the ending part of the current iteration. The parallelism may be highly dynamic, and BOP execution may have frequent rollback and restart. The parallelism does not extend beyond two outermost loop iterations since consecutive iterations as a whole always have at least one conflict. However, the program does not have to wait for every task to finish after one iteration and before starting another. The removal of the barrier synchronization makes it possible that the waiting by one task can always be overlapped with the execution of subsequent tasks. This is useful in hiding the overhead of serial correctness checking in BOP . It is also useful in tolerating imbalances in task sizes and variations due to the processor or system environment. 21

QT clustering An example of iterative computation is quality threshold (QT) clustering. The algorithm was invented in 1999 for gene clustering [23]. In each iteration, the algorithm finds the cluster around every node, which contains all nodes that are within a distance. It then picks the largest cluster. The next iteration repeats the process for the remaining nodes until all nodes are clustered. QT clustering is computationally intensive, but it addresses two shortcomings of traditional k-means clustering. To use QT clustering, one does not have to know the number of clusters beforehand. The result is deterministic and does not depend on initial choices of cluster centroids. To parallelize QT clustering, we need speculative post-wait to communicate dynamic data. In each time step, all clusters are compared to find the largest cluster. This is done by a post-wait region that passes the cluster object global largest from task to task in sequential order. Since the size of a cluster is not known, we allocate an array with size n, the total number of nodes. While the code posts the entire array, BOP run-time system posts only modified pages, so it communicates only part of the array with a size proportional to the size of the cluster. We coded the basic algorithm, where stack variables were private and heap and global data were shared (unless declared bop private). Since we have not implemented deferrable block, we manually strip-mined the clustering loop. We inserted three lines of hints, one for parallelism and two for a post-wait pair. To enable time skewing, we have structured the code carefully so to avoid unnecessary dependence conflicts. We found that the difficulty for time skewing is mostly algorithmic. In implementation, the base code is sequential, and BOP preserves its semantics. BOP parallelization did not go much beyond sequential programming and sequential algorithm adaptation. We could use existing tools such as compilers, debuggers, profilers. We tested our prototype implementation using two machines. One had 8 Intel Nehalem cores, and the other 4 Intel 2.66 GHz Kentsfield cores, all running at 2.66 GHz. We used a large input, so the sequential running time, with no BOP involvement, was 70 seconds. With BOP , the program finished in 7 seconds, obtaining a speedup of 10. This showed clearly the benefit of using two machines, since the single-machine improvement would be bounded by 8.

8

Related Work

Distributive parallelization Parallelism may be inferred by a parallelizing compiler. Powerful compiler techniques such as dependence, array-section, alias and inter-procedural analysis can extract coarse-grained parallelism in scientific code [1, 2, 8, 18, 19, 21, 47]. Automatic parallelization can be improved by language and tool support. High performance Fortran (HPF) lets a user suggest data distribution in Fortran code [1, 2]. Cleanly designed languages can better express and implement parallelism through high-level primitives and advanced techniques such as regions and lazy evaluation as in ZPL [9] and SoC [38]. A user can help compiler parallelization using an interactive tool [29]. Like HPF, BOP provides hints for use in a traditional language. While HPF hints relies on the compiler for parallelization, BOP hints express parallelism directly.

22

A recent approach to distributed parallelization is to automatically translate a Fortran program from using OpenMP to using MPI [4–6]. Parallelization and communication are inserted and optimized by the compiler, which uses static analysis for regular loops and inserts run-time inspectors in irregular loops. With distributed OpenMP, a user can program distributed-memory machines using a shared-memory parallel language. BOP strives for the same goal using a suggestion language in which dependences are specified using post-wait rather than automatically inferred. Post-wait can be used by a compiler or a profiling-based analyzer, so BOP does not have to rely on complete compiler support for correctness. As a result, BOP can handle general C/C++ code. Most automatic techniques target regular computations on arrays and cannot directly handle irregular computations on dynamic data. One exception is Jade, which uses a combination of user annotations and type checking to infer program data accesses and enable dynamic parallelization [37]. Data specification in Jade and dependence hints in BOP represent very different design choices. Data access is the cause of dependences, and data specification in Jade can be used by a run-time system to identify all dependences. BOP allows partial dependence specification, simply the specification of frequent and immediate dependences necessary to enable parallelization. The BOP run-time detects infrequent dependences and satisfies long-range dependences automatically without any specification. Jade aims for complete safety and automatically optimized parallelization. BOP tolerates incomplete program knowledge and supports more direct control in parallelization. Parallel programming languages Parallel languages are based on either single-program multiple data (SPMD) [15] or fork-join parallelism. The SPMD model is used in distributedmemory programming as by MPI, PVM, software shared distributed memory (S-DSM) systems such as TreadMarks [3] and InterWeave [41], and PGAS languages such as UPC [44] and co-array Fortran [32]. Data by default is not shared. Shared data can be declared as shared pages in SDSM and global arrays in PGAS languages and X10 [10]. SPMD programming requires manual partitioning of computation and placement of data. Although such requirements may be necessary for scalable parallel computing, it makes SPMD programming considerably more difficult than sequential programming. Fork-join is used in shared-memory programming as by OpenMP, Cilk, Java, transactional memory, and many other languages in which parallel programming is done by annotating a sequential program. These languages specify parallel tasks using parallel loops, regions, or future-style primitives such as spawn in Cilk [17] and async in X10 [10]. Synchronization is done using concurrency constructs such as atomic sections and transactions. Concurrency constructs are more flexible because they do not restrict a program to sequential semantics. However, the flexibility is also a burden since a programmer must ensure correctness under all possible program interleavings. They cannot be used by profiling-based automatic parallelization. For safety, they requires a user to have complete knowledge of all program dependences that may impair parallel execution. BOP overcomes these shortcomings with the dependence-based construct of post-wait and with run-time speculation support. It ensures sequential semantics at the costs of requiring monitoring and in-order commit. It tolerates these costs by using an asynchronous design and by enabling more aggressive forms of parallelism. 23

Software speculative parallelization Loop-level software speculation was pioneered by the LRPD test for Fortran loops [36]. Many improvements have been developed including array replication and stronger progress guarantees [14, 20]. The support has been extended to C programs using annotations [11], compiler support [43, 50], and techniques developed for transactional memory [31, 33, 40]. Speculation support has been developed for Java to support safe future [46], return-value speculation [35], and speculative data abstractions [28]. Our implementation uses ideas from [16, 24, 52], which is based on processes instead of threads. Process-based systems use the virtual memory protection mechanism, which transparently provides strong isolation, on-demand data replication, and complete abortability. A recent system, Grace, removes concurrency problems such as race conditions and deadlocks by implementing threads using processes and employing similar monitoring and protection mechanisms [7]. Previous speculation systems have a limited interface for expressing dependences. They run on shared-memory systems. We aim to develop a more general programming interface. For example in a software implementation, page-level protection raises the problem of false sharing. Dependence hints such as post-wait can be used to tolerate false sharing by serializing access to falsely shared pages. It is especially useful for dense array data. This report describes support for distributed speculation, task coarsening, and various language extensions such as serial region and pipelined loop. Dependence hints Post and wait were coined by Cytron for do-across loop parallelism [13]. They are paired by an identifier, which is usually a data address. BOP post-wait specifies data for communication. By integrating it with speculation support, we have made it a hint rather than a directive and used it to construct other speculative parallel constructs such as sequential regions and pipelined loops. In the context of speculative parallelization, a signal-wait pair can be used for inter-thread synchronization [49]. Signal-wait did not take parameters, and a compiler was used to determine the data transfer and must do so correctly. In an ordered transaction, the keyword flow specifies that a variable read should wait until either a new value is produced by the previous transaction or all previous transactions finish [45]. Flow is equivalent to wait with an implicit post (for the first write in the previous task). Ordered transaction is implemented using hardware speculation support. It is unclear how it copes with errors such as read before flow. It cannot easily be used to communicate dynamically allocated data as in our example in Section 2.

9

Summary

We have presented the design of the BOP language consisting of four types of hints: parallelism, explicit dependence, computation reordering, and correctness checking. It provides a sequential programming model and a hybrid solution to data sharing, where a user marks likely parallel tasks and identifies their immediate data sharing using dependence hints and leave remaining data sharing unspecified. BOP system implements distributed communication with a mix of user-directed synchronous updates and automatic asynchronous updates. We have demonstrated the expressiveness of the suggestion language in two ways. First, 24

we show that they can be used to implement other language constructs such as serial regions, shared variables, and pipelined loops. Second, we show that dependence hints enable time skewing, which removes the need of barrier communication in many types of iterative solvers. Until recently a user faced with the task of extracting coarse-grained parallelism in complex code had to choose either a compiler to automate the task or a language extension to parallelize a program completely by hand. BOP provides a middle ground where the user manually specifies high-level parallelism and known dependences, and the run-time system automates distributed parallelization and communication.

Acknowledgments We wish to thank Chao Zhang and Xiaoming Gu for their help with the implementation and testing and Michael Scott, Xipeng Shen, and Kai Shen for the helpful discussions and for pointing out related work and possible applications.

References [1] V. S. Adve and J. M. Mellor-Crummey. Using integer sets for data-parallel program analysis and optimization. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 186–198, 1998. [2] R. Allen and K. Kennedy. Optimizing Compilers for Modern Architectures: A Dependence-based Approach. Morgan Kaufmann Publishers, Oct. 2001. [3] C. Amza, A. L. Cox, S. Dwarkadas, P. J. Keleher, H. Lu, R. Rajamony, W. Yu, and W. Zwaenepoel. ThreadMarks: Shared memory computing on networks of workstations. IEEE Computer, 29(2):18–28, 1996. [4] A. Basumallik and R. Eigenmann. Towards automatic translation of OpenMP to MPI. In International Conference on Supercomputing, pages 189–198, 2005. [5] A. Basumallik and R. Eigenmann. Optimizing irregular shared-memory applications for distributed-memory systems. In Proceedings of the ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 119–128, 2006. [6] A. Basumallik, S.-J. Min, and R. Eigenmann. Programming distributed memory sytems using OpenMP. In Proceedings of the International Parallel and Distributed Processing Symposium, pages 1–8, 2007. [7] E. D. Berger, T. Yang, T. Liu, and G. Novark. Grace: Safe multithreaded programming for C/C++. In Proceedings of OOPSLA, 2009. [8] W. Blume et al. Parallel programming with polaris. IEEE Computer, 29(12):77–81, 1996.

25

[9] B. Chamberlain, S.-E. Choi, C. Lewis, C. Lin, L. Snyder, and W. Weathersby. ZPL: a machine independent programming language for parallel computers. IEEE Transactions on Software Engineering, 26(3):197–211, Mar 2000. [10] P. Charles, C. Grothoff, V. Saraswat, C. Donawa, A. Kielstra, K. Ebcioglu, C. von Praun, and V. Sarkar. X10: an object-oriented approach to non-uniform cluster computing. In Proceedings of OOPSLA, pages 519–538, 2005. [11] M. H. Cintra and D. R. Llanos. Design space exploration of a software speculative parallelization scheme. IEEE Transactions on Parallel and Distributed Systems, 16(6):562– 576, 2005. [12] A. L. Cox and R. J. Fowler. Adaptive cache coherency for detecting migratory shared data. In Proceedings of the International Symposium on Computer Architecture, pages 98–108, 1993. [13] R. Cytron. Doacross: Beyond vectorization for multiprocessors. In Proceedings of the 1986 International Conference on Parallel Processing, pages 836–844, St. Charles, IL, Aug. 1986. [14] F. Dang, H. Yu, and L. Rauchwerger. The R-LRPD test: Speculative parallelization of partially parallel loops. In Proceedings of the International Parallel and Distributed Processing Symposium, pages 20–29, Ft. Lauderdale, FL, Apr. 2002. [15] F. Darema, D. A. George, V. A. Norton, and G. F. Pfister. A single-program-multipledata computational model for epex/fortran. Parallel Computing, 7(1):11–24, 1988. [16] C. Ding, X. Shen, K. Kelsey, C. Tice, R. Huang, and C. Zhang. Software behavior oriented parallelization. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 223–234, 2007. [17] M. Frigo, C. E. Leiserson, and K. H. Randall. The implementation of the cilk-5 multithreaded language. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 212–223, 1998. [18] J. Gu and Z. Li. Efficient interprocedural array data-flow analysis for automatic program parallelization. IEEE Transactions on Software Engineering, 26(3):244–261, 2000. [19] M. Gupta and P. Banerjee. PARADIGM: A compiler for automatic data distribution on multicomputers. In International Conference on Supercomputing, pages 87–96, 1993. [20] M. Gupta and R. Nim. Techniques for run-time parallelization of loops. In Proceedings of SC’98, page 12, November 1998. [21] M. W. Hall, S. P. Amarasinghe, B. R. Murphy, S.-W. Liao, and M. S. Lam. Interprocedural parallelization analysis in SUIF. ACM Transactions on Programming Languages and Systems, 27(4):662–731, 2005. [22] R. H. Halstead. Multilisp: A language for concurrent symbolic computation. ACM Transactions on Programming Languages and Systems, 7(4):501–538, Oct. 1985. 26

[23] L. Heyer, S. Kruglyak, and S. Yooseph. Exploring expression data: Identification and analysis of coexpressed genes, 1999. [24] Y. Jiang and X. Shen. Adaptive software speculation for enhancing the cost-efficiency of behavior-oriented parallelization. In Proceedings of the International Conference on Parallel Processing, pages 270–278, 2008. [25] K. Kelsey, T. Bai, and C. Ding. Fast track: a software system for speculative optimization. In Proceedings of the International Symposium on Code Generation and Optimization, pages 157–168, 2009. [26] M. Kulkarni, K. Pingali, G. Ramanarayanan, B. Walter, K. Bala, and L. P. Chew. Optimistic parallelism benefits from data partitioning. In Proceedings of the International Conference on Architectual Support for Programming Languages and Operating Systems, pages 233–243, 2008. [27] M. Kulkarni, K. Pingali, B. Walter, G. Ramanarayanan, K. Bala, and L. P. Chew. Optimistic parallelism requires abstractions. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 211–222, 2007. [28] M. Kulkarni, K. Pingali, B. Walter, G. Ramanarayanan, K. Bala, and L. P. Chew. Optimistic parallelism requires abstractions. Communications of ACM, 52(9):89–97, 2009. [29] S.-W. Liao, A. Diwan, R. P. B. Jr., A. M. Ghuloum, and M. S. Lam. SUIF Explorer: An interactive and interprocedural parallelizer. In Proceedings of the ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 37–48, 1999. [30] M. Litzkow, T. Tannenbaum, J. Basney, and M. Livny. Checkpoint and migration of UNIX processes in the Condor distributed processing system. Technical report, U. Wisconsin-Madison, 1997. [31] M. Mehrara, J. Hao, P.-C. Hsu, and S. A. Mahlke. Parallelizing sequential applications on commodity hardware using a low-cost software transactional memory. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 166–176, 2009. [32] R. Numrich and J. Reid. Co-Array Fortran for parallel programming. ACM Fortran Forum, 17(2):1–32. [33] C. E. Oancea, A. Mycroft, and T. Harris. A lightweight in-place implementation for software thread-level speculation. In Proceedings of the ACM Symposium on Parallel Algorithms and Architectures, pages 223–232, 2009. [34] J.-K. Peir and R. Cytron. Minimum distance: A method for partitioning recurrences for multiprocessors. IEEE Transactions on Computers, 38(8):1203–1211, 1989.

27

[35] C. J. F. Pickett and C. Verbrugge. Software thread level speculation for the Java language and virtual machine environment. In Proceedings of the Workshop on Languages and Compilers for Parallel Computing, pages 304–318, 2005. [36] L. Rauchwerger and D. Padua. The LRPD test: Speculative run-time parallelization of loops with privatization and reduction parallelization. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, La Jolla, CA, June 1995. [37] M. C. Rinard and M. S. Lam. The design, implementation, and evaluation of Jade. ACM Transactions on Programming Languages and Systems, 20(3):483–545, 1998. [38] S.-B. Scholz. Single assignment c: efficient support for high-level array operations in a functional setting. Journal of Functional Programming, 13(6):1005–1059, 2003. [39] Y. Song and Z. Li. New tiling techniques to improve cache temporal locality. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 215–228, Atlanta, Georgia, May 1999. [40] M. F. Spear, K. Kelsey, T. Bai, L. Dalessandro, M. L. Scott, C. Ding, and P. Wu. Fastpath speculative parallelization. In Proceedings of the Workshop on Languages and Compilers for Parallel Computing, 2009. [41] C. Tang, D. Chen, S. Dwarkadas, and M. L. Scott. Integrating remote invocation and distributed shared state. In Proceedings of the International Parallel and Distributed Processing Symposium, 2004. [42] W. Thies, V. Chandrasekhar, and S. P. Amarasinghe. A practical approach to exploiting coarse-grained pipeline parallelism in c programs. In Proceedings of the ACM/IEEE International Symposium on Microarchitecture, pages 356–369, 2007. [43] C. Tian, M. Feng, V. Nagarajan, and R. Gupta. Copy or discard execution model for speculative parallelization on multicores. In Proceedings of the ACM/IEEE International Symposium on Microarchitecture, pages 330–341, 2008. [44] UPC consortium. UPC language specification v1.2. Technical Report LBNL-59208, Lawrence Berkeley National Lab, 2005. [45] C. von Praun, L. Ceze, and C. Cascaval. Implicit parallelism with ordered transactions. In Proceedings of the ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, Mar. 2007. [46] A. Welc, S. Jagannathan, and A. L. Hosking. Safe futures for Java. In Proceedings of OOPSLA, pages 439–453, 2005. [47] M. J. Wolfe. High Performance Compilers for Parallel Computing. Addison-Wesley, Redwood City, CA, 1996. [48] D. Wonnacott. Achieving scalable locality with time skewing. International Journal of Parallel Programming, 30(3), June 2002. 28

[49] A. Zhai, J. G. Steffan, C. B. Colohan, and T. C. Mowry. Compiler and hardware support for reducing the synchronization of speculative threads. ACM Transactions on Architecture and Code Optimization, 5(1):1–33, 2008. [50] A. Zhai, S. Wang, P.-C. Yew, and G. He. Compiler optimizations for parallelizing general-purpose applications under thread-level speculation. pages 271–272, New York, NY, USA, 2008. ACM. [51] C. Zhang, C. Ding, X. Gu, K. Kelsey, and T. Bai. Continuous speculative program parallelization in software. In Proceedings of the ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2010. [52] C. Zhang, C. Ding, K. Kelsey, T. Bai, X. Gu, and X. Feng. A language of suggestions for program parallelization. Technical Report URCS #948, Department of Computer Science, University of Rochester, 2009. [53] E. Z. Zhang, Y. Jiang, and X. Shen. Does cache sharing on modern CMP matter to the performance of contemporary multithreaded programs? In Proceedings of the ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2010.

29