Supporting Distributed Collaborative Prioritization for WinWin Requirements Capture and Negotiations Jung-Won Park, Daniel Port, Barry Boehm University of Southern California Center for Software Engineering Los Angeles, CA 90089, USA { jungpark, dport, boehm } @ sunset.usc.edu Hoh In Texas A&M University Computer Science Department College Station, TX 77843-3112, USA [email protected]

ABSTRACT One of the most common problems within a risk driven software collaborative development effort is prioritizing items such as requirements, goals, and stakeholder win-conditions. Requirements have proven particularly sticky in this as it is often the case that they can not be fully implemented when time and resources are limited introducing additional risk to the project. A practical approach to mitigating this risk in alignment with the WinWin development approach is to have the critical stakeholders for the project collaboratively negotiate requirements into priority bins which then are scheduled into an appropriate incremental development life cycle. We have constructed a system called the Distributed Collaboration Priorities Tool (DCPT) to assist in collaborative prioritization of development items. DCPT offers a structurally guided approach to collaborative prioritization much in the spirit of USC’s WinWin requirements capture and negotiation system. In this paper, we will discuss the prioritization models implemented within DCPT via an actual prioritization of new WinWin system features. We also discuss DCPT’s two-way integration with WinWin system, some experiences using DCPT, and current research directions.

mutually satisfactory to all stakeholders is nearly absurd. In such a case it is inevitable that some requirements will not be fully implemented and the developer is usually left to decide which requirements will be scaled down or dropped. All too often this is done in a panic as a deadline approaches or the budget burns up. Even if a developer is provided priorities, there are too many complex factors that a developer does not or can not account for in order to mitigate the critical risk of not implementing an acceptable subset of the requirements. There must be a balance between stakeholder priorities such as described by Royce [7] in principle 16 of “the old way and the new” software engineering principles. An approach to this difficulty is to incorporate early input of the critical stakeholders through prioritization and to avoid “late risk resolution” - also described by Royce [7] as one of the top five conventional management flaws.

1. INTRODUCTION

Prioritization approaches have proven to be effective, yet cumbersome. The general approach is to have all stakeholders negotiate the relative priority of a set of items through negotiation in the same place at the same time. This is often not practical within many software development projects where stakeholders (including developers) are highly distributed and have greatly conflicting schedules. Naturally this problem sharply increases with the number of people. Even under ideal conditions, the true value of prioritization is never realized due to “collaborative memory loss.” Here vital information generated through discussion and interaction of the stakeholders slips into a veritable black hole. If (i.e. when) requirements need to be re-negotiated, the lost information may need to be re-created at great effort and loss of integrity and original intent.

Prioritizing software development items such as requirements, goals, and stakeholder win-conditions is important yet often neglected part of the development process. Frequently items must be prioritized up or down based on a multitude of complex factors that are amortized across many stakeholders. Under such conditions it is important not only to explicitly prioritize the items, but also to ensure stakeholder concurrence. Failure to do so may introduce critical risk [6] into the project. For example it is common for a project to have more requirements than can be fully implemented within a limited schedule and budget. Implementing requirements under these conditions optimistically expecting a result that is

Prior work in this area that simply records collaborations are well known to generate massive quantities of data wherein the essential information is forever buried thus impractical for use. No guidance or support (such as conflict identification) is provided to drive the process or utilize the results. Other prioritization systems tend to focus on a particular voting scheme or prioritization model, avoiding the important issue of rationale capture and adaptation to project needs. Also missing is a basic foundation that defines the prioritization model, its assumptions, and its consequences. The goal of DCPT is to address these issues in a practical and effective manner within the scope of software engineering. Although

Keywords: Prioritization, WinWin, Requirement Capture, Requirement Negotiation, Distributed System, Risk Assessment and Management.

there exist numerous approaches within the business management context, they are too general to accommodate the particularities of software engineering practice. It is this limited scope however that makes this an interesting research area and allows for the creation and application of practical tools. Throughout the paper will give examples of an actual usage of DCPT for prioritizing new requirements for USC’s popular WinWin system.

2. RELATED WORK Relatively few research efforts have been done on the area of requirement prioritization and application to the software development process [1]. Some recent work by J. Karlsson and K. Ryan [2, 3] uses the Analytic Hierarchy Process (AHP) [10] to prioritize each requirement’s relative value and implementation cost, which are then plotted on a cost-value diagram. H. Jung extended Karlsson and Ryan’s work with a variant of the 0-1-knapsack model that reduces the complexity of prioritization in cases of a system with many requirements or closely grouped cost-value points [4]. However, even if above systems prove useful in examples, they have inherent limitations. Some of these limitations have to do with implicit assumptions of the prioritization model, which may not be in alignment with stakeholder needs. For example the use of fixed priority bins, an issue we will discuss in detail later. Related prioritization systems also do not accommodate multiple prioritization models that may be in better alignment with stakeholder needs; e.g. Jung and Karlsson only use the pair-wise comparison model using AHP. DCPT integrates multiple voting and bin (classification) models, and very unlike prior work, an explicit risk model. This is of increasing importance within software engineering as more developers adopt iterative risk-driven development lifecycles [6]. At a more fundamental level, related efforts neither provide a facility to capture rationale nor address issues of priority ambiguity and resolution – both major sources of difficulty in making effective use of prioritization. Furthermore, related works in this area are not concerned with distribution. That is, enabling collaboration of non co-located stakeholders to interact asynchronously. Although our system by design addresses this, admittedly only in a limited way at present and we have yet to deal with the more complex and subtle issues of distribution. In the following sections, we will discuss the WinWin system and the Distributed Collaboration Priorities Tool (DCPT). We also discuss prioritization models and show how our DCPT uses these models and address the limitations described above.

3. WINWIN

WinWin is a distributed system (implemented in C and Visual Basic) for the collaborative negotiation of requirements. WinWin uses Theory W - Make Everyone a Winner - as its negotiation model [8]. Developed at the USC Center for Software Engineering, WinWin has been applied to the WinWin negotiation process [9] using the WinWin2.0 negotiation tool for facilitating the negotiation of requirements for geographically and temporally distributed stakeholders. Stakeholders write their pre-requirements in the form of Win Conditions. Conflicts or concerns of the satisfaction of Win Conditions are raised as Issues that are addressed by Options. Options are considered and ultimately stakeholders propose Agreements to adopt Options and Issues which then become requirements. Through three years of use on customer based projects within USC's graduate level software engineering course (CS577), WinWin has proven useful, yet not devoid of concerns. The class projects are relatively small with between 6-18 Win Conditions and 4-7 stakeholders. We have noted that if the number of stakeholders or the number of win conditions is increased, the WinWin process tends to stall and agreements become difficult to achieve [9]. Utilizing the large amount of information captured through WinWin to create actual project requirements has also proven difficult.

4. DCPT DCPT is a distributed client-server based system (implemented in Java) used in the collaborative prioritization of development items and risk reduction. Stakeholders assign each item a difficulty and importance (or a probability and loss) for which summary statistics are generated and used to place the items into priority bins that classify them into a relative prioritization. Figure 1 shows the snapshot of DCPT. DCPT has four basic panels. Top left and top right panels are item vote summaries. Each panel represents a particular item for which each stakeholders vote is plotted on the value-pair grid along with the initials of the stakeholder. The ellipse indicates the area of consensus between the votes for that item. The size and shape of this ellipse summarizes the degree and directions of disparity amongst the votes. A votes exact value-pair and rationale can be viewed simply by moving the mouse over the vote of interest. The line shows the linear least squares fit of the value-pairs. Two panels are shown for pairwise comparison of the items. The bottom left panel is used for displaying the relative priorities according to the bin models. Bottom right panel is the result of the prioritization with respect to the bin model selected. An item is placed in the overlap column when the area of consensus of the item does not entirely within a bin area fall (there is a tolerance adjustment that we have set very high for the screenshot examples to avoid a cluttered display).

include some form of comparison with an absolute (i.e. a value assignment) or relative to the requirements, such as pair-wise comparison. The votes are then summarized according to some agreed upon policy (such as majority rules, average value, median) and then a value is assigned to the requirement. This only deals with part of what is needed for a prioritization. Requirements eventually must be grouped with respect to some model based on the voting. Example groupings are “high, medium, low” or “priority I, II, III”. These groupings are what we will refer to as priority bins. A prioritization model is the combination of a voting model and a bin model.

Fig. 1. Distributed Collaboration Priorities Tool (DCPT) snapshot. The diagonal bin model is used indicating that item 1 is in bin I, items 2 and 3 in bin II, items 4, 5, and 6 in bin III. 5. PRIORITIZATION MODELS: VOTING AND BINS Prioritization within Software Engineering The Webster’s dictionary defines priority as: …superiority in rank, position, or privilege, …a preferential rating; especially one that allocates rights to goods and services usually in limited supply, …something given or meriting attention before competing alternatives. Then prioritization is defined as “to list or rate (as projects or goals) in order of priority”. For the remainder of the paper we will use requirements as the items to be prioritized. Other software development items such as goals, win conditions and so forth could be substituted easily with only minor modifications. Within software engineering the definition of prioritization translates into choosing an ordered partition of the requirements with respect to the tradeoffs and constraints of the project. In spirit this is an application of the classic “divide, order, and conquer” paradigm. The goal is to minimize overall risk (through reduced complexity) by identifying classifications and an ordering that maximize satisfaction over all the stakeholders. The satisfaction aspect requires input from the stakeholders. One way to do this is have stakeholders assign judgements through voting on the items. Judgements vary, but must

DCPT uses four different kinds of bin models: the average bin model, the ratio model, the diagonal model, and the risk assessment model. An important feature of these bin models is that the bin locations and sizes are adjusted relative to the vote values of the items. This in contrast to fixed bins, where bin locations are arbitrarily chosen and have no direct relation to the particular values assigned to the items. This we will see enables a more meaningful prioritization and avoids such problems as all items falling into a single bin. At present DCPT accommodates only one voting model where stakeholders assign a pair of values between 0-10 as to their assessment of the relative importance and difficulty, or the relative probability and loss for a given requirement. The voting scheme is used to assign importance-difficulty (benefitcost) values and the probability-loss (risk) values to requirements. The two value assignments are not necessarily totally orthogonal, yet they still provide an important new dimension in which to form valuable (i.e. risk minimizing, satisfaction maximizing) prioritization. We will soon support pair-wise comparison (AHP) and finite budget voting model (points). The exact relationship between different voting models and how it affects bin choice and disambiguating requirements into bins is an important active area of our current research. The Voting Model A stakeholder votes by logging into the DCPT server after which a voting display is presented. The stakeholder selects an item in which to assign a difficulty and importance value pair. Simply moving the cursor to the point on the grid that represents their judgement and pressing the mouse button makes the value pair assignment. To change a vote, the stakeholder moves the mouse over the point to be changed and drags it to a new location. If any vote requires additional notes or rationale, it can be entered in the notes text box. The figure below shows an Importance-Difficulty voting on items 1,2,4,6 from the list at the top (where item 1 is the currently selected item). Note how the stakeholder can visually asses the relative importance (or benefit) of items and the relative difficulty (or cost) of all the items he has an interest in. It is not required that a stakeholder vote on all the items. A stakeholder can access the system anytime and from any location, thereby addressing a simple distribution issue. At present DCPT handles notification of issues through its integration with the WinWin system.

Fig 2. Voting on Importance-Difficulty (Benefit-Cost) for WinWin new feature requirements. Bin Model In this section we describe the currently supported bin models, their assumptions, and application to the prioritization. Given a set of assumptions on the relationship between difficulty and importance, bin models represent various ways in which the value-pair grid is to be partitioned. The regions in which the grid is partitioned are assigned bin labels. Currently DCPT supports partitions into exactly four regions which are labeled I, II, III, and IV. When a requirements summary vote value places it in on the grid, it falls within one of the regions. DCPT bin models only place items into bins according to a voting model. The stakeholders must determine the ordering of the bins with respect to an individual project. For example, a-priori why should an item placed in bin II have higher priority over an item placed in bin III? It should not unless we prefer importance to difficulty.

Fig. 3. Average Bin Model. Numbers within a square indicates the prioritization item number. Item 1 is ambiguous which may belong to Region 1 or Region III. Item 4, 5, and 6 belongs to Region II. Item 2 and 3 belongs to Region III. Ratio Bin Model: The ratio bin model partitions the grid into regions based on baselines where importance over difficulty is a constant. The baseline is set where importance / difficulty = (average importance) / (average difficulty). The other regions are determined by adding and subtracting a value of 0-3 standard deviations (fractions allowed) of the importance and difficulty from the numerator and denominator respectively. This model is similar to Karlsson and Ryan’s cost-value model [3], which can be, interpreted as Return-OnInvestment (ROI). However, in their work the regions were fixed rather than relative to the averages as described above.

Average Bin Model: This model provides independent baselines for importance and difficulty. The base is set by the average value whereby there are an approximately equal number of items above and below the baseline. The horizontal line near Importance 7 indicates the average of importance of items and items 4,5,6 are above and 1,2,3 below. The vertical line near Difficulty 6 indicates the average of difficulty of items with 1,2,3 to the left, and 4,5,6 to the right. Although there are not equal numbers of items in each bin, there are equal numbers above and below the baselines.

Fig. 4. Ratio Model. Item 1 has the largest Return-OnInvestment (ROI). Item 2 has a medium ROI. Items 3, 4, 5, and 6 have the relatively small ROI.

Diagonal Bin Model: This model assumes that importance and difficulty are exchangeable in which the importance minus the difficulty is constant. The baseline is where importance - difficulty = average importance - average difficulty. The other bases are set at this value plus or minus the sum of 0-3 standard deviations of the importance and difficulty. This model can be interpreted as Net Value [5].

exposure necessarily) while satisfaction (i.e. the cost-benefit).

maximizing

stakeholder

Fig. 6. Probability-Loss (Risk Exposure) Model. Risk Exposure(RE) of item 4 over 67. RE of items 4,6 is between 41 and 67. RE of item 3 is between 20 and 41. Items 1,2 have lowest risk exposure. Fig. 5. Diagonal Model. Item 1 has the largest net value. Item 2 has the second. Item 3, 4, have third largest net value. Item 5 is ambiguous. Item 6 has the lowest net value. 6. EXPERIENCE USING DCPT AT USC Risk Bin Model: This model can be used for assessing relative risk of the requirements based on the importance-difficulty value-pairs and additional stakeholder judgements, or an algorithmic determination from theoretical considerations. The latter is complex and highly experimental, but is loosely based on COCOMO approaches to effort and risk determination [5]. At present we do not have much theoretical basis to draw upon. What we do know is that there are important relationships between risk and prioritization (see future work section.) The generally accepted evaluation of risk of individual independent requirements is the product of probability of the risk occurring and the loss of the outcome if the risk occurs (i.e., risk = probability * loss). Figure 6 then illustrates the risk exposure of prioritization items [6]. The regions are partitioned according to contours of constant risk exposure where the probability of loss times the cost of loss is constant. The baseline is set at the average probability of loss times the average cost of loss. The other regions are chosen to be 0-3 standard deviations from the baseline. The addition of risk assessment allows for a different dimension in which to prioritize requirements. Depending on the risk tolerance of the stakeholders, the risk view will help determine an ordering and meaningful interpretation of the bins. The goal of prioritization is to minimize risk (not risk

Aside from the example used in this paper, DCPT has been used to efficiently and effectively prioritize requirements for a number of development projects. We have also used DCPT to prioritize non-requirement development items. For example DCPT was used at 13th International Forum on COCOMO and Software Cost Modeling on October 6-9, 1998. At the forum, attendants wished to prioritize the COCOMO extension candidates such as COCOMO II (COnstructive COst MOdel) [11], CORADMO (COnstructive RAD MOdel), COCOTS (COnstructive COTS model), and COQUALMO (Constructive QUALity MOdel) [12]. DCPT was also used at the Focused Workshop during USC-CSE Technology Week (February 8-12, 1999). DCPT is particularly useful in workshop settings where collaboration among a large number of stakeholders must rapid and not a burden.

7. DCPT AND WINWIN DCPT incorporates an underlying model for stakeholder voting and categorization of items that guides the collaboration effectively without being overly restrictive. Our present work focuses on an intimate two-way integration with WinWin. In this setting the WinWin system is used initially to capture possible requirements in the form of Win Conditions. A new Win Condition is appended by DCPT (through the WinWin API) which states that all Win Conditions should be

unambiguously placed into priority bins that indicate their priority relative to each other. The Win Conditions are extracted from WinWin and entered as items within DCPT. DCPT is used by all interested stakeholders to vote on the Win Conditions captured by WinWin. Summary statistics are used to place the items into priority bins which adjust their size dynamically to provide a true relative prioritization. Some items may be ambiguous when a sufficient consensus could not be established. In such cases DCPT will analyze the stakeholders vote data and automatically generate new WinWin Issues indicating why an item is ambiguous and how it may be in conflict with other Win Conditions. Also generated are possible Option artifacts that offer possible resolutions for the ambiguous item priority Issues. Such Options may be adopted thereby resolving the Issues and ultimately disambiguating the items. Notification of the new issues that need negotiation is handled through the WinWin system.

8. CONCLUSIONS Although still in its infancy, DCPT has already proven to be an effective and valuable tool with respect to software engineering projects. It adapts well to a large variety of development lifecycles and enables early and continued risk reduction. At its core, the distributed voting model enables collaboration between large numbers of disparately located stakeholders. DCPT provides enough guidance through the integration of basic voting and bin models to rapidly archive a risk minimizing consensus without over constraining the projects fundamental assumptions and the workflow of the stakeholders. DCPT greatly assists in managing the complexities of a development effort through sensible prioritization. Without the use of DCPT, large prioritization collaborations are nearly impossible due to the additional complexity and as such are often neglected, introducing additional risk into a development effort. DCPT integrates naturally and easily with WinWin, and likely with other software engineering tools. Through refinements and extensions, DCPT can add attractive “agent based” issue generation and resolution that help further reduce the complexity of a prioritization task, thereby enabling the use of risk driven iterative development lifecycles. Aside from the topics discussed above, DCPT opens up the exploration into the fundamental nature of software risk management through collaboration, classification, and partitioning.

9. FUTURE WORK Prioritization Constraints Our current version of DCPT asks stakeholders to vote importance and difficulty at the same time. However, some stakeholders such as a typical user may not know the relative difficulty (cost) of implementing a requirement, yet still provide valuable input as to the relative importance. Similarly, some other stakeholder such as a developer may not know the relative importance (benefit) of a requirement. We are currently working on a way to handle such events. In addition, we are exploring ways to add weighting factors to the votes

and handling voting by groups of stakeholders in terms of representation and weighting. Disambiguation and Reconciliation of Item Priorities Some prioritization items may have unusual regions of consensus that non-trivially span bin partitions. We are looking into ways to provide more explicit guidance as to how such cases can be reconciled. Presently we simply place the item into an “overlap” bin and generate WinWin issue describing the nature of the ambiguity. Handling of Prioritization Item Interdependencies There should be a in-depth analysis of prioritization interdependencies that arise under a various conditions that implicit structural constraints on the prioritization process within software system development. We are looking into questions of how the prioritization of one item is interdependent on the prioritization of other items. Risk Assessment and Resolution Support As described in the introduction, one of our main objectives in prioritization was to reduce project risk by obtaining a consensus of the stakeholders as to how development effort should be focused. The assumption here being that the more stakeholders involved, the more assurance we would have of an item’s priority. We soon noticed that this introduced a form of risk in that the degree of consensus determines the level of assurance in the priority of a requirement. At present we deal with this by having an “overlap” bin. If a requirement could not unambiguously be placed into a bin (as defined by the consensus ellipse fitting entirely within a single bin model region), it is placed in the overlap bin. Subsequently, a WinWin Issue is automatically generated stating the need to negotiate stakeholder priority votes in order to resolve the ambiguity. It became clear that this was not entirely adequate, as there appears to be an intimate relationship between the priority of a requirement and its individual risk. In any attempt to resolve an ambiguity, the risk in some manner "weighs" or biases non-trivially in a way we do not understand at present. Ambiguities in risk can be defined similarly and have the same issues. Suppose a requirement’s risk exposure is ambiguous between high and medium and the majority of stakeholders deem it very important, but not difficult. This would imply a bias towards medium risk despite the actual stakeholders risk assessments, which in turn influences the priority. This only touches on the possible relationships between risk and priority. Clearly the collaborative assessment of risk is valuable for the same reasons as prioritization, yet the two combined may prove to be exceptional. The potential benefits and opportunities to apply DCPT into the risk assessment area in the future are in the following: 1. Negotiating the priority of risk resolution among multiple stakeholders effectively through a distributed system for showing and voting on the priority of risk resolution in the spirit of USC's WinWin requirements capture and negotiation system 2. Providing the following visualization of a current situation of risk assessment among multiple stakeholders and notifying the changes of risk assessment if stakeholders change their assessment:

•

3.

4.

Presenting current risk assessment, voting situation and identifying the points of disagreement among multiple stakeholders. • Indicating the nature of consensus among the votes. The size and shape of an ellipse indicate the degree and directions of consensus and disparity among the votes. • Providing an organized view of risks based on the Risk Taxonomy. • Managing (i.e., adding, deleting, changing, tracing, and querying) the rationales of the risk assessment per risk item per stakeholder. Providing risk-reduction guidelines for the prioritized risk items. • For example, a better risk-resolution strategy in Point A in Figure 7 may be to reduce risk probability rather than loss in terms of the distance to the next lower level of risk area. For Point B in Figure 7, a better strategy is to reduce loss. • Another example is that a consensus (vertical-linelike ellipse) having broad distribution of risk probability and narrow distribution of loss requires a focus on further sensitivity tests for determining and verifying the risk probability. Given limited resources (e.g., money, schedule, or personnel), supporting the tradeoff analysis of resolving prioritized items • Which one is the first to resolve risks under limited resources? • How much should the risk items be reduced? • How does the relationship between risk and priority affect tradeoff analysis?

A

B

Fig. 7. Risk assessment and resolution support

10. REFERENCES [1]

P. Zave, "Classification of Research Efforts in Requirements Engineering", 2nd IEEE International

Symposium on Requirements Engineering, 1995, pp. 214216. [2] J. Karlsson, "Software Requirements Prioritizing", Proceedings of ICRE 96, IEEE, pp. 110-116. [3] J. Karlsson, K. Ryan, "A Cost-Value Approach for Prioritizing Requirements", IEEE Software, Sept. 1997, pp. 67-74. [4] H. Jung, "Optimizing Value and Cost in Requirements Analysis", IEEE Software, July 1998, pp. 74-78. [5] B. Boehm, "Software Engineering Economics", PrenticeHall, 1981. [6] B. Boehm, "Software Risk Management: Principles and Practices", IEEE Software, Jan. 1991, pp.32-41. [7] W. Royce, "Software Project Management", Addison Wesley, 1998. [8] B. Boehm, R. Ross, "Theory-W Software Project Management: Principles and Examples", IEEE Transactions on Software Engineering, July 1989, pp. 902916. [9] B. Boehm, A. Egyed, "WinWin Requirements Negotiation Process: A Multi-Project Analysis", Proceedings of the 5th International Conference on Software Processes, 1998. [10] T. Saaty, "The Analytic Hierarchy Process", McGrawHill, 1980. [11] B. Boehm, B. Clark, R. Madachy, R. Selby, C. Westland, "Cost Models for Future Software Process: COCOMO 2.0", Annals of Software Engineering, Vol. 1, 1995. [12] S. Chulani, B. Boehm, "Modeling Software Defect Introduction Removal: COQUALMO(COnstructive QUALity MOdel), USC-CSE Tech Report 99-510, 1999.