Scaling down large projects to meet the agile sweet spot

Kruchten, Philippe. "Scaling down projects to meet the Agile sweet spot." The Rational Edge, August 2004. http://www-106.ibm.com/developerworks/ration...
Author: Erik Jordan
1 downloads 0 Views 160KB Size
Kruchten, Philippe. "Scaling down projects to meet the Agile sweet spot." The Rational Edge, August 2004. http://www-106.ibm.com/developerworks/rational/library/content/RationalEdge/aug04/5558.html

Scaling down large projects to meet the agile “sweet spot”

By Philippe Kruchten University of British Columbia [email protected]

If the mountain will not go to Mahomet, let Mahomet go to the mountain. (Proverb) I often hear the phrase “scaling up agile processes” to tackle “large” projects, and I also hear varied opinions about whether it can be done or not. In this paper I view this problem slightly upside down. If agile processes work in some well-identified conditions, is there a way to transform large projects such that most of the work is done under the optimal conditions of agility, and therefore using all the good practices associated with those conditions? What are the consequences, the cost, and the constraints on the organization and the individuals of doing so?

The “Sweet Spot” for an Agile Project What is the ideal context for an agile project? Under which conditions are agile methods most likely to be successful? •

Small group. When no more than ten to fifteen people are on the project team, it is possible to maintain high-bandwidth, one-to-one communication. But this starts to become difficult when the team size exceeds twenty people. With smaller teams, no specialization of roles (analyst, designer, coder, tester), with handover of artifacts, is necessary.



Collocation. Again, this allows high-bandwidth communication and synchronous communication between people.



Customer availability. When the project team has direct access to the customer, it becomes easier for course corrections to be made periodically, thus rendering a better product at the end of the project. Better yet, your team has direct and continuous access to a good customer representative empowered to make decisions about the needs to be satisfied by the software under development.



Business application. Agile development works better for business software projects, in which interactive types of applications are being built, It works less well other kinds of

1

projects: for example, an embedded real-time system. •

New development. New software projects, often called “green field” projects, are better suited to agile methods than maintenance projects.



RAD programming environment. Agility requires a rapid application development environment that allows fast turnaround from construction of code to testing, and collective code ownership; ideally, development teams do daily builds and monthly delivery.



Short lifecycle. Projects should not exceed a few months and ideally have a rapid internal loop (one day).



Common culture. Vocabulary and an implicit process are shared, and individual developer practices rapidly align with each other.

There is no doubt that in this sweet spot for agile development projects, there are few elements that require consigning information to some intermediary artifact, such as change management tools or sophisticated modeling tools. Communication is direct, including communication with the most interested stakeholder, the customer. Note that I am not saying that agile development would not work outside of the sweet spot, but simply that the conditions listed above describe the perfect case.

The “Bitter Spot” for an Agile Project Conversely, we can describe the “bitter spot” for an agile project. Here are the conditions that cast agile projects into jeopardy: •

Large group. When one-to-one communication is possible, but it is hard to identify the right person to communicate with, agile methods tend to break down, especially when some team members become swamped by communication requests. Communication starts to be asynchronous, and teams begin to rely on documents of various forms, including specific channels and storage mechanisms, for managing requests and project information.



Distributed team(s). This again makes impromptu, rapid communication more difficult; various electronic media can be used (phone, Internet conferencing tools, instant messaging systems), but often organizations will opt to produce documents for asynchronous communication and rely on travel for resolving critical issues.



No empowered customer representative. Under this condition, requirements -- and the clarification, negotiation, and feedback about requirements -- tend to evolve over long and inefficient routes, via documents, occasional meetings, and decisions that are dragged up and down multiple levels of management and of organization boundaries.



Long cycle. When first delivery occurs two years or more after project inception, agile methods don’t hold up. Inevitable personnel turnover leads to “loss of memory” within the project; it gets difficult to find meaningful intermediate milestones or short term objectives for assessing progress or simply for motivating the team.

2



Inefficient programming environment. Lack of automated tools usually means long turn around, and no collective code ownership; handover of artifacts proceeds slowly from designer to coder, from coder to tester, etc.



Different development cultures. This is often found when multiple organizations are involved, which often includes subcontractors and outsource partners. To remedy this, the process has to be made much more explicit, but that means the process itself can become the dominant driver, the monster to satisfy: “We have to take this step because the process demands it.”

We should note that in the extreme, large projects have traditionally relied solely on extensive documentation for all stages of development, with project managers insisting on written documentation rather than on people to span the gaps that occur in a distributed environment, multiple organizations, long project cycles, and the disruptions of personnel turnover.

Large Projects “I never want to work in a large project like that again,” someone told me recently. Well, let’s face it: large projects are not going to go away because of the emergence of agile processes. “Size is some kind of grit, if not the grit.” wrote H. Erdogmus,1 and this grit is causing much friction, slowing projects to a crawling speed. I have spent most of my professional life involved in large projects: telecommunications, command and control, programming tool development, and so on. For the sake of this discussion, let me characterize a very large project to avoid questions about marginal conditions, such as “But how about twenty-five people instead of twenty?” or “What if teams are distributed over just two sites?” My large project has these characteristics: • 200 people, • is distributed across multiple time zones • workers report to three different companies • the lifecycle is 2 ½ years to first delivery, and six months for subsequent updates. There are some conditions that can be achieved equally in small and large projects: • good programming environment, • skilled people • access to the customer Let’s assume all three of these optimal characteristics to make it easy to compare my large project to a small one. However, in large projects certain testing capabilities may not be readily available to all developers -- e.g., there are only so many telephone switches or submarine simulators you can make available to any given programmer in the lab. This deficiency is usually not the case with small projects.

1

H. Erdogmus, "Let’s Scale Agile Up," Agile Times, vol. 2, pp. 6-7, 2003. 3

Introducing the Large, but Agile, Project Now that we’ve considered the optimal and suboptimal conditions for an agile project, and we’ve described the basic scope of a large project, the question becomes: Can this large project be managed via agile methods? My answer is, yes. It is possible (though not easy) to organize a large software development project so that a good proportion of the software practitioners will work in an environment close to the agile “sweet spot.” The rest of this discussion will describe how software development teams can go about this.

Setting up an ideal large project Of the 200 people involved in this project, suppose that I organize 10 teams of 15 people each, and use the remaining 50 as the superstructure, the glue that makes it work. In each of the 15 small teams, members are co-located, they have their own development playground, and they share a common culture. The key problems are: who is the customer? Is this customer empowered? How do I communicate with other people outside the boundary of my team? A large system is not just a loosely coupled set of small systems. How do we start on day one? As the project manager for this monster project, I will need to set up the project from various perspectives: • Organization. I need to define project structure, the various teams, their roles, responsibilities, and objective. • Schedule and rhythm. I need to define the milestones and their associated objective alongside a timeline. Communication. I need to decide how the various teams will communicate. Most of the answers lie in the software architecture, which is produced by the architecture team, and the function of the Elaboration phase (as it is known in IBM Rational Unified Process®2 and MBASE3 process frameworks).

Gradual organization: Inception and Elaboration The development work can start under one team of seven to twelve developers (the architects), working in an agile setting that is co-located, has a common culture, and so on, with direct access to the customer, and using an iterative work flow. The objective of the architecture team is slightly different from a classic agile project, however, since the immediate goal is not to deliver a functioning system, but to identify enough of the significant user requirements (user stories, use cases, nonfunctional requirements) to set up an architectural prototype that will facilitate •

Organizing use cases (and other aspects of requirements)

2

P. B. Kruchten, The Rational Unified Process: An Introduction, 3rd ed. Boston: Addison-Wesley, 2003. B. W. Boehm, D. Port, M. Abi-Antoun, and A. Egyed, "Guidelines for the Life Cycle Objectives (LCO) and the Life Cycle Architecture (LCA) deliverables for Model-Based Architecting and Software Engineering (MBASE)," USC, Los Angeles, USC Technical Report USC-CSE-98-519, 1999. 3

4



• • •

Identifying underlying mechanisms needed to support the implementation of these use cases, as well as the key nonfunctional requirements (scalability, performance, availability, etc.) Developing an architectural prototype to validate the design choices made, the technologies selected Organizing the code in subsystems And, as necessary, setting up guidelines and tools for further development: e.g., design and code standards, code management, change request and trouble report, and so on.

This initial architecture team proceeds iteratively, and the team gradually grows in size. When this team reaches fifteen members or more, they split into: • The architecture team, mostly customer involved, and making key architectural choices • The prototyping team, gradually building up and testing the architectural prototype, refactoring it, and pushing back up any issues detected, thus closing the loop. The architecture team (seven to twelve people) is the customer for two or three prototyping teams (consisting of thirty-five or more people).

Staffing the Project: Construction and Transition Towards the end of the elaboration phase, when the structure of the code starts to take shape and stabilize, additional teams are created, each one “owning” a chunk of the architecture. The implementation view (according to the “4+1 view model”4) serves as the basis for the partition of the code and the allocation of ownership to a set of teams. Each chunk of code now belongs to one and only one team. Ideally, each of the newly created teams is seeded with one or two members of the original architecture and prototyping teams to bring some of the project knowledge and emerging culture. Typically, each of these seed members is the new team’s “technical lead.” This person will play the role of the local architect and serve as the communication conduit to the core architecture team that remains, and, in some cases, with the customer. There are two types of teams involved in the development of code. • Feature teams, which will develop code directly related to user functionality. • Infrastructure teams, which develop common elements and services used across the feature teams, or support nonfunctional requirements (e.g., middleware, services, domain specific framework, and reusable assets). For each team, we will strive to create the ideal conditions of agility. The customer for the feature teams is the real customer(s). But the real customer(s) cannot spread themselves too thinly across multiple teams and locations. The customer for the infrastructure team is…all the other teams!, which may have multiple and conflicting needs. This is where select members of the core architectural team start playing the role of customers. They mediate between the real customer and the various teams, via the tech leads, and identify 4

See P. Kruchten, "The 4+1 View Model of Architecture," IEEE Software, vol. 6, pp. 45-50, 1995. 5

common needs across the team to be realized by the infrastructure teams. They become the customer representative for the infrastructure teams. Note that this does not prevent direct access by the feature teams to the real customer, but that contact should occur only when specialty, or subdomain, issues arise. For example, an air traffic control system has several subdomains: flight management, radar data, clearance, aeronautical information, conflict detection and resolution, and so on. As long as the customer involved is competent and empowered to make decisions about these subdomains, the project is in good shape. The architecture and infrastructure teams will continue to fine-tune, refine (refactor), and expand the architecture of the system throughout the Construction phase, and ideally this will be conducted with minimal disturbance to the other teams.

A note on system integration As we tackle this large project using agile methods, we need to be realistic in our approach to integrating the complete system. We not only need to assemble the pieces built by the various teams, but at the same time we need to provide each team the right “environment” to be able to perform their own builds and tests. Each team cannot do this on its own, as this may require resources beyond their capacity. This is the role of a system integration team, which is at the receiving end of all the software subsystems and commands access to the testing facility for systems level tests. This system integration team is derived from the initial prototyping team. It may also be responsible for certain types of test such as performance, load, availability, usability, and so on, or when special equipment is needed.

What about those agile “sweet spot” conditions? At this point in our project, we have completed the Inception and Elaboration phases, we have assembled the Feature and Infrastructure teams, and are about to begin the Construction/Transition phases of the project. Figure 1 shows the evolution of team structure over the course of Inception, Elaboration and Construction/Transition phases. The “blue” teams are the ones that could operate close to the agile “sweet spot.” The “green” teams are the ones that provide the necessary communication and glue to make it happen. During Inception and Elaboration a system architecture is prototyped, based on customer needs, and this serves as the blueprint for a team structure; it shows where small, cross-functional teams are set up, and a how superstructure is put around them to allow them to efficiently operate.

6

Inception

Construction

Elaboration Management team

&

Transition

Management team Architecture team

Initial team

Feature team 1

Architecture team

Feature team 2 Prototyping team

Infrastructure team A

Feature team 3 Infrastructure team B

integration team Figure 1: Evolution of team structure over time

Problems I said this was not going to be easy. Several dangers lie ahead in this set up. I’ll cover a few of them.

Dependencies between teams We have created a problem that does not typically occur in our ideal (“sweet spot”) agile projects. As I have indicated, any given team, such as a feature team, is the customer of another team, and is dependent on that team’s delivering some functionality. To reduce the impact of this dependency on its ability to proceed building and testing their code, they will not only need to build tests associated with their “user story,” but also to build a stub, proxy, or mock-up that will be passed along in their role as customer to the infrastructure team.

Stove pipes Each feature team becomes a single project in itself; large portions of functionality end up being replicated across the team. This is exacerbated by the geographical distribution of the team members in this large project and by the different organizations at work, which do not share a common culture. The remedy for this? The architecture team must provide some communication glue, participate 7

regularly in design reviews, and bring in people from other teams. They can identify potential for common code or functions to be developed by infrastructure teams, and they can indicate which elements should be reused from one team to another. They can also suggest rotation of staff when possible across teams.

Figure 2. Reducing the “stove-pipe” effect. The improvement from side A to side B respresents identifying and developing more common code across various teams, and having an organization to match.

Imbalance Because the workload is uneven between teams, overloaded teams can potentially harm the whole project by delays or poor quality. Sometimes the imbalance is more in the mix of expertise among team members. The solutions to this include: change of personnel allocation (ratio) of subsystems to team; rotation of staff to spread knowledge and expertise; and temporary reallocation of staff from underloaded teams to overloaded teams.

Communication breakdown When communication between teams becomes inefficient, delays, frustration, and work blockage can result. The architects must step in and establish and facilitate direct communications between the involved parties.

Too much staff too early In general, it is not a good idea to have too many people involved in the Inception-Elaboration phases. But if your only alternative is idling a portion of your staff during these phases, it may be possible to start using some of the staff in small agile projects to develop parts of the system that are obviously necessary from day one. 8

Loss of a common goal, lack of focus Again, although small teams operate with their own objectives, they must never lose sight that the real success is in delivering the complete system, not just their chunk. They are not in competition with the other teams. And they must be open to communication, suggest improvements, and always be ready to rebalance the project workload.

Dual rhythms (two nested beats) When the lifecycle is long, measured in years, for example, it may be necessary to introduce two levels in the project timing, in the project “beat.” Figure 3 shows development teams working with a monthly iteration cycle (an internal release), with a daily “scrum.”5 But this beat is inside a slower system level beat, with a system major increment of six months, and a system level scrum of a week or two.

Weekly/biweekly Weekly/biweekly scrum scrum

System System increment increment beat: beat: 66 months months

Daily Daily scrum scrum Team Team increment: increment: 11 month month

Figure 3. Two beats (scrum of scrum) 5

See K. Schwaber and M. Beedle, Agile Software Development with SCRUM. Upper Saddle River, NJ: PrenticeHall, 2002. 9

Organizing documentation Some tell me that as soon as I use the unfortunate word—documentation—everything collapses: the project is not agile anymore. But this shouldn’t be the case. Project managers should only introduce permanently maintained artifacts (documents, models etc) where they are needed: that is, when they resolve a key problem and allow the complete project to be globally more efficient.6 Architecture. The architects will need at some point to document the architecture, if only to avoid spreading themselves too thin when the overall team grows, and having to explain the structure, the rationales, the “metaphors” again and again to newcomers. A well-documented architecture provides a single visible reference for all team members. Other non-architectural aspects of the design may never be documented explicitly in a separate document, but only in comments attached to the code, from which tools like JavaDoc can extract it when necessary. Requirements. Requirements are kept very fluid at first, but as the various feature teams progress in the implementation, use cases are collected, together with any nonfunctional requirements and organized to facilitate tracing, and testing, and to serve as the starting point for user documentation and training.

Risks, issues, and backlogs Risk issues and product backlogs should be treated at the level of each team; only the risk issues and elements of backlog that cannot be rapidly resolved at the level of a single team should be escalated to the project level. Technical issues are typically handled by the architecture team, and organization issues (staffing, resources, etc.) by the project management team. Often issues arise where the “owning” team is not clear, or the problem straddles two teams. Architects should arbitrate to resolve these problems. A common project-wide tool to manage local and global issues is a useful part of the development environment (for example, ClearQuest).

Process Description of practices, standards, guidelines, and templates is best made explicit in order to increase uniformity of practice across the whole organization. A clear description of the process should support the teams in their efforts to succeed, and should not be used in a coercive, administrative fashion.

Concrete examples Do I have any evidence that large projects can be managed, at least in part, by agile techniques? Or did I pull this out of my hat? Not quite. Most aspects of this strategy have been applied to large, iterative development projects I was personally involved with. Here are a few examples:

6

Let’s ignore for now the documents that must be produced for contractual reasons, or for compliance to standards and certification organisms -- FAA, FDA, etc. -- since these forms of required documents don’t necessarily benefit project organization. In any case, they constitute another topic. 10

1. Ship System 2000 (SS2000), a family of ship control systems developed by Philips/Bofors/NobelTech/CelsiusTech in Sweden7 2. The Canadian Automated Air Traffic System (CAATS) developed by Hughes Electronics/Raytheon in Canada8 , Both these projects had the large-project characteristics I indicated at the beginning of the paper. Ideas about matching the team structure to architecture -- i.e., that the architecture team, integration team, and feature and infrastructure teams should match the static view of the software architecture, as shown in Figure 4 -- grew out of the SS2000 project. These ideas where applied and refined in the CAATS project, with the introduction of use cases. Additionally, we were fortunately able to convince the customer to give us access to the entire team of end users on site. The architecture team took over the communication role, and they were able to resolve issues as they arose across the various teams.9 Another large project was of a different kind: 3. The Rational Suite, developed by Rational Software Corporation, with 700 developers and a delivery beat of six months. The product group that produced the Rational Suite had a dozen teams on eight sites around the globe, with a central product definition (i.e., the customer representative) in Lexington, Massachusetts, a distributed architecture team, and a central integration team. The feature teams were each co-located on the various sites. Each of these teams was more or less agile depending on its own origin, history, and culture. Each team has its own internal beat, matched to align with the beat imposed by the central product teams. My team in Vancouver, British Columbia, was very agile, accommodating lots of tactical changes, with a daily build and a more major release every other week or so. A few years ago, my colleagues at Rational Software reported many other iterative projects where these ideas have been implemented, with some variants, and with success.

7

See L. Brownsword and P. Clements, "A Case Study in Successful Product Line Development," Software Engineering Institute, Technical report CMU/SEI-96-TR-035, 1996. 8 See T. Paine, P. Kruchten, and K. Toth, "Modernizing Air Traffic Control through Modern Software Methods," in 38th Annual Air Traffic Control Association Conference. Nashville, Tenn.: ATCA, 1993. Also P. Kruchten, "Iterative Software Development for Large Ada Programs," in Proc. Ada-Europe conference, Montreux, Switzerland, vol. LNCS 1088, A. Strohmeier, Ed.: Springer-Verlag, 1996, pp. 101-110. Also P. Kruchten and C. J. Thompson, "An Object-Oriented, Distributed Architecture for Large Scale Systems," in Proc. of Tri-Ada'94, Baltimore, November 1994: ACM, 1994. 9 I describe the role of the architects and the architecture team in P. Kruchten, "The Software Architect, and the Software Architecture Team," in Software Architecture, P. Donohue, Ed. Boston: Kluwer Academic Publishers, 1999, pp. 565-583. 11

Applications

Common classes

Infrastructure middleware

Figure 4. On SS2000 and CAATS, the team structure exactly matched the implementation view of the architecture (i.e., the code view). For details, see P. Kruchten and C. J. Thompson, "An Object-Oriented, Distributed Architecture for Large Scale Systems," in Proc. of Tri-Ada'94, Baltimore, November 1994: ACM, 1994.

Summary I have described the gradual organization of a large project as a team of teams, where an objective is to set the ideal conditions of agility for a subset of the teams—the ones producing the quasi totality of the code—and with a few ancillary teams whose main objective is to ensure efficient communications between teams, and to implement the external interface that each team would expect to have in a smaller project. For very large projects this is done through: •

Communication inside and outside the team, using the architecture team, the management team, or the integration team as nodes of communication



“Scrums” and other facilitation and resolution practices at the team level and inter-team



Builds and releases at the team level and inter-team 12



Issues and risks managed at the team level and globally.

It is preferable that all teams operate according to the same overall rhythm. Several ingredients are key to making this successful: •

The ancillary teams (architecture and integration) must remain at the service of the other teams, and not the reverse. Their role is to ensure that the development teams are as efficient as possible, and therefore the ancillary teams must address the needs of the development teams as rapidly as possible through external communication or resources: for example, through communication with the customer or with other teams.



The team involved in code development must keep in mind the ultimate global objective, the final system, which has priority over their own team objectives, performance, schedule, and so on. Teams involved in infrastructure must be very responsive to the needs of their customers, who are the other teams.



Documentation and an increase in formality (sometimes called the “degree of ceremony”) must be introduced gradually and only when and where it allows the organization to be more efficient.



Architecture is the key to setting up the team structure in a way that minimizes the need for cross-team communications and dependencies.

The project as a whole needs ready access to a competent and empowered customer representative, who can make decisions about the system. The project needs powerful tools to manage quick turnaround of code, tests and test suites, and to support change management. And, as in any project, skilled and motivated people are the most important ingredient.

Acknowledgements This paper benefited greatly from discussions with the participants to the Canadian Agile Workshop in Banff, February 2003, the USC affiliates research review in March 2003 and March 2004, and internal reviews by Gary Pollice, Walker Royce, and Joe Marasco from Rational Software Corporation, now IBM Rational, as well as Hakan Erdogmus from NRC. References H. Erdogmus, "Let’s Scale Agile Up," Agile Times, vol. 2, pp. 6-7, 2003. P. B. Kruchten, The Rational Unified Process: An Introduction, 2 ed. Boston: Addison-Wesley, 2000. B. W. Boehm, D. Port, M. Abi-Antoun, and A. Egyed, "Guidelines for the Life Cycle Objectives (LCO) and the Life Cycle Architecture (LCA) deliverables for Model-Based Architecting and Software Engineering (MBASE)," USC, Los Angeles, USC Technical Report USCCSE-98-519, 1999. P. Kruchten, "The 4+1 View Model of Architecture," IEEE Software, vol. 6, pp. 45-50, 1995. K. Schwaber and M. Beedle, Agile Software Development with SCRUM. Upper Saddle River, NJ: Prentice-Hall, 2002. 13

L. Brownsword and P. Clements, "A Case Study in Successful Product Line Development," Software Engineering Institute, Technical report CMU/SEI-96-TR-035, 1996. T. Paine, P. Kruchten, and K. Toth, "Modernizing Air Traffic Control through Modern Software Methods," in 38th Annual Air Traffic Control Association Conference. Nashville, Tenn.: ATCA, 1993. P. Kruchten, "Iterative Software Development for Large Ada Programs," in Proc. Ada-Europe conference, Montreux, Switzerland, vol. LNCS 1088, A. Strohmeier, Ed.: SpringerVerlag, 1996, pp. 101-110. P. Kruchten and C. J. Thompson, "An Object-Oriented, Distributed Architecture for Large Scale Systems," in Proc. of Tri-Ada'94, Baltimore, November 1994: ACM, 1994. P. Kruchten, "The Software Architect, and the Software Architecture Team," in Software Architecture, P. Donohue, Ed. Boston: Kluwer Academic Publishers, 1999, pp. 565-583.

14