The Problem with Requirements: Why Is It Still a Problem? Understanding Persistent Problems and Offering New Perspectives May 2014
Sponsored by:
Robbins-Gioia, LLC
PERFORMANCEINSTITUTE.ORG
The Problem with Requirements: Why Is It Still a Problem? Understanding Persistent Problems and Offering New Perspectives May 2014
The Performance Institute | performanceInstitute.org
The Performance Institute 11 Canal Center Plaza Alexandria, VA 22314 The Performance Institute is a private, nonpartisan think tank improving government results through the principles of performance, transparency, and accountability. Produced by The Performance Institute; Ron Bohlin as the executive sponsor. Doug Jackson and Susan Martin as key contributors. © Copyright 2014 by The Performance Institute No part of this book may be reproduced by any mechanical, photographic, or electronic process, or in the form of a phonographic recording, nor may it be stored in a retrieval system, transmitted, or otherwise copied for public or private use, without written permission from the publisher, except for the purposes of official use by the U.S. government. Printed in the United States of America
Table of contents 5
the problem with requirements
13
requirements as financial assets to the organization
14
The Next Step: Making the Case for Change
15
summary
16
references
17
survey questions
23
interview subjects
24
interview questions
The Problem with Requirements: Why Is It Still a Problem?
The Problem with Requirements Understanding Persistent Problems and Offering New Perspectives Organizations have not conclusively addressed the “requirements problem” despite having known for decades about its proven impact on outcomes and solution quality. We have evidence of the persistence of the problem from several research organizations, including the Standish Group, which has collected case information on real-life project failures and environments, and prepares its well-known CHAOS Report to disseminate its findings. That research shows that right from the beginning, poor requirements were a key contributing factor to project failure—and the statistics have barely budged since they first came out in 1995. What was true then is still true in 2005, and there’s not much reason to think that 2015 will be any different. Standish isn’t alone in its dismal view of the current state of requirements. According to Meta Group Research (now a part of Gartner), challenges between business teams and technologists are chronic. They estimate that 60 to 80 percent of project failures are directly attributable to poor requirements gathering and requirements analysis. Forrester Research concurred, saying: “Poorly defined applications have led to a persistent miscommunication between business and IT that largely contributes to a 66% project failure rate for these applications, costing U.S. businesses at least $30B every year.” In July 2013, the United States General Accountability Office (GAO) cited requirements as a leading factor in failed or at-risk IT acquisition projects. And as early as 1999, Karl Wiegers in his seminal book, Software Requirements, pointed to improving the requirements process as the single best investment IT organizations could make. He’s making the same assertion in the book’s third edition, which has just been released.
We also decided to focus on large, complex, high-value projects. This is not to say that smaller initiatives are not having some of the same problems, but because the delivery risk is most acute and present on large initiatives, our data collection prioritized large projects. Research performed by McKinsey in 2012 concluded that large IT projects (defined as those with initial price tags exceeding $15 million) run 45 percent over budget and 7 percent over time, while delivering 56 percent less value than predicted; some IT projects can even threaten the existence of a company.1 To support the conclusions in this brief, data was gathered using the following standard methods: Existing Research Review This research review analyzes a body of existing research into problems related to requirements, including effects on organizational performance, and cites causes when they were identified. Emphasis is placed on research that has been called out over a period of time, so that trends can be analyzed. In addition to research directly pertaining to “requirements quality,” research into other factors that may contribute to manifestations of a requirements problem are explored. All research used in this brief is cited in the endnotes of this document. Survey The Performance Institute administered two targeted surveys to selected demographic groups, including representatives from commercial organizations, non-profits , and governmental agencies. The objective was to gather original data by organization type and job function. Interviews
Requirements issues are a major contributing factor to poor project outcomes; but as one current advertisement puts it, “Everyone knows that.” Our question is, why hasn’t anything changed? In this paper, we set out to identify the reasons the requirements problem still exists, thinking that if those reasons could be understood they could also be addressed.
We interviewed representatives from two groups with expertise in the business analysis and requirements: the International Institute of Business Analysis (IIBA) and the British Computing Society (BCS). We also conducted interviews with leaders of IT project and competency improvement efforts at three large organizations—two in the commercial space and one non-profit. Finally, we interviewed thought leaders in the business analysis space. All interviewees were asked the same set of questions (in appendix).
Methodology
research focus
When The Performance Institute decided to investigate the requirements problem, the understanding was that it was indeed a problem, and that it had been proven as such by well-documented studies. We started from this premise. So we are asking why, despite being a wellknown problem for decades, hasn’t it been addressed?
In light of the persistence and apparent inability to solve the requirements problem, we were looking to find out: • Why haven’t more organizations been able to effectively address the requirements problem? • In organizations that have made improvements, what actions have had the greatest impact?
1
“Delivering large-scale IT projects on time, on budget, and on value,” Michael Block, Sven Blumbert, and Jurgen Laarz, McKinsey & Co. Published in McKinsey on Business Technology, Number 27, Fall 2012
performanceinstitute.org | 5
The Problem with Requirements: Why Is It Still a Problem?
We decided to investigate commonly held, but contrasting viewpoints, including: 1. Are requirements addressed early enough in the life cycle, or too early? 2. Is too much time spent on requirements definition, or too little? 3. Are requirements not adhered to as the project progresses, or are they too rigid to adapt to evolving needs? 4. Are the wrong people charged with defining requirements?
highlights of the research Requirements are the bridge between desires and reality, and therefore fundamentally shape the outcomes of change initiatives. Despite their value to the enterprise, most organizations apply surprisingly little rigor in their approach to requirements elicitation and management, and many executives appear to underestimate the impact of their requirements discipline on organizational success. Our survey asked respondents to comment on both their successful projects (if any) and on their challenged projects. “Challenged” projects were defined as those that (a) delivered more than two months after initially projected delivery date; (b) 15 percent or more over planned budget; and (c) the solution contained significant enough defects that it had to be re-worked to satisfy its stakeholders. The survey indicated that over 62 percent of organizations experienced one or more of those challenges in over 30 percent of the projects. Almost 11 percent felt that 100 percent of their projects launched with those challenges. With this as background, following are some of the major findings from our research. Too Many Projects Focus on the Wrong Things Based on the results of our research, “the wrong things” manifest in three key ways: 1. Building the wrong thing—deliverables don’t meet a strategic, business need 2
US Government Accountability Office. “Information Technology: OMB and Agencies Need to More Effectively Implement Major Initiative to Save Billions of Dollars,” July
6|
2. Focusing on the wrong thing—working to meet budget and schedule to the exclusion of tasks more critical to success, for example, ensuring that stakeholders are on board with the project’s direction, especially if there are changes along the way 3. Valuing the wrong thing—discarding requirements
at the conclusion of a project, when they could be an asset for subsequent work FROM AN INTERVIEW:
When a movie is a bomb, it’s typical that an executive somewhere has said, “Hey, you know what would be a great movie? ... We’ve got a great start, this big famous director, and we are going to make this movie.” They look at schedules and everybody involved and they block it off and say “this movie is going to start filming in April.” But sometimes you need a new script. And so while you think they should have waited for a movie with a decent script in hand—who knows why they don’t—the movie is made when the talent is available, the director is available, and the star is available when that scene starts, no matter what state the script is in. And that’s often how we handle it [in business] too. Projects get locked [in] and we assign the project manager, the development team, and you guys get coding, right? We don’t ... figure out what we really need to do, and [wait until] ... we really understand the problem and ... what it is we are really trying to achieve. Chief Business Analyst, IIBA
Tom DeMarco made this point in 1995: “For years we’ve tortured ourselves over our inability to finish a software project on time and on budget. But ... this never should have been the supreme goal. The more important goal is transformation, creating software that changes the world or that transforms a company or how it does business.” In delivering strategic need, we found that a laser focus was needed to ensure stakeholders were engaged, consulted, and understood. The GAO Report on Information Technology cited the top critical factor predicting project success as being “Program officials were actively engaged with stakeholders”—and this factor stood far and away above anything else.2 Our own research confirmed this through a number of questions about stakeholders. We included questions on whether solution stakeholders understood the requirements they were accepting, as well as whether they understood the likely impacts of the requested solution on the business. In a related question, we asked how often an important stakeholder was discovered too late to be included in requirements gathering effort. For successful projects, stakeholders were much more likely to have been identified and, once known, to have been involved enough to truly contribute to, un-
The Problem with Requirements: Why Is It Still a Problem?
Figure 1: Stakeholder understanding of requirements in challenged vs successful projects derstand, and review the requirements (Figure 1). The ability to engage stakeholders seems to be a crucial factor in project success. Kitty Haas, in her recent study of Business Analysis practices, states that, “standout organizations establish formal customer relationship management processes ... that help Business Analysts better manage the expectations of internal and external customers.” 3 Karl Wiegers best summarized the importance of stakeholders in this quote from 1999: “Only user representatives can determine the correctness of user requirements, which is why it is essential to include the ... Requirements inspections that do not involve users can lead to developers saying, ‘That doesn’t make sense. This is probably what they meant.’ This is also known as ‘guessing.’ ” Now what about the problems of valuing the wrong thing and discarding requirements at the conclusion of a project? Our survey respondents confirmed that in successful projects, requirements were useful after the solutions were developed, and many stressed that pre-existing requirements from previous projects helped them jump-start a new effort. This makes a strong argument for using a requirements management tool (beyond Word or Excel) that helps catalog requirements both during project execution and after. 3 Lori Lindberg and Kathleen Hass, The State of Business Analysis Practices in Organizations, 2011
performanceinstitute.org | 7
The Problem with Requirements: Why Is It Still a Problem?
Figure 2: Length of time allotted to requirements gathering for challenged vs successful projects
FROM AN INTERVIEW:
Going back to a bank I worked with years and years ago, there were all sorts of problems with how they developed their requirements. The reality was, given that many of their requirements were actually quite stable, if they had had an effective method to capture and store all those business rules about what a mortgage is and how things are calculated and so on, they would have saved themselves a lot of time and money over the years. What they ended up doing was recreating that work every time they developed a new system ... you’d start to see a falloff in cost if you were actually managing this information correctly. Chief Business Analyst, IIBA
Requirements Are Not Given Enough Time Perhaps not too surprisingly, for challenged projects a high percentage of our survey respondents felt that either too little or far too little time was spent on requirements (Figure 2). Respondents felt that the time allotted for requirements for successful projects was overwhelmingly “about right.” Notice that in our survey it was the exception to say that too much time was spent on the requirements, although the majority of our respondents were either business analysts or project managers, not solution stakeholders who are sometimes the ones pushing to “get it done.” From an interview:
I think that people don’t know what they want. Let’s say there is a business customer that wants a system, or a service that requires a system. They don’t know 8|
what they want, there’s resistance to change in existing business processes, some of them don’t even understand what a business process is ... People don’t realize that they have to step back and define thing ... They don’t know what they want and they don’t want to take the time to define ... and they want it now. CIO, Higher Education Institute
from an interview:
I think requirements need to start in understanding the problem. When systems engineering takes this on in the public sector they use techniques called MODAF or TOGAF—which are frameworks that are trying to understand the problem from a systems engineering perspective—the idea being that a system contains people and processes, as well as the technology and organizational structures. And I think the problem that happens is that people only start getting involved in the requirements process once an IT solution has been agreed upon, then it’s too late to uncover whether IT was the only way of solving that problem. Examiner for the British Computing Society (BCS)
The Wrong People Are Tasked with Doing Requirements Are the wrong people tasked with requirements? The answer is “yes.” But once again there are multiple dimensions to this simplistic response. • In some instances, solution vendors are creating requirements—never a good idea and we’ll talk about that a bit later
The Problem with Requirements: Why Is It Still a Problem?
Figure 3: Actual project length vs projected length for challenged vs successful projects • Where the business analyst role (or functional equivalent)is used, there is a significant difference between theory and practice because these resources: - Are not being brought into the project early enough - Do not have enough of the right skills - Are not consistently used and fully committed to the project Let’s start by exploring when the business analysts themselves may be “the wrong people.” Referencing the previously cited GAO report, where it identified nine critical factors for successful projects, two factors stand out: 1. Program staff had the necessary knowledge and skills 2. Government and contractor staff were stable and consistent
of business value from the proposed business improvement. Though this might make intuitive sense, one thought leader stated that she has seen “countless” projects where 1) the business case has been built 2) the project has been scoped 3) the budget allocations have been made and 4) the project schedule has been set—all without any requirements having been gathered. The result? Project managers are placed in the awkward position of “taking a swag” at project duration. Because there has been no investigation into possible impacts or alternative courses of action, program managers end up promising project and solution outcomes based on flawed or incomplete information. This unhealthy dynamic inevitably leads to sponsor disillusionment and, ultimately, challenged projects (Figure 3).
Business Analysts aren’t brought into the project early enough As one of our interviewees said, “fundamentally a business case is a document that at a moment in time describes the benefits and costs that you think we will have for a particular [problem’s] solution.” But the problem has to be investigated before you can even have a business case. The organization must determine if there is value in making a change, and how the proposed change will align with larger objectives. Those are all requirements, yet several interviewees felt business stakeholders did not have this view. Understanding the requirements all the way from strategy to actual delivery is critical to the realization
performanceinstitute.org | 9
The Problem with Requirements: Why Is It Still a Problem?
FROM AN INTERVIEW:
I can give you an example [from] within our organization. They have a model where they are asking for 90 percent accuracy with project estimates prior to the Business Analysts even being assigned to the project. The Business Analysts haven’t even been involved in the project by that point and it’s prior to any involvement with the entire project team. Once Business Analysts get involved and derive all the requirements and things that need to happen at a detailed level, we find our initial estimates are nowhere close – it’s just what shouldn’t have happened. SDLC Core Team Member, large, research-driven healthcare company
Good business analysts need to • Have a set of techniques that they consistently use to ensure that they have a solid representation of their business systems and a consistent approach to requirements development • Have a strong, consistent, and repeatable review process—including peer reviews and input from stakeholders—for requirements documents to be signed off • Have the people-skills to engage all the stakeholder groups and to ensure an understanding of the real business problem (more on this important point more in a subsequent section) FROM AN INTERVIEW:
Business Analysts don’t have the right skills The history of who has been responsible for requirements is an interesting reflection of the perceived value and difficulty of creating good requirements. It’s progress to have a defined role, but it’s also true that having the title “Business Analyst” doesn’t guarantee that the individual with the title has the right skills—or enough skills—for difficult, complex projects. And having a certification in business analysis doesn’t mean that either, although it can be a good indication of experience. As one of our interviewees put it: “It doesn’t matter what you call me. It’s important that someone has the core skills; the harder skills of software development and the softer skills of teaming and personality traits.” FROM AN INTERVIEW:
What we’ve seen is that, at least from my own personal experience, a lot of Business Analysts are brought into business analysis without any real formal background in it. That’s certainly what happened to me. I came out of a consulting firm and they hired us to write software requirements and I had never written a requirements document in my life! The fact is a lot of my coworkers hadn’t really done it either. They were all people who perhaps understood how a business operated. A lot of them were the more analytically minded people in their department, but they were told “you know the business so you can write requirements.” And I think that that attitude is a big part of the problem. It’s the idea that requirements are simply a matter of documentation rather than a matter of problem solving. Chief Business Analyst, IIBA
10 |
I have a lot of people [who] are just coming in to capture requirements because that’s what they have to do for the project. [They have] never really done it before. They are not professional Business Analysts, so they don’t understand the methodologies, the tools that we have available to them, and the whole process of requirements management which starts [with] talking to your customers to get that information ... and then managing it through a process. They will go right to the most common form of requirements, which is just to write them down. They might not be testable or statements of fact among other things so it’s a matter of experience and overall training. SDLC Core Team Member, large, research-driven healthcare company
Further, a more highly skilled Business Analyst can strike a balance between what are the value adds to a project, and what are not. For example, does a small project need use cases, models, and everything else that may not add real value to it? A seasoned veteran will know the answer to that when they go into the project. From an interview:
People don’t take requirements seriously—they think anyone can do it. But requirements [are] not ... about going out and gathering them, it’s [about] the whole skill of eliciting them, and making sure they’re delivering the business needs in line with business objectives and policies, and business goals and strategy…. But even when requirements are well gathered, sometimes the people who are gathering requirements are not communicating with the people that are delivering the requirements.
The Problem with Requirements: Why Is It Still a Problem?
Figure 4: Who Does Requirements: A Capsule History 4
Examiner for the British Computing Society (BCS) for the Diploma in Business Analysis
As Edwards Deming said, it’s key to “get the very best help you can possibly find.” Business Analysts are not USED consistently and ARE NOT fully committed to the project This important point was discussed in a paper in the Harvard Business Review in 2011.5 The authors examined a very successful project and determined that of the seven factors critical to its success, two involved the composition of the team: • They assembled the right team, including experts from the company, outside, and vendors • They prevented turnover among team members The authors strongly suggested that having a skilled, consistent team was a key to reducing project risk. Communication, communication, communication Here’s the bottom line: for business analysts, soft skills are perhaps the most important traits needed for project success. Many other skills can be learned, but the ability to communicate, to analyze, to elicit, to ensure the right people are involved at the right time, those skills are difficult to teach. Some individuals have great people-skills, while others do not. As far back as 1997, Rob Thomsett noticed that many systems analysts were too solution-focused and lacked the appropriate interpersonal and communication skills required to build an open relationship with key stakeholders. With stakeholder involvement being such a key factor in project success, business analysts must above all have the people skills to ensure stakeholders are engaged, consulted, and understood.6 As discussed earlier, the GAO Report on
Information Technology cited the top critical factor predicting project success as having program officials “actively engaged with stakeholders.” 7 However, we believe there may be an additional, almost imperceptible problem at play. We describe this as a difference in the requirements paradigm. IT looks at requirements as an extension of code—the statements need to be explicit, structured, and technical in order to be useful. In contrast, business stakeholders express requirements in the vocabulary of their industry (insurance, defense, banking, etc.), and expect to be understood. This situation can result in a high potential for misunderstanding. From an interview:
A common cause of requirements problems you ... see is ... issues within the business itself. You find a way to write the requirements to be satisfactory for two different groups, but when it comes time to implement it, you can’t implement it in a way that leaves that vagueness in place. Part of our job is to force those kinds of issues to the surface, and actually resolve how the business is going to operate ... that includes understanding things [such as] where change is likely [to occur] so that systems can be developed in ways that can accommodate it. Chief Business Analyst, IIBA
4
This figure is based on information in an article by Rob Thomsett in American Programmer, April 1997
5
Our research uncovered key elements of the definition of a requirement: • To be a requirement, there is always a source. This source may be implicit or explicitly stated, but to be a requirement, there is always an originator. The source may be human or non-human. • To be a requirement, there is always an expressed need. The explicitness of the expression
Bent Flyvbjerg and Alexander Budzier, “Why Your IT Project May Be Riskier Than You Think”. Harvard Business Review, September 2011
6
Rob Thomsett, “It’s the Expectations, Stupid”. American Programmer, April 1997
7 US Government Accountability Office. Information Technology: OMB and Agencies Need to More Effectively Implement Major Initiative to Save Billions of Dollars. July 2013
performanceinstitute.org | 11
The Problem with Requirements: Why Is It Still a Problem?
of the need varies dramatically, but the concept of an expressed need is always part of a requirement. • The term ‘requirement’ is a noun. Requirements are objects rather than actions. Though these elements are accepted and may seem obvious, variations in each statement are fundamental contributors to the requirements paradigm breach. • In the case of a requirements source, mission and line of business stakeholders rely on experience and expertise to develop authoritative requirements. Technical solution developers assume that these requirements are correct because “they know their business.” However, this assumption is flawed. As organizations become more complex, business stakeholders often have a viewpoint that is either not complete or not current. • In the case of requirements explicitness, there is a difference in the expectation of explicitness. The paradigm of the organization is extremely implicit; terminology and phrasing between business/mission functions carry immense amounts of implied meaning which can include constraints and assumptions (i.e. organization, industry, legal) that are presumed to be understood by all parties. Conversely, the expectation of technical resources is highly explicit. If a requirements statement is not explicitly stated, then technical resources assume that the requirement does not exist. • The concept that requirements “are done” (an action that is completed) contrast with the view that requirements as objects that have some kind of useful life. Most organizations have a requirements phase in their solution development lifecycles. This phase reinforces the concept of requirements as an activity, versus requirements as a product. It’s up to the skills of a business analyst to sort all of these differences out. They must continually engage with business leads and stakeholders to ensure genuine alignment between business needs and the corresponding IT solutions being developed. They also need to understand not only the client’s conscious requirements (“we need to fix this problem”), but have the critical thinking skills to investigate and test their real requirements—the strategic intent of their processes and the rationale for improvement. This requires an intimate involvement with the stakeholder to tease out how the organization wants to do business in the future and what’s really important. It also requires reconciling different stakeholder perspectives to gain an accurate understanding of the true requirement.
The Contracting Process Prevents Effective Requirements Management
12 |
An interesting result of our survey and interview process was that it illuminated the differences between project success in the commercial world and in government. We found that an overriding issue with government projects involved the contracting process, which is very complex and can result in significant delays between the hen the request is made and when the product or service is actually contracted. The effect of this delay is that requirements information can become “OBE” (Overcome By Events), and the final solution might fall out of touch with the original business need. Government investment efforts can introduce a delay of 18–24 months between the time business needs were defined and when solution development begins. When the contracted product or service evolves quickly, such as is the case with IT related solutions, this time difference often means that requirements become “stale.” One thought leader involved in both large public and private organizations states that the issue comes down to a fundamental issue of trust. She offered a useful model for procuring new products or services with these features: • Think of how requirements should be packaged for acquisition very early—when business requirements are being defined • Requirements should specify what the government client wants in very explicit terms, but not how they should be built • Make the procurement packages as small as possible to test the vendor relationship • The acquisition strategy should include how these packages will fit together, including priority • The overall objective is to build trust “Trust” is the operative word. It can set an atmosphere of constructive collaboration, which helps incorporate rapid advances in technological innovation. The Association of Government Accountants (AGA) agrees with the concept of small acquisition packages. These packages should be “smaller than you ever thought would be possible” in order to keep complexity to a minimum. “On average, senior executives rotate out of their positions every 18 months. When a new executive takes charge, his or her priorities may be different from his or her predecessor, which can imperil a program.” Having small packages that can be developed within that 18-month time frame mitigates this risk. Because the investment management phase of a program can take time, most of our interview participants felt that it was critical that requirements information be
The Problem with Requirements: Why Is It Still a Problem?
stored in a relational repository. One of the thought leaders was emphatic on this point. “Requirements information can become useless within six months if it is stored as a series of documents in a SharePoint repository. The analyst must be able to assess traceability—this allows analysis of strategic alignment and potential impacts as business needs and technology changes. We recognize that requirements tools are absolutely essential to manage dynamic requirements information over time.”
treated as an asset. Part of our research was to look at what that might mean if requirements really were considered to be an asset of the organization in a financial sense. The first step was to understand the definition of an intangible asset. Intangible assets are “an identifiable non-monetary asset without physical substance” according to the International Financial Standards Board (IFSB).8 To meet the test, requirements must: • Provide future economic benefit
FROM AN INTERVIEW:
• Be measurable
One of the largest U.S. government agencies has commissioned a task force to address how to effectively keep up with the fast paced world of mobile technology. Specifically, the Department constantly tried to acquire mobile solutions, many of which were out of date by the time they were implemented. A recent directive to move to agile development sought to address the issue, but only seemed to make the problem worse; while solutions were produced more quickly, they continually failed to address the true business capability need. Not surprisingly, the inflexibility of requirements information was the cause. But the reasons may offer a surprise. First, an agile approach focused too much on technical needs (how the technology should work) rather than a detailed description of what it takes to address the business capability need. This agile focus ironically made the request too rigid, because it did not allow vendors to compete effectively based on innovation and how they could most effectively address the required capability. Line of business stakeholders often stopped being involved at a high level. When a program required a capability to be procured (bought versus made), stakeholders felt their obligation was fulfilled. They did not participate in explicitly describing what would be an acceptable solution for their business problem. This introduced an interpretation gap leaving the OAO in a position where they needed to “fish” for possible solutions from industry for what they thought the business might want. Contracting Director—Office of Acquisition Operations (OAO)
Requirements as Financial Assets to the Organization Some industry thought leaders have called for organizations to recognize that requirements should be
During our interviews, participants generally responded with a qualified agreement; acknowledging that some requirements would fit into this category, specifically citing the following types of requirements information as assets: • Reusable requirements information such as nonfunctional requirements and certain functionality that the organization feels must be consistent (i.e. the way a user would sign on to different systems within an organization). Storing functional definitions as reusable assets would save future costs to redefine the requirements. Requirements that would be fit for a specific purpose would not qualify. • Product and other business rules, especially those that describe a unique market advantage for the organization. Business rules can (and should) be reused for future projects and thus meet both tests. • Higher level requirements including business process documentation, system interface information, and organization capabilities. These requirements are typically represented as part of the business architecture and describe the way business activities deliver outcomes. Measurable reuse has been demonstrated most recently in an MIT Sloan Center for Information Systems Research (CISR) research brief that provides a guide for potential increases in revenue and net margin and therefore meets the intangible asset tests.9 The second step was to learn if financially evaluating requirements as assets is a good idea. From the government perspective, it did not seem to be relevant. As AGA stated during our interview, an agency could not reasonably be considered to be more valuable because its requirements are managed more effectively. However, for commercial organizations the concept would be viable if the organizational processes to manage requirements information (and assets in gen-
8 Definition of Intangible Assets are in IAS 38 provided by the International Accounting Standards Board (IASB), http:// www.ifrs.org/IFRSs/Documents/ English%20IAS%20and%20 IFRS%20PDFs%202012/ IAS%2038.pdf 9
Top Performing Firms are Better at Digital Reuse, Weill, Woerner & McDonald, Center for Information Systems Research, MIT Sloan School of Management, Volume X, Number 10, October 2010
performanceinstitute.org | 13
The Problem with Requirements: Why Is It Still a Problem?
eral) are more mature. In other words, if the practices that prevented the systemic requirements problem were in place and if those practices provided were shown to be sustainable, then the requirements information listed above could (and probably should) be valued as an enterprise asset.
Assuming that an organization uses projects as steps to achieve a strategic outcome, then when projects are abandoned, run over budget, or fail to deliver an acceptable solution for the underlying business need, those projects actually prevent executives from achieving what is expected of them.
It is important to note that none of the organizations interviewed currently considered their organizations to be ready to commit to this idea or to invest in making it a reality. One large pharmaceutical organization we interviewed put it bluntly—“if you value requirements information as an asset, you must also be prepared to consider it as a liability.”
Our research suggests that an organization-specific cost-benefit analysis, focused around the findings described in this research, is the best way to permanently address the persistence of poor requirements. This cost benefit analysis will provide the foundation for a business case that permanently addresses the requirements problem using project success and solution quality as the “yardsticks” for incremental improvement.
The Next Step: Making the Case for Change Most of those interviewed during our research pointed out that a business case for change would be needed to convince senior leadership to address the issues described, and to provide the rationale to implement the recommended practices. Though executives were generally sympathetic to the “requirements problem,” few would take time to make it a strategic initiative unless there was a solid, quantified cost-benefit analysis that demonstrated how the goals of the organization would be achieved more effectively through better requirements practices. The Performance Institute has developed a cost-benefit approach that uses cost avoidance as the means by which organizations can identify hard cost savings associated with the implementation of requirements definition and management best practice. At a high level, this approach: • Looks at the current program portfolio to learn how well it is performing • Investigates common contributing factors to projects that have not been successful • Understands how much the risk of the contributing factor is affected by the poor application of requirements practices • Determines the quantified cost avoidance which can be realized as requirements practices are strengthened within the organization This approach leverages industry benchmarks, but should be optimized based on an assessment of each individual organization. In this way the quantified cost reductions are specific for that organization.
14 |
The Problem with Requirements: Why Is It Still a Problem?
summary The conclusions we reached from our surveys, research, and interviews are summarized below:
Findings
recommended practices
Too many projects focus on the wrong things, which can result in:
• Focus on understanding business needs to ensure deliverables will meet a strategic, business need
• Deliverables that do not meet a strategic, business need
• Utilize requirements definitions and management tools that help maintain requirements alignment throughout the entire solution delivery process, starting before the business case if possible
• Focusing on budget and schedule to the exclusion of more important factors • Discarding requirements when the project is “done,” thereby losing important assets that would help jump-start subsequent projects Not enough time is allocated to requirements
• Create a specific plan for requirements gathering, using experience and project complexity as a guide • Create the requirements plan very early, before the projects begin. Ideally, the plan would guide resource allocation in the business case
The wrong people are responsible for requirements
• Vendors should never create system or solution requirements • Being a “business analyst” isn’t enough. those performing business analysis need to: - Be brought into the project right from the start - Have excellent analytical and “people skills” - Be fully committed to the project throughout its life
Business Analysts often do not recognize the importance of engaging stakeholders and project team members on a consistent basis
• Elevate the role of requirements definition in the organization, investing in experienced resources who can act as liaisons between IT and stakeholders at a high level in the organization • Consider resources with an operations background rather than a technical background for the business analyst role • An independent viewpoint is vital
Government contracting requests are too large to be successful
• Elaborate requirements to a level that describes in detail what is needed to support the business or mission need, but leave how it will be done to potential vendors • Contract requirements should be in packages that are very small in size—just large enough to produce a measurable outcome of business value to the business stakeholder • Requirements information is constantly changing; management methods must be dynamic
performanceinstitute.org | 15
The Problem with Requirements: Why Is It Still a Problem?
References Bloch, M., Blumberg, S., & Laartz, J. (2012). Delivering large-scale IT projects on time, on budget, and on value. McKinsey on Business Technology, Number 27, pp. 1-7. Blueprint. (May 30, 2013). The Rework Tax: Reducing Software Development Rework by Improving Requirements. http://www.blueprintsys.com/asset/the-rework-tax-reducing-software-development-rework-by-improving-requirements/
Budzier, A. & Flyvbjerg, B. (August 2011). Double Whammy- How ICT Projects are Fooled by Randomness and Screwed by Political Intent. n.d. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2238057
Ciralskyk, A. (September 16, 2013). Will It Fly? Vanity Fair, Retrieved September 23, 2013. http://www.vanityfair. com/politics/2013/09/joint-strike-fighter-lockheed-martin
Dronzek, L. & Lanowitz, T. (August 12, 2010). ROI Scenarios: Invest for Success – Adopt Requirements Definition Solutions to Deliver Strategic Value and Economic Efficiencies. Nevada: Voke, Inc.
Flyvbjerg, B. & Budzier, A. (September 2011). Why Your IT Project May Be Riskier Than You Think. Harvard Business Review. Retrieved April 4, 2013, http://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think/ ar/pr
Grant, T. with Murphy P. & Anderson, A. (December 7, 2011). High-Value Requirements Are Changing App Dev & Delivery: Requirements Go From Unfortunate Necessity to Strategic Asset. Massachusetts: Forrester Research, Inc.
International Accounting Standards Board. Definition of Intangible Assets are in IAS 38 provided by the International Accounting Standards Board [IASB], http://www.ifrs.org/IFRSs/Documents/English%20IAS%20 and%20IFRS%20PDFs%202012/IAS%2038.pdf Jackson, D. (June 11, 2013). Develop Requirements Analysts. www.projectsatwork.com Jones, C. (May 1, 2012). Software Quality in 2012: A Survey of the State of the Art. PowerPoint Presentation, Slides 10-11, 17, 25-26, 40, 42-44. Namcook Analytics, LLC.
Lindbergh, L. & Hass, K.B. (June 2011). The State of Business Analysis Practices in Organizations. Research Report. Lorius, LLC and Kathleen Hass & Associates, Inc. Robertson, J. & Robertson, S. (March 8, 2011). Ten Tests for Requirements. March Newsletter, Volere Requirements Resources.
Sessions, R. (November 8, 2009). The IT Complexity Crisis: Danger and Opportunity. Houston, Texas: Object Watch Inc.
Simms, J. (June 26, 2007). Why Projects Fail: Part Two, Poor Business Requirements. CIO. Retrieved September 16, 2013. http://www.cio.com.au/article/204848/why_projects_fail_part_two_poor_business_requirements/
Thomsett, R. (April 1997). It’s the Expectations, Stupid! American Programmer, 10 (4), pp. 30-33. United States. Government Accountability Office. House of Representatives. Subcommittee on Government Operations. Committee on Oversight and Government Reform. (July 25, 2013). Information Technology: OMB and Agencies Need to More Effectively Implement Major Initiatives to Save Billions of Dollars by D.A. Powner. (GAO-13-796T). p. 3, 15. www.gao.gov
Weill, Woerner & McDonald (October 2010). Top Performing Firms are Better at Digital Reuse, Center for Information Systems Research, MIT Sloan School of Management, Volume X, Number 10. Wiegers, K. E. (1999). Writing Quality Requirements. n.d., http://www.processimpact.com/articles/qualreqs.html
16 |
The Problem with Requirements: Why Is It Still a Problem?
survey questions The Performance Institute Requirements Survey
• 71% - 89%
1. Which of the following best describes the principal industry / sector of your organization?
• 100% (all)
Answer Options • Commercial - Manufacturing
• 90% - 99% 4. Please select all of the statements you generally believe to be challenges for your agency: Answer Options
• Commercial - Service
• Our capability requests are often unclear or ambiguous
• Commercial - Software
• Capability requests do not reflect true understanding of the business problem
• Government - Federal • Government - State or Local
• Impacts to existing systems and business processes are not fully considered
• Not for Profit • Other (please specify)
• Reuse of existing systems, data, and capabilities are not fully considered
• Non material (materiel) options are not explored before a capability request is made
2. Of the large, complex projects, what percentage launched but were challenged? (“Challenged” is defined as any of the following: 1) more than two months after projected delivery date; 2) over 15% beyond budget; 3) the solution contained significant enough defects that it had to be re-worked to satisfy its stakeholders.) Answer Options • 0% (none)
• There are no clear links to agency strategy or mission objectives • Our agency has none of the challenges listed above 5. Please evaluate the following statement:
• 11% - 30%
“Business requirements defined in the acquisition can be traced to all corresponding functional and technical requirements documented in the project lifecycle.” At your agency this is:
• 31% - 70%
Answer Options
• 1% - 10%
• 71% - 89%
• Always true
• 90% - 99%
• Generally true
• 100% (all)
• Sometimes true
• Occasionally true
3. Of the large, complex projects, what percentage were abandoned or failed post launch due to project problems or inaccurate understanding of business need?
• Never true
• 0% (none)
6. After a contract has been awarded to deliver the requested capability, how much influence does the successful vendor generally have on solution requirements?
• 1% - 10%
Answer Options
Answer Options
• 11% - 30% • 31% - 70%
• Complete influence, agency may be informed • Significant influence, agency retains final accep-
performanceinstitute.org | 17
The Problem with Requirements: Why Is It Still a Problem?
• Too much
tance authority • Moderate influence, collaborates with agency on impact and business process impacts • Low influence, responds to agency direction with any configuration constraints • No influence, vendor role is not involved in requirements – only capability implementation
• Far too much 11. How did the actual length of the project typically compare to Project Management projections? Answer Options
• shorter by more than 10% • within +/- 10%
7. As a percent of overall time needed to develop a solution (i.e., needs analysis to implementation / deployment), what percentage would you estimate was devoted to requirements definition and analysis in total?
• between 11% and 25% longer • between 26% and 50% longer • > 50% longer
8. When was the scope for these solution efforts generally approved (i.e., signed off, formalized)? (Refer to the figure above and select closest match)
12. What are top 3 ways you spent gathering requirements information?
Answer Options
Answer Options • Read and analyze existing documentation
• When the need was analyzed
• Conduct in-person interviews
• When the project was initiated and planned
• Shadow subject matter experts/users
• When the requirements were gathered and documented
• Distribute questionnaires
• When the solution was architected and designed
• Conduct facilitated meetings/workshops
• When the solution was developed
• Conduct JAD sessions
• When the solution was tested
• Build prototypes and/or mockups
• When the solution was deployed
• Build visual models
• I do not know
• Describe user stories and/or use cases • Other (describe and give Rank (1,2 or 3))
9. Who was generally accountable for setting the scope for these solution efforts? Answer Options • Project Manager
13. What is the primary way you captured and managed requirements information ? Answer Options
• Business Analyst
• Text documents and spreadsheets
• Architect
• Requirements definition and management software
• I do not know
• I do not know
• Other Stakeholder (please specify)
10. How would you characterize the length of time generally allotted to requirements gathering? Answer Options • Far too little • Too little • About right
18 | METHODS AND MODELS
• Other (please specify)
14. How often were visual models used to define your requirements? Answer Options • We consistently used visual models • We often used visual models
The Problem with Requirements: Why Is It Still a Problem?
• We sometimes used visual models
• About half the time
• We rarely used visual models
• More than half the time
• We never used visual models
• Almost always
• I do not know 15. Before accepting requirements information (sign off), on average how well do you think the solution stakeholders understood the requirements information they were accepting?
19. Were the requirements useful after these project solutions were developed? If yes, please explain. Answer Options • Yes • No
Answer Options
• If Yes, Please Explain
• Understood all of the requirements well • Understood most requirements well • Understood some requirements well • Understood few requirements well • Understood none of the requirements well 16. On average, how completely do you feel that business stakeholders understood the likely impacts of the requested solution on the business (e.g., on business process, operating costs, etc.)?
20. Considering the subset of these projects in which you were directly involved, when did you typically realize that the solution simply didn’t reflect what was truly needed? Select the Phase that most closely matches your answer using the figure above: Answer Options • When the need was analyzed
Answer Options
• When the project was initiated and planned • When the requirements were gathered and documented
• Understood all of the impacts well • Understood most impacts well
• When the solution was architected and designed
• Understood some impacts well
• When the solution was developed
• Understood few impacts well
• When the solution was tested
• Understood none of the impacts well
• When the solution was deployed
17. Once the solution began development, how often was it found that an important stakeholder was not included in the project from the start? Answer Options
• I was not involved in any challenged, failed or abandoned projects • I do not know
• Less than half the time
21. Of the large, complex projects, what percentage were successful, that is, considered neither challenged nor abandoned or failed post launch?
• About half the time
Answer Options
• Almost never
• More than half the time
• 0% (none)
• Almost always
• 1% - 10%
• 11% - 30%
18. How often did you feel the technical approach drove the solution (versus the business approach)?
• 31% - 70%
Answer Options
• 90% - 99%
• 71% - 89% • 100% (all)
• Almost never • Less than half the time
performanceinstitute.org | 19
The Problem with Requirements: Why Is It Still a Problem?
22. As a percent of overall time needed to develop a solution (i.e., need analysis to implementation / deployment), what percentage would you estimate was devoted to requirements definition and analysis in total? 23. When was the scope for these solution efforts generally approved (i.e., signed off, formalized)? (Refer to the figure above and select closest match)
• between 11% and 25% longer • between 26% and 50% longer • > 50% longer 27. What are top 3 ways you spent gathering requirements information? Answer Options • Read and analyze existing documentation
Answer Options
• Conduct in-person interviews
• When the need was analyzed
• Shadow subject matter experts/users
• When the project was initiated and planned
• Distribute questionnaires
• When the requirements were gathered and documented
• Conduct facilitated meetings/workshops
• When the solution was architected and designed
• Conduct JAD sessions
• When the solution was developed
• Build prototypes and/or mockups
• When the solution was tested
• Build visual models
• When the solution was deployed
• Describe user stories and/or use cases
• I do not know
• Other (describe and give Rank (1,2 or 3))
24. Who was generally accountable for setting the scope for these solution efforts?
28. What is the primary way you captured and managed requirements information ?
Answer Options
Answer Options
• Project Manager
• Text documents and spreadsheets
• Business Analyst
• Requirements definition and management software
• Architect
• I don’t know
• I do not know • Other Stakeholder (please specify) 25. How would you characterize the length of time generally allotted to requirements gathering? Answer Options
• Other (please specify) 29. How often were visual models used to define your requirements? Answer Options • We always used visual models
• Far too little
• We often used visual models
• Too little
• We sometimes used visual models
• About right
• We rarely used visual models
• Too much
• We never used visual models
• Far too much
26. How did the actual length of the project typically compare to Project Management projections? Answer Options • shorter by more than 10% • within +/- 10% 20 |
• I do not know
30. Before accepting requirements information (sign off), on average how well do you think the solution stakeholders understood the requirements information they were accepting?
The Problem with Requirements: Why Is It Still a Problem?
• No
Answer Options
• Comment (if “Yes” answer)
• Understood all of the requirements well • Understood most requirements well
• Understood some requirements well • Understood few requirements well
35. When do you think your organization considers requirements to be complete or finished?
• Understood none of the requirements well
Answer Options
• Before the project receives funding
31. On average, how completely do you feel that business stakeholders understood the likely impacts of the requested solution on the business (e.g., on business process, operating costs, etc.)?
• When the requirements phase of the project lifecycle is complete • When testing is complete • It is unclear when the requirements process is finished
Answer Options
• Never - requirements are continuously managed
• Understood all of the impacts well
• Other (please specify)
• Understood most impacts well • Understood some impacts well
• Understood few impacts well
36. Does your organization typically maintain requirements information after solutions are developed? If no, why not?
• Understood none of the impacts well
Answer Options
32. Once the solution began development, how often was it found that an important stakeholder was not included in the project from the start?
• Yes
Answer Options
• If No, Please Explain.
• No
• Almost never
• Less than half the time • About half the time
37. On these projects, your typical role is best described as:
• More than half the time
Answer Options
• Almost always
• Solution Stakeholder
• Project or Program Manager
33. How often did you feel the technical approach drove the solution (versus the business approach)?
• Business Analyst
Answer Options
• Technology Architect
• Business Architect • Development Team/Scrum Team Member
• Almost never
• Other (please specify)
• Less than half the time • About half the time
• More than half the time
38. How long have you been in this role?
• Almost always
Answer Options
• < 1 year
34. Were the requirements useful after these project solutions were developed? If yes, please explain. Answer Options • Yes (please explain in Comment field)
• 1 to 3 years • 4 - 6 years • 7 - 9 years • > 9 years performanceinstitute.org | 21
The Problem with Requirements: Why Is It Still a Problem?
39. How long have you been with your organization? Answer Options • < 1 year • 1 to 3 years • 4 - 6 years • 7 - 9 years • > 9 years 40. Which development methodology is used within your organization? Answer Options • Agile • Waterfall • Agile or Waterfall based on project • Blend of Agile and Waterfall (hybrid) • I don’t know 41. Please enter your email address below for a complementary copy of the research findings.
22 |
The Problem with Requirements: Why Is It Still a Problem?
Interview Subjects We interviewed thought leaders from several industries and think tanks, including: • Head of strategy development and enterprise architecture for the IIBA. The role includes directing the development of the business analysis body of knowledge (IIBA, BABOK®). • Project Manager and head of the SDLC Core Team developing the system methodology lifecycle, system lifecycle methodologies, and practical applications of project management at a large, research-driven healthcare company • Associate Director of Business and Technical Analysis and a practicing business analyst at a large, research-driven healthcare company • Vice President of Technology and CIO at a large four-campus university. Previously, head of IT at a large life insurance and financial services company. • Two thought leaders at the Association of Government Accountants (AGA), a member organization for financial professionals in government. The AGA leads and encourages change to benefit the accounting field and all citizens. • Former president and CEO of a small consulting organization, focused for over 20 years on business analysis and requirements excellence. • Thought leader working in the public sector for nearly 24 years in business analysis, training, and consulting. Also serves as an examiner for the British Computing Society (BCS) for the diploma in business analysis and as a subject matter expert for the solutions development and diploma syllabus. • Thought leader who has been a pioneer in the delivery of requirements solutions. Her firm has worked for over 20 years with public and commercial organizations, successfully delivering requirements for complex projects as well as creating sustainable requirements centers of excellence programs.
performanceinstitute.org | 23
The Problem with Requirements: Why Is It Still a Problem?
Interview questions All interview subjects were asked the same series of base questions, although allowed to go in multiple directions depending on the answers, interest levels, and experience of the subjects. The base questions are as follows: • Please describe your role and the length of time you’ve been performing that role. • Do you agree that there is a persistent, systemic problem with the way organizations—large and small—manage requirements, and that these problems are a root cause for project failure? • Why do you think the requirements problem hasn’t been solved by now? • In what areas, if any, have you seen any improvement? • Do you see a difference with regard to the requirements problem, between projects that have an IT based or highly technical solution, and projects that are predominantly production solutions or business improvements that don’t have an IT solution at their core? Do you think the requirements problem is handled any better by one type of project, or do you think there is a problem across the board? • Have you seen any organizations that have made significant progress in the quality of the way they manage requirements? • We’ve heard two different viewpoints in our research. The first viewpoint is that requirements take too long. Too much time is spent gathering and analyzing requirements, only to have them change or to have some unforeseen issue come up causing the project to run behind as a result. The other viewpoint is that not enough time is spent focusing on requirements. The sense is that since a thorough and rigorous requirements definition was not executed in the first place, project outcomes have been adversely affected because requirements issues were not discovered early enough to be remedied effectively. Do you have any viewpoint as to which perspective is more credible? • When does the requirements effort begin and end in your organization? When should it begin and end? • Do you think the software development methodology—whether it is agile methods or waterfall—affects the quality of requirements? Is one approach preferable, or should good requirements be agnostic of the software development methodology? • An intangible asset doesn’t have a physical 24 |
presence, but it either provides future economic benefits that can be attributed to it, or it has a cost that can be measured. Can requirements information be considered an intangible asset of an organization? Would you categorize requirements information, or the tools by which requirements information can be held to be assets of an organization? • If an organization decided to consider requirements information at the enterprise level as an intangible asset with a prescribed value, in your opinion, would that have an effect on the persistence of the requirements problem since the requirements would have a monetary value to the organization? Would a possible solution to the requirements problem be funding and getting the recognition that the problem deserves?
Copyright 2014 by The Performance Institute
