Software Estimation Process Overview

Software Estimation Process

Contents The Purpose of Estimation ..................................................................................... 4 A Historical Perspective ......................................................................................... 4 Estimating Principals............................................................................................. 4 Common Misconceptions ....................................................................................... 4 Estimation as a Business Simulation ....................................................................... 5 Purposes of Estimation.......................................................................................... 5 Estimate Process Flow........................................................................................... 6 A Process for Estimating ........................................................................................ 8 Step 1: Estimate Planning ..................................................................................... 8 Step 2: Estimate Project Size................................................................................. 9 Step 3: Create Estimate Scenarios ....................................................................... 10 Step 4: Estimate Reconciliation ............................................................................ 11 Step 5: Present Estimates ................................................................................... 11 Company Estimation Management Report ............................................................. 12 Project Tracking and Adaptive Forecasting .......................................................... 14 Ongoing Metrics Collection .................................................................................. 14 Variance Analysis ............................................................................................... 15 Adaptive Forecasting .......................................................................................... 16 Benchmarking and Historical Trends .................................................................... 18 Historical Metrics Collection ................................................................................. 18 Baselining and Benchmarking .............................................................................. 20 Trend Analysis ................................................................................................... 22

www.qsm.com

2

Software Estimation Process

Figures Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure Figure

1. Estimation Overview ..................................................................................... 5 2. Four Dependent Activities of the Estimation Process .......................................... 7 3. Five Steps in the Estimation Process ............................................................... 8 4. Estimation and Company Corporate Enterprise Methodology ............................ 10 5. Company Estimation Management Report Example ......................................... 14 6. Project Control Process ................................................................................ 15 7. Example Project Dashboard & Snapshot Summary .......................................... 16 8. Example of a Re-forecasted Project ............................................................... 17 9. Example of Completed Project Staff & Phase Reconstruction ........................... 19 10. Example of an Internal Baseline .................................................................. 20 11. Example of a Comparison to External Benchmarks ........................................ 21 12. Example of Long Term Improvement Trends ................................................ 22

www.qsm.com

3

Software Estimation Process

The Purpose of Estimation A Historical Perspective The world of software estimating is filled with horror stories of projects that are years late and millions of dollars over budget. It is not uncommon when estimating the cost and schedule on software projects for companies to be off by as much as 100%. A recent Standish Group study reported the following statistics: • • • • •

80% to 90% of software does not meet performance goals 80% of systems are delivered late and over budget 40% of developments fail or are abandoned Less than 25% of systems properly integrate business and technology objectives Only 10% to 20% meet their success criteria

There is an alternative to this sad state of affairs. Through measurement and estimation practices based on empirical performance, accuracy can be improved dramatically. Some companies can estimate project cost and schedules to within 97% of actuals. In a recent study of 564 recently completed IT projects QSM found that the best projects completed nearly 3.5 times faster and 7.5 times less expensively that the worst ones. In analyzing the reasons for this vast difference in performance QSM found that the most important factor separating the best projects from the worst was the ability to control changing requirements. Next on the list were skilled people with substantial domain knowledge and, after that were development tools.

Estimating Principals Here are some good general principals to keep in mind with respect to implementing an estimating process in an organization. •

Begin simply. Start small and build on success. -

•

The estimation process is part of the overall metrics processes and depends on them. -

•

When implementing a new process it is good to start out with just the basics. With regard to estimating we should try to streamline the process to require the minimum information possible to produce defensible estimates. As the process is rolled out and we achieve some success the process can be made more robust driven by feedback from the organization.

Reliable estimates require sound sizing and productivity assumptions. These are best derived from empirical data. Therefore an active metrics collection process is critical to support the estimation process. Without it we are simply guessing and can expect similar results to that mentioned above in the Standish Group survey.

Keep the estimating function independent. -

The purpose of estimation is to honestly assess the schedule and cost to deliver a set of requirements at a demonstrated level of capability. Keeping the estimation function separate in the chain of command ensures that we provide an honest evaluation rather than a political one.

Common Misconceptions There are a number of popular misconceptions about the purpose of estimation. The most notable with regard to estimation are: •

Estimation is the same as project planning and commitment. -

•

Estimation is the technical process of evaluating all possible scenarios to meet a project’s goals and assessing the risk associated with each. Planning and commitment are based on business decisions around how much risk we are willing to accept on a given business opportunity.

Project schedule and cost/effort are interchangeable. -

They are in fact intimately tied together. There are diminishing returns on the amount schedule compression that can be achieved by adding more staff to a project to meet a tight schedule deadline.

www.qsm.com

4

Software Estimation Process There is also a very large penalty to be paid in increased cost and reduced reliability by pursuing this strategy.

•

It is always possible to accurately estimate cost and schedule. -

The accuracy of an estimate is directly tied to the quality of the assumptions on which it is based. Project size and productivity can only be approximated prior to project completion. Basing these assumptions on empirical data can substantially increase the accuracy of these assumptions.

Estimation as a Business Simulation Estimation can be viewed as a business simulation. We are not allowed to, or don’t have the resources necessary to run the same project a number of different ways to see what might happen. There are however, good statistical methods and estimation processes that when backed with appropriate tools, can simulate what will most likely happen when we actually run a project. See figure below.

Requirements

Σ= Total Size

Process Productivity

Resources & Constraints

Software Elements

Initial Estimate

Tasks, Effort Commitment

Alternative Scenarios

Figure 1 Estimation Overview

Purposes of Estimation Estimation serves two purposes: 1. Provide sufficient business knowledge to allow the appropriate level of commitment www.qsm.com

5

Software Estimation Process This function is “upward directed” toward senior decision makers in the organization. The purpose of the estimation process is to develop sufficient business knowledge to expedite the decisionmaking process. Specifically, it must provide enough information to allow the authorizing management to decide whether or not to continue with the project, and what level of resources to commit to it. 2. Provide the basis for project planning This function is “downward directed” toward the project and technical management of the project. Its purpose is to define the detailed tasks to be accomplished while considering the overall guiding constraints for the project. Typical constraints include the project’s schedule, staffing profile, and cost.

Estimate Process Flow The overall estimation process flow is broken into four highly dependent activities, see figure below: •

Gather Estimate Assumptions — extract the relevant information from the available sources of data relating to the customer, the system being built, the environment, and the project. o Size Assumptions: Specifically we need to quantify the functionality that will be designed, developed, integrated and tested prior to delivery to the customer. This is generally known as the software size. It is usually measured early in the project lifecycle at a fairly high level of abstraction in units like high level business requirements. Later in the project these are decomposed to lower level measures like technical requirements and use cases. Ultimately these requirements will be mapped into physical construction components like screens, reports, programs, etc. These size measures should be expressed as a range of values. The size of the range should be consistent with the level of confidence of our size estimates. Usually a three point range is used where low, most likely and high values are provided. These can then be weighted and averaged to produce an expected size where the weight and any skew in the distribution can be accounted for. o Productivity Assumptions: The other major assumption to support our estimate is a measure of how productive the development team will be in implementing the customer requirements. The best place to get this information is to look at relevant project history. A development organization’s own history will contain all the uniquenesses pertaining to its development environment including the skills of its people, the technical complexity of its products and the tools and methods it is using. In some cases when no relevant history is available industry data can be used. As with the sizing information a range should be associated with our expected level of productivity. The more we drive this assumption from real history the smaller the uncertainty range will be. When industry data is used as the basis for this assumption the wider this range should be because it is unclear how similar or different we may be from the industry norms.

•

Estimation — turn this information into the usable commitment and planning information of expected schedule, effort/cost, quality and staffing. If there are constraints on schedule, resources or budget, these should be explicitly identified. Risk can be assessed by mapping through input uncertainties on size and productivity to output uncertainties on schedule and cost. These output uncertainties are expressed in terms of the probability of meeting the project constraints/goals. The estimate output will be in the form of a set of possible scenarios with their associated risks.

•

Commitment — this process selects one of the possible scenarios, makes the commitment to the customer, and allocates the resources necessary to be successful.

•

Project Planning — this process converts the selected scenario, the associated risk, and the allocated resources to produce a viable execution plan that will use the resources to produce the system within the committed time while managing the identified risk.

www.qsm.com

6

Software Estimation Process

Customer Expectations

Sizing:

Estimation:

Commitment:

The sizing activity determines the size and scope of the project in some quantifiable unit usable by the estimation process.

The estimation process converts the size and productivity into a set of likely project results.

The commitment process selects the optimal solution from the estimation process and provides the resources to make it happen.

Productivity: This should be determined based on empirical performance derived from historical data.

System Spec

Variance in the input values will cause variance in the estimate result, This allows for a assessment of risk of meeting project goals.

Target & Development Environment

The input variances are used to calculate the range of possible results and the risk associated with each result.

Resources Approved Schedule Effort/Cost

Size & Productivity

Quality/ Reliability

Variance/ Uncertainty

Staffing

Goals & Constraints

Risk

Project Plan: The project planning process uses the approved plan to create a detailed execution plan to deliver on the commitment and manage risk.

Execution Plan

Team Experience

Figure 2 Four Dependent Activities of the Estimation Process

www.qsm.com

7

Software Estimation Process

A Process for Estimating There are five steps in the estimation process. See figure below.

Estimation Process Steps Estimate Planning Estimate Project Size Create Estimates Reconcile Estimates Present Estimates

Accepted

Rejected Figure 3 Five Steps in the Estimation Process

Step 1: Estimate Planning Estimate planning addresses two issues. The first deals with the organizational structure to support estimation and specifies: •

Centralized or de-centralized estimation. Will a few persons perform estimates or many?

•

Data collection and storage. How will project history be collected and stored to support estimation and data analysis?

•

Standards and templates. How will estimates be requested? What presentation templates will be used? What naming conventions will be used?

•

Linkage to other processes. How does estimation interact with time tracking, project start-up, and project close-down? Are triggers to invoke the estimation process embedded in them?

The organizational aspects of estimate planning addresses decisions about who, what, when, why, how, and how much. It provides the framework within which individual estimates are conducted. The second issue of estimate planning addresses is determining how individual estimates are conducted. As a rule, the effort dedicated to estimating a project should be proportional to its value and risk. Individual estimate planning determines: •

Timeframe for estimate. When does the estimate need to be completed?

•

Roles and responsibilities. Who will size the project, estimate it, and present the results?

www.qsm.com

8

Software Estimation Process •

Deliverables. What outputs from the estimate will be presented, and to whom?

Step 2: Estimate Project Size Estimating the size of a software project is often the most difficult and time consuming step in the estimation process. Project size is a measure of the functionality that a software project will create or modify. It is emphatically not the cost nor the amount of effort required to complete this functionality. When estimating the size of software projects there are three principals that should be observed. •

Determine what sizing artifacts are available. The sizing metrics used will depend both on organizational standards and where in the development lifecycle the estimate is being generated. Early in the lifecycle only abstract measures such as high level requirements are available. Later, more concrete measures such as screens, reports, and database tables can be used. Concrete measures tend to produce more accurate size measures for estimating since they are more closely aligned with what the developers must create.

•

Use multiple sizing techniques. If the size estimates converge, it increases your confidence in the accuracy of the estimates. If not, this is a good indicator that more work remains to be done.

•

Present size as a range. A size estimate is just that: an estimate. By presenting the size estimate as a range from best case, through most-likely, to worst-case scenario you provide a range of possibilities which permits the project estimator to model the risk involved with an estimate and account for it.

Company Implementation COMPANY has identified three places in the lifecycle where estimates should be created. The artifacts available for sizing differ. Estimate input forms are provided in the appendix that will capture the relevant sizing artifacts at each estimation point. •

End of Concept (Milestone 1 – High Level Requirements Review). High level requirements are the sizing measure that is available. Requirements are counted and extrapolated to implementation units based on an average gearing factor (IU’s per HL Rqmt). Requirements can also be weighted according to complexity if sufficient details are known. o o

•

Mid-Iteration Zero (Milestone 2 – Iteration Plan Complete). Mid-Iteration size estimates are based on detailed requirements in the requirements document and stories from the product architecture documents. The stories are counted and extrapolated to implementation units based on an average gearing factor (IU’s per story). Stories can also be weighted according to complexity if sufficient details are known. o o

•

Size Artifact – High Level requirements Document Sources of information – Project Business case Document

Size Artifact – Stories broken out by complexity (complexity bins, tee shirt sizing) Sources of information –PM Business Lead, Development Lead, Infrastructure Lead, Test Lead, Architecture Lead, CM Lead

End of Each Iteration (Milestones 3-N – Milestone 1-N Complete). The end elaboration size estimates are based on refined used case definition and identified construction components. Examples of construction components are screens, reports, classes, programs, etc… o o

Size Artifact – Refined Iteration Plan Document Sources of information –PM Business Lead, Development Lead, Infrastructure Lead, Test Lead, Architecture Lead, CM Lead

www.qsm.com

9

Software Estimation Process

Figure 4. Estimation and Company Corporate Enterprise Methodology

Step 3: Create Estimate Scenarios SLIM-Estimate® The SLIM-Estimate® tool is the industry leading estimation tool. It uses a probabilistic approach to estimation. SLIM-Estimate® can create a number of quite different “scenarios” as potential project solutions given a set of inputs. Each of these solutions may have varying levels of schedule, staffing, effort, quality, and risk. The tool allows the very rapid creation of multiple scenarios at different levels of scope and risk exposure. SLIM-Estimate® can also output comprehensive analyses of the various scenarios for consideration by the commitment process. One of SLIM-Estimate’s® major strengths is the calculation of risk in each of the major dimensions. This can be used to determine the viability of a given project solution scenario by comparing the risk-reward balance. The steps followed in creating estimate scenarios will depend on whether or not the estimate can reasonably meet the project’s time and cost constraints. •

Determine size based on artifacts available at the time the estimate is performed. See Step 2 above for a review of the sizing process.

•

Determine productivity to use. Company history will be used to support productivity input to the estimate. Productivity is based on size, schedule and effort of relevant completed company project data. QSM industry database can be used to supplement productivity information when relevant company data is not available

•

Consider project schedule and budget goals when generating estimate scenarios.

If the estimate fits within the schedule and budget goals no other scenarios are needed. When the schedule and cost goals cannot be met, the following scenarios will be generated:

www.qsm.com

10

Software Estimation Process 1. Reduce scope to meet schedule and cost goals 2. Increase staffing to meet schedule  Decreases probability of meeting cost and lowers reliability 3. Negotiate for more schedule and budget 4. Examine productivity required to meet goals  Usually not an realistic option in the short term  Should be compared with COMPANY’s demonstrated productivity to identify risk associated with adopting this plan 5. Combination of the above

Step 4: Estimate Reconciliation Estimate reconciliation is concurrent with Step 3, Create Estimate Scenarios. Each scenario is viewed from two perspectives. First, is it plausible? Do company historical data for time, effort, quality, and cost to support the scenario? Second, does the scenario meet the project’s operational constraints? The reconciliation process works like this. •

A recommended estimate scenario will be selected in accordance with project objectives and priorities

•

Other estimates (not produced by SLIM-Estimate) will be compared to the recommended estimate and reconciled with respect to schedule, effort & productivity o o

If estimates converge, they increase our confidence in them If they diverge above an acceptable threshold, investigate, determine why, and make adjustments

Step 5: Present Estimates Estimate scenarios are presented to the appropriate decision makers •

Assemble estimate package for presentation. This includes o Assumptions and constraints o Sizing information o Supporting documentation o Estimate report(s): electronic and/or paper

•

Present only relevant alternatives

•

Anticipate and prepare for questions

•

Address strengths and risks of each alternative

If an estimate scenario is approved, the estimation process is complete and the project progresses to project planning and tracking. Alternatively, based on the information presented, leadership may decide to not pursue the project. A third alternative is to return to the estimation process to generate additional scenarios. There are two possible places to re-enter the estimation process:

www.qsm.com

11

Software Estimation Process •

Create Estimate scenarios. If no estimate scenario is approved but alternative solutions are possible within the project constraints this becomes the re-entry point. Additional scenarios are created and the process is followed to its conclusion.

•

Estimate Project Size. Frequently, no solution will realistically meet the project constraints. If this is the case it will be necessary to determine if sufficient functionality can be removed from the project to meet its constraints while still meeting the project’s business objectives.

Company Estimation Management Report This report summarizes the estimation activity in a form that is easily processed by the authorizing management. The key components are: Summary of Results This includes the desired constraints, the recommended work plan, and the recommended commitment plan (the work plan plus risk management contingency), in the following areas: • Schedule • Effort • Projected cost (effort times staff labor rate) • Peak staff • Projected reliability (Mean Time to Defect) • System size (expressed in IU equivalents) Key Supporting Screen Shots Between one and three key descriptive screen shots that describe the content of the estimation summary should be included. Typical examples include: • Staffing profile—showing rate of staff addition • Estimate comparison • History comparison • Key factor interrelationship • Total risk summary • Solution probability profile • Alternative tool support Estimate Analysis This section contains single sentence key considerations in the estimate solution. Solutions are graded as advantages or disadvantages of the solution. Advantages are bulleted with a green upward arrow () and disadvantages with a red downward arrow (). Pertinent factors that are neither advantages nor disadvantages are bulleted with a black dash mark (-). Typical analysis elements will include: • Particularly high/low risk areas • Combined risk considerations (product of all risk factors) • Major assumptions • Reduction in scope • Particularly high or low values of staffing, cost, quality, etc. • Key relevant parameters and considerations Each summary will occupy one page, but may be backed with supporting documentation. Supporting Documentation Additional supporting data may be included as necessary. For more information see the COMPANY Estimation Management Report format.

www.qsm.com

12

Software Estimation Process Recommended Solution (Estimate Example): A solution that meets all desired values is not feasible. This recommended solution is a compromise of constraints.

    

Estimate Metric

Goal

W o rk Plan

Schedule (Months)

8

Effort (Phrs)

Commitment Plan Value

Probability

8.4

9.1

80%

7,420

12,271

15,791

70%

Cost ($1000)

350

578.8

745.8

70%

Peak Staff (FTE)

5

12.4

13

53%

MTTD P1-5 (Days)

7

5.7

4

70%

Size (IU’s)

32,750

32,750

32,750

50%

To stay within internally available resources schedule will modestly exceed goal of 8 months Estimate assumes maximum internally available resources which drives cost above desired goal Because of the tight schedule any increase in scope/under-scoping will have immediate impact Estimate assumes a PI within the range of other CVS completed projects Mean Time To Priority 1-3 Defects should meet desired goal.

www.qsm.com

13

Software Estimation Process Figure 5. Company Estimation Management Report Example

Project Tracking and Adaptive Forecasting Ongoing Metrics Collection Once the project is underway metrics will be collected at regular intervals to support variance analysis and project status. The core metrics to be collected are: • • • •

Schedule dates o Phase start and end dates o Milestone completion dates Effort & Cost Size/Functionality o Requirements (traceability matrix) o Code Defects

These core metrics provide the basis for tracking not only our schedule and cost performance but also let us assess progress on how the product is being built and when it will be reliable enough for production deployment. These measures map directly to the core metrics advocated by the Software Engineering Institute for an organization to meet CMMI level 2 and higher compliance. SLIM-Control® SLIM-Control is an analysis tool for managing ongoing software projects. It uses statistical process control techniques to assess your progress, provide early warning if the project goes off-track, and forecast the most likely end date, cost, and reliability for your project based on performance to date. The following figure shows the overall control process flow:

www.qsm.com

14

Software Estimation Process SLIMEstimate

Make the Forecast the Current Plan

Bottoms Up

Plans

Other Tool

Manual or via API

Variance Analysis & Assessment

Forecast to Complete

Time Reporting System Config. Mgmt. System Defect Tracking System

Actuals

SEI Core Measures

Project Mgmt System

Figure 6 Project Control Process Plan and actual data is the basic information that drives our control process. •

•

Plans can come from a variety of sources. If you are using QSM’s SLIM-Estimate product plans can be brought in automatically. Plans can also be set up manually based on a bottoms up estimate or they can be brought in from other tools via the Application Programming Interface (API) Actual data in many cases is already available in other existing systems in the organization. Depending on the level of integration desired the data can either be manually entered into SLIMControl or automated via the Application Programming Interface (API)

•

Before the project begins, we establish control bounds around the plan. If our actual data crosses these thresholds, a variance assessment provides instant feedback for each metric. Whether the project is GREEN (on target), YELLOW (watch carefully) or RED (requires action), this process lets us see where the project is in relation to the planning baseline.

•

If the project strays from the plan, re-planning may be needed. This control process will evaluate the actual data and finds the most probable schedule, cost, and reliability based on the productivity derived from the core metrics. We can quickly compare forecasts, perform trade-offs by adding or re-assigning staff, and show our customer the schedule, cost, and quality implications of scope changes late in the development cycle.

Variance Analysis Once the project is underway we assess progress by comparing the actual data for various metrics to their planned values. The plan contains the expected values for each time period. Ideally, our actual data would always match the plan values exactly. But even with the best plan in place, the actual data can (and often does) differ from the expected values. The question then becomes: how much variation between the plan and actual values is acceptable? At what point should we consider corrective action or adjust the project plan?

www.qsm.com

15

Software Estimation Process The following variance analysis and statistical process control features are designed to help us assess our progress: •

Control Bounds - define regions of acceptable and unacceptable variation around the plan. They provide visual feedback, allowing you to assess the metric status at a glance. As long as the actual data falls within the acceptable region, no action is needed. The width of our control bounds can be adjusted to be consistent with the priority given to a project’s requirement to meet its baseline plan.

•

Variance indicators - Variance indicators such as Stoplights provide a simple visual metaphor for the project status. Green means our actuals are tracking the plan. If the actual data strays into the unacceptable regions, a yellow or red traffic light warning is generated. With this early warning, you can take the appropriate action to get the project back on track.

•

Project Dashboards and Snapshots – Metrics dashboards will consolidate all our important metrics and variance assessments into a single management display. This will give us an overall assessment of a project’s health. Snapshots can be taken from time to time to capture the overall status of a project based on the individual metrics variance. Over time these can provide a useful trend analysis showing us when a project started to get trouble and the probable causes.

Below is an example of project dashboard:

Figure 7 Example Project Dashboard & Snapshot Summary

Adaptive Forecasting Sometimes it becomes necessary to re-plan a project. This can be caused by many things such as changing requirements, over optimistic planning assumptions or underestimating the size/scope of the customer’s requirements. When these situations occur we need a mechanism to predict when a project will finish and what the final cost will be. Adaptive forecasting is our solution to re-planning part way through a project.

www.qsm.com

16

Software Estimation Process •

Curve fitting actual data – Actual progress data can be evaluated to determine the implied productivity that is being achieved on the project. By fitting theoretical progress curves associated with different productivity rates to the actual project metrics we can see which curve best fits the progress data. We do this independently for each of the core metrics. Once we know the productivity associated with each metric’s actual data we can weight them together to determine the overall project productivity. This then becomes the basis for our forecast to complete.

•

Reassessing project size and resource assumptions – Project assumptions need to be reassessed on a regular basis. As more information becomes available throughout the project lifecycle we need to reconfirm our size estimates. If our analysis shows that the size is substantially different from what we initially estimated we need to feed our new size assumptions into the forecasting process. Similarly if our resource assumptions have changed we need to factor these into to the forecast as well.

•

What-if analysis – After we generate our new forecast we can explore different alternatives. These generally fall into two categories, changing staffing assumptions and changing functionality assumptions or a combination of both. It should be noted that increasing staffing to get schedule compression becomes less effective as we move farther along in the project lifecycle.

Below is an example of project that has been re-forecasted:

Figure 8 Example of a Re-forecasted Project Re-Baselining Once all re-planning alternatives have been evaluated a decision is made on best course of action going forward. The forecast selected will then become the baseline that we will measure our progress going forward. New control bounds will be applied to this revised plan for measuring future progress. The original and all subsequent re-baselined plans should be saved to support the project post mortem review.

www.qsm.com

17

Software Estimation Process

Benchmarking and Historical Trends Historical Metrics Collection Historical data: As stated earlier this is extremely important because it provides the basis for an objective appraisal of an organizations software development capabilities. There are two sources for the historic information, an organization’s own historic data and available industry data. The primary source should be any previously completed projects within the organization. If SLIM Control® was used to track a project through to completion this process is easy, we simply import the data into the SLIM-DataManager® repository. If SLIM-Control® was not used then we need to reconstruct the project actuals (see Appendix D Project History Data Collection Form). The metrics we need to reconstruct are: • • •

•

•

Schedule dates o Phase start and end dates Effort & Cost by Phase Size/Functionality o Requirements (traceability matrix) o Use cases o Code (new, modified and unmodified) Defects o From systems test through then end of construction o By priority o Cumulative number per month after end of construction Environmental factors o Tools & Methods o Technical Complexity o Personnel Profile

If the data must be manually collected a good technique is to reconstruct the staff loading profile for the project. This will provide the schedule and effort information. We can then capture the size artifacts and gather the defect information to complete our project history. A profile of the environmental factors should also accompany the quantitative project history data. With this information it is possible to calculate a “Productivity Index”. The Productivity Index is a key assumption used in the SLIM Estimate tool. The fact that it can be based on actual performance rather than more subjective input greatly improves the independence and credibility of the analysis.

www.qsm.com

18

Software Estimation Process

Figure 9 Example of Completed Project Staff & Phase Reconstruction

If no relevant historical data is available then industry data should be used. This secondary source could come from any relevant projects from the QSM database. SLIM-Estimate® has a built in capability to derive productivity from a preprocessed set of industry data. If more specific data is required then the QSM database can be queried directly to pull out a sample of projects that meet given criteria. Some typical criteria would include: • • • • •

Time frame (i.e. completed after 2003) Size regime (i.e. between 43,000 and 97,000 new and modified source lines of code) Language(s) (i.e. primary language = Java, secondary language = Visual Basic) Peak staffing (i.e. peak staffing between 18 and 35 people) Application Domain ( i.e. IT application with a financial orientation)

The idea is to create a relevant independent sample of completed projects that can be used to compare with estimates. If the customer expectations or external developer estimates fall significantly outside of our own past performance and also falls outside of the relevant QSM data sample then the data provides an objective way to question the validity of the performance expectation. The data provide a “reality check” prior to commitment. SLIM-DataManager® SLIM-DataManager® is an open source metrics repository that can be used to capture project metrics. Its primary focus is to capture metrics on completed projects to be used to tune future estimates to www.qsm.com

19

Software Estimation Process demonstrated performance. The repository can also capture metrics on estimates and projects that are under way. Once the repository is populated with project records SLIM-Estimate® & SLIMControl® can use this data for calibration and validation purposes. An additional product from QSM, SLIM-Metrics can use this data for benchmarking and historical trend analysis.

Baselining and Benchmarking •

Once some data has been collected it is useful to generate some organizational baselines. These are internal benchmarks on things like schedule, cost, productivity and quality performance. Future estimates and completed projects an then be compared with these baselines.

Figure 10 Example of an Internal Baseline

We can also perform external benchmarks in order to see how our organization’s performance compares with other companies in our industry sector. The primary objectives of this measurement process are: • • •

Quickly and accurately assess productivity and quality measures being achieved in our current software development environment. Set realistic goals for improvement based on our current productivity and quality position. Benchmark the performance of our organization and determine our competitive position relative to others in the industry for cost, cycle time, productivity and quality (For Example, "We are in the top 85% for schedule performance within the financial services industry").

www.qsm.com

20

Software Estimation Process • • •

Create a metrics database to monitor improvements over time. Bring stability and accuracy to project estimation, planning and control. Transfer our knowledge of metrics collection and analysis to our organization.

Figure 11 Example of a Comparison to External Benchmarks

www.qsm.com

21

Software Estimation Process Trend Analysis After we have been collecting data over a long enough period of time we will be interested in analyzing any observable trends in our data. It is important to be able to point out any beneficial trends and connect them with our process improvement initiatives. These improvements in reduced cycle time, cost or improvement in productivity and quality help us build the business case for continued investment in long term process improvement that make the organization more competitive. We can also compare our rate of improvement over time with the average rate of improvement in our industry. This lets us assess our competitive position within our industry.

Figure 12. Example of Long Term Improvement Trends

If you’re interested in implementing a formalized estimation program or if you would like to see a SLIM product demonstration, contact us today. QSM, Inc. (Corporate Headquarters) 2000 Corporate Ridge, Suite 700 McLean, VA 22102 (800) 424-6755 [email protected]

www.qsm.com

22