Successful software project outsourcing - how to limit the risk of failures H.S. van Heeringen
@haroldveendam
SMEF 2012 (Rome)
Introducing Harold • Sogeti Nederland – senior Metrics consultant − Internal + external
• NESMA – board member − Wg Benchmarking − Wg COSMIC
• COSMIC – International Advisory Counsil • ISBSG – president • Request For Proposal Management: Answering metrics based bids
2
Agenda • Request for Proposals (RFPs)
• Typical questions in RFPs • Challenges from the suppliers’ point of view • Why realistic estimates are important • How to limit the risk of failures? 3
Agenda • Request for Proposals (RFPs)
• Typical questions in RFPs • Challenges from the suppliers’ point of view • Why realistic estimates are important • How to limit the risk of failures? 4
Request for Proposal (RFP) • An invitation for suppliers, through a bidding process, to submit a proposal on a specific product or service (source: wikipedia) • Information provided: − Corporate information
− Schedule of bidding process − Project Summary − Detailed overview of the project − Decision criteria
5
Demand issues (RFP sender) • Provide the right information − Detailed
− Up-to-date
• Ask the right questions • Build a good decision model
• Evaluate the proposals • Choose wisely
6
Supplier issues (RFP responder) Can we: • Deliver the required functionality ? • Meet the technical and quality requirements ? • Within the time limits required ? • Answer all RFP questions ? • Estimate the project costs accurately ?
• Score the best on the clients decision model ? • Support our claims with proof ? 7
Sogeti SEC • Division AS – RVO’s in Center of Excellence • Sizing, Estimating & Control (CoE) − (COSMIC) Function Point Analysts − Metrics consultants
• Responsible for metrics part of a quotation. − Size: FPA/COSMIC − Estimation: SEER-SEM / QSM SLIM / Sogeti tool / ISBSG − Scenario’s !
− Product: Methodical Estimation Report ◦ Scenario’s ◦ Risk 8
Bid process Client procurement organization RFP
proposal
Sogeti delivery management
Estimate request
Project estimate
SEC department Sizing (FP) Estimating (duration, effort, cost)
challenge
Estimate request CoE architects Expert estimates Estimating (duration, effort, cost)
9
Agenda • Request for Proposals (RFPs)
• Typical questions in RFPs • Challenges from the suppliers’ point of view • Why realistic estimates are important • How to limit the risk of failures? 10
Typical RFP questions 1. What is your productivity for Oracle projects? 2. How long will it take for you to build a .Net application of 500 FP? 3. What is your price per function point for a 500 FP Java system? • Are these the right questions? • Is it possible for the client organization to make the right choice based on the answers to questions like these? 11
Sizing projects with function points • Function Point Analysis (NESMA, IFPUG, COSMIC) − Objective (ISO/IEC)
− Repeatable − Verifiable
• Quantifies the size of the functional user requirements − Independent of the technology used − Independent of the implementation method
• A measure of the size of the product, not the project ! • ‘non-functionals’ are not measured 12
Project Estimation based on functional size • Size objectively measured − Size = xxx function points
• Estimation of: − Effort (hours) per function/role − Duration (months) and milestones
− Team size (in fte) − Quality (defects during test and after delivery)
• Tools − Galorath SEER-SEM − ISBSG data portal − Sogeti Estimating wizard
− QSM SLIM
13
Generic Estimation Model Effort
Energy Size
Need
Productivity Software development process
Metric: Effort Number of hours Manpower buildup Peak staff Size
Software Defects
Time Metric: Process productivity Metric: Size Durationteam Skills and experience Function Points Metric: Quality Development environment Metric: Size Metric: Duration Waste Number of defects Complexity Function points Number of weeks Quality Management System Defects External influences 14
Agenda • Request for Proposals (RFPs)
• Typical questions in RFPs • Challenges from the suppliers’ point of view • Why realistic estimates are important • How to limit the risk of failures? 15
Size: Cone of Uncertainty Size: Function Points 4x
RFP
3x 2x
Project
CD Rate
1 2 3 4 5 6 7 8 9 10 Average
5 3 1 2 1 3 5 4 4 3 3,1
time
1x 0.8x 0.5x
Idea Concept High Definition Level Design
Why
Low level Design
What
Realization
How 16
Size always increases!
RFP
Size
time
Challenge: What size will we use in our estimate Global Detailed Idea Concept and which size will the competitorRealization use? Definition design design
Why
What
How 17
Creep!! • Scope creep − Adding / changing / deleting functionality during the project as a result of new ideas or changing environment − Literature: about 3% per calender month − Change management!
• Requirements creep − Requirements are described in a more detailed way, resulting in a bigger size !! − Project manager should be aware that the size to be delivered is larger than the size estimated and plan accordingly! 18
The effort / duration tradeoff
Effort
Size/productivity = Effortx x durationy
Plan A: 6 months, 4.500 hours
Plan B: 7 months, 3.400 hours
Duration
19
Same project, different durations
Effort (hours)
A (minimum time) Duration: 6 months Effort: 4.500 hours Max. team size: 5,8 fte MTTD: 1,764 days B (optimal effort) Duration: 7 months Effort: 3.400 hours Max. team size: 3,9 fte MTTD: 2,816 days
Duration Size and Productivity constant
20
The different zones
Effort or Cost
Minimal duration / highest effort and cost
Impossible zone
Impractical zone
Realistic zone
Optimal duration / lowest effort and cost
Duration 21
Effort hours
The impact of Duration in practice Minimal Time
Scenarios based on duration
Optimal Effort
Example Scenario 7: 1: 2: 3: 4: 5: 6: Duration: 6,3 5,5 months 5,2 4,8 4,5 5,8 6,1 Effort: 4.700 5.000 Mhr 5.500 5.900 6.300 5.200 4.900 Team size: 5,5 6,7 fte 7,5 8,3 9,4 6,2 5,8 Cost: € 360.000 430.000 480.000 530.000 620.000 400.000 380.000
Estimate / Business Case Cost depended on Time-tomarket
Duration 22
Back to one of the typical RFP quesions 3. What is your price per function point for a 500 FP Java system?
Minimal duration / highest effort and cost
Price/FP
1000 Impossible zone
Impractical zone
Realistic zone
500 Optimal duration / lowest effort and cost 6
Duration (months)
12
23
Agenda • Request for Proposals (RFPs)
• Typical questions in RFPs • Challenges from the suppliers’ point of view • Why realistic estimates are important • How to limit the risk of failures? 24
Professionalism and realism • Expertise − Use of function point analysis − Database with experience data − Repository with Benchmarkdata / tooling
• Realism − Opportunism: ‘Buying projects’ ◦ They really can’t afford to…and will find a way to profit
− Commercial interests
• To make an unrealistic offer is in nobody’s interest! • Professionalism and Realism are crucial! • On both client and supplier side!!
25
Extra costs with incorrect estimations Non- Lineair extra costs >100%
-Plannings errors -Larger team much more expensive, barely faster
Extra Costs
-Extra management attention / overhead -Stress: More defects, lower maintainability of the code !! Lineair extra costs Extra hours will be spent Underestimation
Overestimation
0% Too low estimates
Realistic estimates
Too high estimates 26
In practice
A: Optimistic 3.000 hours 5 months B: Realistic 5.000 hours 7 months
Result Fails ! 10.000 hours 12 months
15.000
Realisation (hours)
Proposal
10.000
Succesful ! Efficient! 5.000 hours 7 months
7.000
5.000
C: Pessimistic 7.000 hours 11 months
Succesful ! Not efficient !
3.000 hours
A
5.000 hours 7.000 hours
B
7.000 hours 11 months 27
C
Agenda • Request for Proposals (RFPs)
• Typical questions in RFPs • Challenges from the suppliers’ point of view • Why realistic estimates are important • How to limit the risk of failures? 28
How to select the right proposal? • First assess reality value of the proposal − Make sure you can identify the unrealistic proposals
− Unrealistic proposals should not be chosen
• Choose the best one of the remaining proposals − The proposal that scores best in the decision model of the client organization
29
Client recommendations • Ask the right questions − objective comparison, keeping as many relevant factors as possible equal
• Perform a reality check of the proposal − Compose a range in which the proposal should be
− Tools: Galorath SEER-SEM / QSM SLIM or the ISBSG database
• Ask for objective proof − Experience data of the suppliers − Assess if the supplier can deliver software as productive as promised
30
What is a good question? • Metric to compare, for instance: − Productivity (hours/FP, FP/month) − Cost (Price/FP) − Quality (defects/FP, Mean-time-to-defect (MTTD), Maintainability index)
• Technology − For instance Java, Cobol, Oracle or MS.NET
• Size (in Function points or COSMIC FP) • Technical/ Functional complexity − For instance: high/average/low
• Phases/Activities included − For instance Technical design, Coding, Unit test, systems test.
• DURATION !!
31
Example of a good question ‘What is your PDR (hours per function point) for a moderately complex Java project of 500 function points and a duration of 20 weeks? Activities to include are technical design, coding, unit testing, systems testing and support of the user organization during the user acceptance test’ and also includes all overhead activities, like project lead and quality management. 32
Three proposals
33
Three proposals Proposal 1 2 3
Size (FP) 500 500 500
Effort (hours) 2.275 3.550 8.600
Duration (months) 6,8 6,8 6,8
Productivity (hour/FP) 4,5 7,1 17,2
• In practice: many ‘not so mature’ client organizations will grant the contract to supplier of proposal 1. • Is this wise? • What should we do?
34
Reality value of the proposal • ISBSG data portal − International Software Benchmarking Standards Group
− >5.800 projects ‘Best in Class’
ISBSG R11 VALUES IN INTERVAL PERCENTILE 10% (P10) PERCENTILE 25% (P25) MEDIAN PERCENTILE 75% (P75) PERCENTILE 90% (P90)
Hours/FP 24 3.5 7.0 8.4 11.6 19.6
Duration 24 3.3 months 4.5 months 6.0 months 9.5 months 12.2 months
• Realistic range: 7.0 hours/FP – 11.6 hours/FP
• Realistic range: 4.5 - 9.5 months
35
SEER-SEM • Reality assessment in SEER-SEM • Simulate the project based on the appropriate knowledge bases in the tool
SEER-SEM PDR (Hours/FP) Duration (months)
Minimum Requested Optimal Time duration Duration 14,8
7,4
6,2
4,3
6,8
8,2
• Realistic range: 6,2 h/FP – 14,8 h/FP • Realistic range: 4,3 months – 8,2 months 36
QSM – Productivity index (PI) PI vs Effective FP
20
18 Proposal 1
Proposal 2
16 Proposal 3
PI
14
12
10
400
420
440
460
480
500
520
540
560
580
8 600
Effective FP 37 All Systems
QSM 2008 Business FP
Avg. Line Style
1 Sigma Line Style
Which one to chose Proposal 1 2 3
Size (FP) 500 500 500
Effort (hours) 2.275 3.550 8.600
Duration (months) 6,8 6,8 6,8
Productivity (hour/FP) 4,5 7,1 17,2
• Proposal 1 is not realistic if not supported by evidence that they can perform at this productivity • Proposal 2 is more realistic. If supplier 2 has the data to back it up, they should be chosen! • Proposal 3 is realistic, but probably too expensive 38
Recommendations summarized • Ask the right questions: − Size, Cost, productivity, duration en quality are highly interdependent − The goal is to try to get answers that are as comparable to each other as possible
• Reality check of the proposals − Analyze Benchmark repositories or tools to come up with a realistic range. Don't accept unrealistic proposals − Always ask the supplier for evidence that they are as productive as they claim.
• Choose wisely − When the cheapest proposal always wins, too few good questions have been asked! 39
Summary • Suppliers face a number of difficulties when they have to answer a ‘one dimensional’ question
• More mature suppliers that can prove their performance based on experience data are often outbidded by suppliers that have no idea about their performance and just take the risk • However, unrealistically optimistic expectations lead to huge failures! • Clients as well as suppliers should create a common basis of understanding, so that the industry can become more mature. 40
Sogeti Sizing, Estimating & Control Thanks for your attention !
Harold van Heeringen Sizing, Estimating & Control
[email protected] @haroldveendam @Sogeti_SEC Sogeti Sizing, Estimating & Control NESMA – board member NESMA – chair working group COSMIC NESMA – chair working group Benchmarking NESMA – working group Sizing Packages NESMA – working group Estimation maturity COSMIC – International Advisory Counsil COSMIC – Benchmarking Committee ISBSG – President
41