Reducing the costs of software development is really pretty simple. (I didn’t say easy, but rather, simple.) Using a development maturity assessment it has been done repeatedly. The Software Engineering Institute (SEI) Software Development Maturity Assessment Methodology is used to assess the software development capability of organizations. Research by Lawrence Putnam of Quantitative Software Management (QSM) demonstrates a strong relationship between Capability Maturity Model (CMM) maturity level and the QSM ‘Productivity Index’ (PI). Specifically, rising CMM levels result in higher Productivity Indices, which result in lower development costs.
In a nutshell, higher Productivity Index values are associated with projects that cost less, finish
faster, and have fewer defects. Ideally the CMM process improvements should be associated with more efficient projects and better quality. What’s covered in this article is that the QSM methodology, benchmark database, and tool set measure of the benefits of CMM improvements.
This article points to the economic benefit of effective software process improvement, and the role that measurement plays in proving it.
Background
Many companies have undertaken software process improvement (i.e., Software Engineering Institute’s CMM/I) with the hope that better process will somehow produce better results. For example by moving to CMMI level 3 they would expect to experience projects with shorter schedules, reduced costs, improved reliability, fewer emergencies, etc. However, without a metrics plan and a quantitative toolset in place – apart from “anecdotal evidence” – they will never really know. Even worse, process focus, without the right measures encourages individuals to default to rote compliance with ‘process.’ Eventually, lacking any believable measures of improvement the organization abandons the improvement effort altogether. After all, process improvement requires discipline and continuous investment.
If you’re going to pursue software process improvement, protect the investment from the outset with a solid metrics plan and benchmark database.
Start a Basic Metrics Plan
Rather than trying to measure too much, organizations need a basic ‘starter set’ of metrics for basics such as duration, effort, size, and defects. Furthermore these metrics should not be used to measure individuals, but rather to better understand the software development process, making continuous improvement for repeatable success. Metrics misapplied will submarine data quality and put a halt to improvement. Fear drives out learning. Without learning there is no improvement.
Apply a Measurement Framework
Software projects are ‘different’ from other projects, such as construction. Software does not obey the laws of physics and science. Rather it requires human learning, discovery, problem solving and communication, which makes schedule prediction more difficult. (When asked to estimate “how long” to complete an unfamiliar programming task the developer responds, “Don’t know. How long does it take to catch a fish?”)
Software development projects – or most design-type problem solving projects follow a non-linear resource staffing pattern. The sample Raleigh Curve below (Figure 2) shows the common pattern. At the beginning of the project there is a staffing ramp-up (Physical Design), the project peaks at the conclusion of programming and the beginning of testing, and finally the long tail (Testing and Debugging) represents the effort to find and remove bugs over a considerable time frame.
Software development and circuit design projects tend to follow this effort/staffing distribution.
Also included in the QSM Measurement Framework are these measures:
- Effort – Person hours of work
- Duration – Elapsed days
- MBI (Manpower Buildup Index) – Rate at which people are added to the project
- Defect Density – the number of bugs to be removed
- Size – Some characterization of what is delivered.
- Productivity Index – a macro measure of the organization’s development efficiency
Understanding the Productivity Index at a Glance
The software process Productivity Index (or PI) is a QSM metric, representing the level of an organization’s software development efficiency. The PI is a macro measure of the total development environment. PI values from 1 to 40 have been adequate to describe the full range of projects seen so far.
Low PI values typically are associated with poorer working environments, poor tools and/or high product complexity. Higher values are associated with good environments, tools and management and well-understood, straightforward projects [Ref. 1, 5].
“Productivity” encompasses many important factors in software development, including:
- Management influence
- Software development process and methods
- Software development tools, techniques and aids
- Skills and experience of development team members
- Availability of development computer(s)
- Complexity of application type
Note that the PI is calculated from the size, schedule and effort that were applied to a completed project. This means that the PI is objective, measurable and capable of being compared on a numeric scale.
Projects normalized around the PI can be meaningfully compared to one another. Without this normalization projects’ performance cannot meaningfully be compared. For example, project A took 6 months to complete, and project B took 4 months to complete. What conclusion can be drawn? None. But if both projects have PI’s then we might see that one had greater size, or greater complexity – or had a team that ramped too slowly. In any event, an organization that gets a bead on its PI has incredibly valuable information to help estimate future projects.
CMM Transition Breakpoints
Research conducted by QSM on the relationship between CMM Level and PI shows in table 3. The benchmark database consisted mostly of Level 1 and Level 2 projects (which characterize the world), therefore the relationships are statistically strongest at these levels. In the Business Systems column we can see that average PI improves from a 12 to a 17 as the organization graduates to Level 2.
|
CMM Level |
Business Systems PI Value |
Engineering Systems PI Value |
Real-Time Systems PI Value |
|
I |
12 |
10* |
6* |
|
II |
17 |
15 |
9 |
|
III |
19.5 |
18 |
11.5 |
|
IV |
22 |
20.5 |
14 |
|
V |
25 |
23 |
16.5 |
[Table 3] Transition Breakpoints for Three Application Types – Note: * Estimated
PI Improvement Reveals Economic Savings
The graph in Table 4 shows results for three Business Systems projects of similar size and complexity. Process improvements – which are related to PI improvements – have lead to more efficient development capability, and a much lower cost. The transition from CMM Level 1 to 2 shows a 50% cost reduction. The transition from Level 1 to Level 3 shows a 75% reduction in costs and a 250% improvement in reliability.
But if were to realize a 50% savings, how could it be reasonably spent? How about these options:
- Finish faster
- Use fewer people
- Deliver more scope
- Use the saved money on other projects
- Give back to the Business
| CMM Level | PI | Duration | Effort | Peak Staff | Mean Time to Defect | Cost |
| 1 | 12 | 15 | 123 PM | 12 | 1.76 Days | $1.2 M |
| 2 | 17 | 12 | 67 PM | 8 | 2.65 Days | $0.67 M |
| 3 | 19.5 | 10 | 31 PM | 5 | 4.85 Days |
$0.31 M |
[Table 4] – The Economic Value of Software Process Improvement for Business Systems
The related graph in Figure 5 displays the Raleigh curve effort distributions for the three Business Systems projects. Note that project duration and peak staffing decrease with CMM level improvements. This is good news.
[Figure 5] – The Economic Value of Process Improvement (Courtesy of QSM)
Is your software process improvement effort paying off?”
Companies rightly undertake process improvement initiatives looking for some kind of improvement in delivery and cost. To some the method of choice is the CMM, to some it’s Agile methods, to others it’s custom processes and project management offices (PMO). But a word of caution: the project and organizational processes that can be implemented have the possibility – but not guarantee – of improving software delivery. Organizations frequently become totally lost in process, and confuse ‘process sophistication’ with ‘real maturity’.
Using the right measurement framework we can actually tell the difference between process improvement efforts that are paying off and those that are not. Are we finding more bugs earlier in the lifecycle? Are the schedules becoming shorter? Is the Productivity Index increasing? Are costs dropping? Do we require fewer people to get the project completed? Is the user finding fewer bugs? Which techniques are giving us the best return on investment? Are project estimates improving? Are fewer dates slipping? What are the priority areas we should next focus on? The point here is that the right measurement framework helps us to know and intelligently manage the investment in process improvements to yield the best return on that investment.
You may have process improvement efforts from which you are already seeing benefit. It’s possible that what you are seeing is only a fraction of what is possible. Without a benchmark database you will not know the extent to which you are performing beneath your capability.
Conclusion –
QSM’s research suggests a strong relationship between CMM level and the QSM Productivity Index. These improvements lead to significantly reduced software development costs. With a metrics plan and the right measurement tool set, meaningful measures position an organization to manage significant economic benefit.
Considering that the QSM tool set has a framework including industry data from over 8,000 projects, and a method to normalize project experiences, it is suitable as a software project management estimation and analysis tool.
This article points to the business case for software improvement. If we understand the nature of CMM level 2 and 3 improvements, they are primarily focused on project management. For this reason this article also suggests the business case for investing in project management.
Next steps:
As you no doubt know, project life is complex and fraught with many obstacles. A seasoned professional can help you with the most efficient path to your project goals.
Why wait? If you are an executive overseeing projects that have:
- Significant financial consequence for delays
- Repeated schedule slips
- High degree of uncertainty or risk
- Large number of cross-functional team members that are geographically dispersed
- High-level and constant focus, intensity, and attention from executives
We have over three decades in successfully managing technical projects of all kinds for companies of varying sizes, markets and cultures.
We offer a no-obligation consultation discussion that will help us determine a solution for your situation.
Request a Free Consultation
At no cost you will receive the benefit of an expert, independent review of your business growth goals or organizational issues. If you are located within or near the San Francisco Bay Area we will visit in person. If you are more than 2-hours outside of the Bay Area we can meet over the phone or web-meeting. You will receive a written summary assessment that will include recommendations that are specially tailored to your situation.
There is no cost or obligation for this service.
To get started, use this form to send us your request. We’ll respond within 1 business day.
Or call (415) 246-7264
References
1. Putnam, Lawrence H. and Ware Myers, Measures for Excellence: Reliable
Software On Time Within Budget, Prentice Hall, Englewood Cliffs, NJ, 1992, pp. 32-36.
2. Humphrey, Watts, David H. Kitson and Tim C. Kasse, The State of Software Engineering Practice: A Preliminary Report, CMU/SEI-89-TR-1, ESD-TR-89-01, Software Engineering Institute, Carnegie-Mellon University, Pittsburgh, Feb. 1989, 27 p.
3. Putnam, Lawrence H., “The Economic Value of Moving up the SEI Scale”,
Managing System Development, Applied Computer Research, Inc., July 1994
4. Putnam, Lawrence H., Arlyn D. Schumaker and Paul E. Hughes, Economic
Analysis of Re-Use and Software Engineering Process, (Final Draft Report) TR-
9265/11-2, prepared for Standard Systems Center, Air Force Communication
Command, Maxwell Air Force Base, Alabama 36114, under contract FO1620-90-D-0007, February 1993.
5. Putnam, Lawrence H. and Ware Myers, Executive Briefing: Managing
Software Development, IEEE Computer Society Press, Los Alamitos, CA, 1996, 79 p. Linking the QSM Productivity Index with the SEI Maturity Level,
Version 6, 2000
6. Putnam, Lawrence H, Linking the QSM Productivity Index with the SEI Maturity Level. July, 2000
7. Kolinger, Joe, Seven Signs You Have a Bad Project Estimate, Project Management Institute SF-Bay Area Chapter, January 2010
J Kolinger, February 2010
