http://www-128.ibm.com/developerworks/rational/library/jan07/bittner/index.html

Measuring project health: Part One

Level: Introductory Kurt Bittner,Program Director, Emerging Technologies, IBM 15 Jan 2007 from The Rational Edge: What project managers choose to measure as a gauge on performance generally receives the team's special attention. Naturally, project health depends on accurate metrics, but more importantly it requires that the right things be measured. This article describes some of the fallacies associated with traditional software project metrics, then focuses on effective measurement during the Inception phase. Most organizations use measurements in various ways to assess performance or effectiveness of various courses of action. Without measurement they have little more than subjective observation to determine whether things are on track; measurement is inevitable. But measurement is a tricky thing -- many times you cannot directly measure what you really want to know about, as in cases where you measure increases in output (productivity) when you really would like to know about effectiveness of resource utilization. There is a more subtle aspect to measurement: People pay attention to things being measured, and assume those measured things are important. Even when measurements are not directly tied to rewards, they will work to improve their performance in areas where they perceive they are being measured. Perception is key: Sometimes managers overtly say one thing and reinforce something else through measurements, effectively undermining their stated objectives. Measurement implies that the thing being measured is meaningful. Here's the point: People look to the measurements for cues about what is important. They will interpret this information, and they will often change their behavior to deliver what they think the people directing the measurement want. This effect shows itself in a number of ways -- from telling managers what they think they want to hear when asked "how is the project going?" to reporting sometimes more, sometimes fewer, hours than were actually worked, depending on whether they think working more hours than expected is a good thing (often the sign of a "hero culture") or a bad thing (working more hours than expected is sometimes a sign of a project in trouble). The best way to counter the unintended consequences of measurement is to be completely transparent about what you are measuring and why, which requires you to be completely transparent about project goals and how measurements relate to project goals. This is the first in a series of articles that addresses measurement in the iterative software development lifecycle. It describes some of the fallacies associated with traditional software project metrics, then focuses on effective measurement during the Inception phase. Subsequent installments will cover measurement during the Elaboration, Construction, and Transition phases. Measurements and goals Before you decide what to measure, you need to decide what is important. Most organizations focus on goals that fall into three areas: Reducing the time it takes to deliver a solution (time to market), subject to a minimum level of functionality, and a given level of quality and cost Reducing cost, subject to a minimum level of functionality, and a given level of quality and usually constraints on time to market Improving quality, subject to constraints on a minimum level of functionality, and on cost and time to market The essence of project management is to manage these trade-offs in an acceptable way to achieve the desired result. In order to do so, targets need to be set for these four areas. Most projects measure cost and time, but scope and quality affect cost and time and need to be measured as well. In fact, most measures of project success must include more than cost and time -- the delivered solution needs to do something useful (scope) and must do so with acceptable quality. Variability and goals "Prediction is very difficult--especially about the future." -- Niels Bohr A goal is a kind of prediction -- a statement about what will constitute success at some point in the future. Reality may fall short of the goal, or, if you're lucky, it may exceed the goal. In statistical terms, project results are random variables, and goals are estimates of desired values for those random variables. The key thing to remember about random variables is that actual observations are centered around an average, or mean value. The spread of the distribution, called its variability, indicates the degree to which observed values will vary from the mean. In a symmetrical distribution, values are equally likely to fall above the mean as below it, and the most likely value, the mode, is equal to the mean. Figure 1 shows a typical normal probability density function, so called because most observations fall into this sort of distribution. Figure 1: Normal probability density function, which illustrates that actual observations may vary from the expected (or average) One of the key mistakes that people make about goals is that they specify only one number for a goal -- the expected value (or mean value) to be achieved. But reality is variable -- and to take this into account we need to establish tolerances for goals, within which performance is considered acceptable. Doing so is a useful exercise, because it forces you to consider the amount of variability you will find acceptable, and it will prevent unnecessary focus on minimizing irrelevant variability while maintaining focus on important variance. Measurement and iterative development One of the principal benefits of an iterative approach is that it makes a conscious effort to reduce variance over the course of the project, as shown in Figure 2 below. Note that this illustration turns the sample from Figure 1 on its side, achieving a very narrow distribution toward the end of the iterative lifecycle. Figure 2: The iterative development approach reduces variance over the course of a project The line labeled "x" represents the expected value of an estimate, while the dashed lines above and below this line represent the expected variation around these estimates. Over the course of the project, the variation decreases as risks are retired as shown in Figure 3. Figure 3: Risk, which greatly contributes to variance and unpredictability, is reduced over time in an iterative project Each of the four phases shown in Figure 3 -- Inception, Elaboration, Construction and Transition -- focus on mitigating different types of risks. As a result, the goals for each phase differ as well. This means that the measurements used to track progress toward project goals need to change phase by phase. When measurement goes wrong Among the absolute worst practices followed by many organizations is "detailed planning, then measuring to plan." Planning and measurement, by themselves, are not bad things, but the timing employed by most organizations actually does more to encourage the wrong behavior than any other single practice. Here's what normally happens: A project is initiated, and as one of the earliest actions a project manager creates a project plan. Based on sketchy information about the problems to be solved and vague goals, the project manager does his or her best to "guess" what the schedule and costs will look like. No matter how many caveats are put forth about the tentative nature of the plan, the plan takes on a life of its own. Managers higher in the organization lose sight of the fact that early plans are really just thought experiments that are no more valid than the assumptions on which they are based. The plan becomes a kind of prediction about what success will look like, and the focus of management oversight shifts to measuring deviations from plan, rather than on steering the project toward a successful outcome. The plan becomes a kind of albatross around the neck of the project manager.1 Project team members are discouraged from taking actions that are not in accordance with the plan, even when everyone seems to know that the plan is a complete fiction, disconnected from the daily realities of the project. While the plan could, in theory, be correct, it is very unlikely that it will be. When such a plan inevitably fails to perfectly anticipate the future, managers who unquestioningly follow it often lead their teams and projects to failure. When the plan fails to match reality the only logical course is to deviate from the plan, but when the management system punishes deviations from plan, it often leaves the team with no option but to create the illusion that they are following the plan even when they are not. This sort of behavior does no one any good. Why does this sort of thing happen? The problem is not planning per se, but rather planning in the absence of reality and in opposition to experience. The problem is planning in detail things that cannot be planned at all, because not enough is known, and then holding the team "accountable" for adherence to the fabrication. A better approach is to understand where the project needs to end up, and then to use flexibility in steering toward that goal continuously throughout the entire project. In order to do this, you need to be looking for the right information at different points in the project. What's wrong with detailed up-front planning? While the preceding discussion touches upon what is wrong with planning in detail at the project outset, the practice is so pervasive and such a part of the project management tradition that it needs special attention to discourage its use. Briefly, creating and managing to a detailed plan early in the project is wrong because: It is unrealistic to expect that a project manager should be capable of the technical depth necessary to determine project activities and their timing for the technical work on the project. Milestones and goals can be determined, but the actual work that will be needed to reach those goals is beyond the technical understanding of the project manager. The technical expertise needed to plan at any meaningful level of precision requires the involvement of the larger development team, and any "planning" done by the project manager is likely to be speculative at best. The uncertainty of specific courses of action early in the project is sufficiently high that even detailed plans created by the development team are highly speculative. The early days and weeks of the project are characterized by lack of reliable information about even what problems need to be solved and what goals need to be achieved; detailed planning in light of this lack of information is simply a waste of time. If detailed planning is a waste of time, measuring to a detailed plan created early in the project is dangerous and often dooms the project to fail. Holding people accountable for flawed and faulty plans discourages people from taking the right actions when information becomes available. It can also force people to take the wrong actions because the plan tells them to do so, or, more often, it encourages them to create "two sets of books": one for the "real" project work and one that tracks back to the original but flawed plan against which they are being measured. Either way, the early detailed plan encourages the wrong behavior. So if measuring the project against a detailed project plan is not the right approach, what should be measured to ensure that a project is on track? Deciding what to measure -- phase by phase The short answer is, you need to measure different things at different points in the project. The project can be broken into the four phases of the unified process (Inception, Elaboration, Construction, and Transition), each of which focuses on a different set of goals pertaining to their different associated risks. That is, each phase focuses on mitigating a different set of risks. But before we get to a discussion of risk, it's important to understand the broad goals we recommend for each of the phases, as shown in Figure 4. Figure 4: Goals and objectives associated with each of the four lifecycle phases This article and subsequent articles will describe how to establish appropriate measurements to determine if the project is on track to achieving these goals. What to measure in the Inception phase The Inception phase has the following goals: To identify and mitigate business, project, and funding risks To assess the viability of the project, both technically and financially To agree to the scope and objectives of the project To form an overall plan for moving ahead Every project needs to be funded for at least the Inception phase; funding for subsequent phases will depend on the outcome of the Inception phase. The focus of the Inception phase is on mitigating the risk that the project might be either economically undesirable or technically infeasible. For this, the team must explore the benefits and costs of the project so that a firm decision can be made about whether to proceed. At the end of the Inception phase, the stakeholders agree that the project is feasible and that the business case for the project is achievable if the project proceeds as planned. Everyone must agree that the project is viable -- that is, that the project is worth doing, and the time and cost estimates are credible. If these things cannot be agreed, then it is best to forego the project and turn your attention to more worthy projects. Measurement in this phase is focused on answering the following questions: What is the business benefit of the project? What is the expected cost to deliver the business benefit? Is the project worth completing? The first two questions are the hard ones, and will dictate the answer for the third question. In the next two sections we will discuss how to measure the business benefit and expected cost, and how to factor these answers into an overall measurement plan. Assessing financial viability The expected business benefit establishes constraints on what the business is willing to spend to obtain those benefits. At this phase of the project, precise estimates are not needed; rough estimates should be sufficient to establish that there is clear value in pursuing the project. Business benefits are usually expressed in terms of net present value, which takes into account the time-value of money obtained, as well as risk. Mathematically, it is defined as: where the "end of the project" is really the lifespan of the solution and r is the discount rate, or required rate of return, of the project. Some attention should be given to choosing the required rate of return to make sure that it adequately accounts for the risk of the project. Typically the required rate of return is based on projects of equivalent risk. Ignoring the initial investment (covered in the next section about project cost), the business benefits are estimated by the future cash flows arising from the solution -- both the positive ones related to increased revenues as well as negative cash flows resulting from expected maintenance. Supporting evidence for the business benefit estimates usually includes market research data for new products or services, business cost data for projects focused on cost reduction, or both. As the NPV formula implies, the specific timing of the estimates is important; benefits received sooner are much more valuable than benefits received in the future. The benefits for projects driven by compliance regulations are easier to estimate. The project benefit is estimated by the penalty avoidance. The timing of the benefits of compliance is still important, however. Assessing technical viability and estimating overall project cost Obtaining estimates on the expected cost can, in theory, be obtained from parametric estimation models based on historical cost data2, but the scarcity of available data relevant to the project at hand usually means that you must build some small but representative part of the system in order to extrapolate project cost and schedule data. It helps to choose a few representative scenarios and develop the software to deliver them. Doing this will usually confirm whether or not your assumptions about the project's technical complexity are correct, even if you have to "stub out" a fair amount of the system functionality. Even if you have parametric estimation models, the early prototyping effort will help you to validate the parametric estimates. From this early prototyping effort, two main results emerge: a qualitative assessment of whether the project is technically feasible, and quantitative measures of project schedule and cost. The raw data obtained from the early prototyping is typically fed into one or more estimation models to derive cost information. Using more than one model is a useful way to cross-validate estimates -- wide differences in predictions provided by the models are signals that you should analyze your assumptions more carefully. Figure 3 presented an expected risk curve for the project, with risks peaking in the early Elaboration phase and declining afterward. Note that this curve represents the ideal risk profile you should strive for; you will achieve that only if you are actively and aggressively managing risk. There is nothing magical about the mere passage of time that causes risks to decline in number and severity. In order to bring risks under control, you must have a sense of what risks you face and when you expect to mitigate them. Many projects collect a risk list, usually early in the project; but unless the risk list is used as a key management tool, it will do little good. It is not enough to simply collect a risk list; you also need to map the risks onto iterations in an overall project plan, then later plan activities that will mitigate the risks assigned to the iteration as part of the iteration plan. In the early mapping of risks to iterations you don't need to create a detailed plan of project activities; just mapping risks to iterations is good enough for now. Doing so will give you a sense of whether you are addressing risks early enough. As you assign the risks to iterations, make sure to assign big risks to early iterations. In order to do this mapping, some exploration needs to happen in the Inception phase to even understand what risks you may face. Mapping out these risks and then making some educated guesses about what kind of work will need to happen in order to address the risks will give you a sense of what resources will be required, by iteration. Again, the goal is not to create a detailed plan but to provide a general roadmap for project staffing. This staffing model (or "map" if the word "model" sounds too formal) will provide input to the overall cost estimate by establishing timing for the costs. Iteration in the Inception phase Sometimes it takes more than one iteration to determine the business viability of the project. This usually happens because of unexpected difficulty in determining the technical or financial viability of the project. Consistent with what we have discussed so far regarding risk, it sometimes takes more than one iteration to mitigate the business risks associated with the project. Usually the Inception phase is "timeboxed," meaning that a pre-determined amount of time, usually a month, is given to the phase. Normally the Inception phase has only a single iteration; at the beginning of the project there is usually no reason to assume otherwise. As the iteration progresses and nears the end, sometimes the results are sufficiently inconclusive to warrant further exploration, but not so conclusive to dictate abandoning the project. As a result, an additional iteration is funded to conclude the investigation. Early in my career my employer used a process which had as its first phase a "feasibility study". The Inception phase serves this purpose, but with a twist: it is typically essential to do some prototyping to explore the technical feasibility of the project, whereas the old "feasibility study" was often a largely theoretical undertaking. Determining project viability Determining financial viability is usually a two-step process. First, there is often a time to market factor to consider: sometimes if a solution cannot be delivered to market within a specific time window, the project is not worth completing. If the time to market requirement can be met, then the NPV of the proposed solution is calculated. If the NPV is negative then the project is not worth pursuing. If the NPV is positive then the project may be worth pursuing, provided that it does not compete for funding with other higher-valued projects.3 Factoring risk into goals tends to be subjective and is best accomplished by increasing the variability of the estimates -- in effect, by widening the "margin for error". Most estimates in this phase are, in reality, highly subjective and can be off by wide margins. The most valuable question to ask about an estimate is not "is it accurate?" but "how wide a range of values is possible?" Alternatively, variability in the estimates can be accommodated by increasing the required rate of return for the project in the NPV calculation. Typically, a number of scenarios are analyzed to determine the expected case and the worst case values for NPV. Often the decision to go ahead with a project is based on whether the worst-case NPV is within acceptable limits. Once the solution is deployed, the estimated NPV will become important as a measure against which actual results are compared. Other measures While estimated NPV remains the most important measure in the Inception phase, other measures must be considered for those projects that will be funded: Subjective measures such as: Customer satisfaction. Is the sponsoring organization happy with progress being made? Morale. Is the project team happy with progress being made? Are they confident (at least guardedly so) in success? Objective measures such as: Absolute progress. As measured by whether the phase completed within expected time frames. Requirements stability. As measured by how much the project's goals changed over the course of the phase. A lot of change means that the projects goals may still be in flux, which would mean that the NPV estimates may be partially or fully suspect. Risk stability. As measured by how much the project's risks are still being discovered. It's hard to know what you don't know, so this is also partially a subjective measure, but if new major risks are still being discovered at a significant rate it is a sign that you may not have a good enough understanding of the solution or the business needs to proceed. Over the course of the project, you will need to be gathering data and comparing it to expectations to keep the project on track. You will not have a lot of data on the following measures in the Inception phase, but you need to have mechanisms in place to gather the data before you leave the phase. In all these measures, trends are more important than absolute values. Effort expended. Actual versus expected Defect discovery and fix trends. Actual and expected Risk discovery and mitigation. Actual and expected Unplanned work identified and implemented, including scope changes Planned work implemented and tested. Actual and expected In all of these, the estimates are fairly coarse. For things like planned work, a map of scenarios onto iterations is sufficient. How to work with these measures in the subsequent phases of the project will be the topic of subsequent articles. Taken together, these initial measures, coarse as they are, provide a framework for measuring project progress and health that will be used throughout the project to determine whether the project is on track. It is worth noting that determining not to fund a project is a perfectly acceptable outcome of the Inception phase and should not be regarded as a failure so long as the decision is based on measures of the business value to be delivered and the technical feasibility. In fact, deciding not to further fund the project may be the best possible outcome if it frees resources for other more worthy projects. Summary The goal of every project team should be to increase the probability of project success while minimizing time and cost. This general goal ties back to the general goals of the business, which are to deliver some improved capability to its customers within a desired time frame and with acceptable quality and within acceptable ranges of business value delivered. Using the right measurements will help achieve these goals by providing clear guidance to team members about what is important. Clear measures are those that are visibly tied to project performance and project health, and that actually help the team to deliver successfully. Keeping measures simple helps the team focus. And simple measures have other benefits as well: they reduce reporting overhead, and make interpretation of the measurements easier as well. Measuring the right things will encourage healthy project behavior, while measuring the wrong things is not only a waste of time and resources, but also sends the wrong message to the team about what is important. Varying the measures according to the lifecycle phases lets people focus on the risks and issues that are of greatest importance, at the time when this attention will have the greatest impact. While we say that the value of the iterative approach is to reduce project risk over time, reinforcing this message with the right measurements enables the team to put this into action. In the Inception phase, measurement is mainly focused on establishing the financial and technical viability of the project, and in so doing establishing cost and overall schedule and risk mitigation goals for the remainder of the project. Financial viability is based on analyses of market needs and opportunities, expressed in terms of cash flows over time, while technical viability is assessed through one or more prototypes that explore alternative technical approaches, as well as their costs and benefits. By the end of the phase, the information obtained from these exploration will provide the bases for a measurement framework that will be used later in the project to determine whether the project is on track. Further reading This article was drawn from material initially presented in Managing Iterative Software Development Projects, by Kurt Bittner and Ian Spence, and published in 2006 by Addison-Wesley. Notes 1 If the reference to the albatross is confusing, I encourage you to read "The Rime of the Ancient Mariner" by Samuel Taylor Coleridge. 2 E.g., models such as CoCoMo II, Coarse-grained Wide-band Delphi (i.e., consensus-based estimates), and function point analysis (including "use-case points"). 3 Project portfolio management is actually a little more complex than suggested here, due to project interrelationships, but for now we'll leave it at this. We will return to portfolio management concerns in a later article.