Manage Metrics / Techniques

2.7.2T Manage Metrics / Techniques

Make Sure Your Measures Add Value (2.7.2T.P1)

Identifying, gathering and leveraging the right mix of metrics are ways to add value to a project. The value can be quantified in a number of areas including:

Improved performance of the overall project fulfillment and delivery process
Improved estimating for future projects
Validation of duration, cost, effort and quality objectives for the project
Identification and communication of best practices
Improved client satisfaction

Metrics provide a more factual and quantitative basis for understanding how you are doing and the things that can be done better. Without at least some basic metric information, all discussions on performance and improvement are based on anecdotal evidence, perceptions and guesses. If you want your project's success or failure to be based on factual information, you need to determine the success criteria ahead of time and how to measure these criteria. Then collect the metrics, even if they are imperfect and imprecise. They still provide a better foundation than recollections, perceptions and guesses.

Use the Metrics that You Collect (2.7.2T.P2)

You don't want to collect metrics just for the sake of collecting them. That doesn’t make sense from a project management perspective and it just ends up being a waste of time. If certain metrics are required by your organization, collect them. In addition, you should collect any other metrics that are needed by your particular project. However, if you don't have a purpose for the metrics, or if your project is not long enough that you can really leverage the information, these customized project-specific metrics are not worth collecting for your project.

Compare the Cost of Collecting a Metric vs. the Benefit (2.7.2T.P3)

Just as there is some cost associated with most project management activities, there is a cost to collecting and managing a metrics process. In the case of scope management or issues management, this is a cost the project manager needs to invest in to be successful, since they are core project management processes. The effort associated with managing metrics, however, is more under the discretion of the project manager and is dependent on the overall organizational culture. In many cases, the cost to collect and leverage a certain type of metric is prohibitive. These metrics should not be pursued. Other metrics are interesting, but do not provide the type of information that can be leveraged for improvement. The bottom line is that the cost to gather each metric must be balanced against the potential benefit that will be gained. Start by gathering metrics that are required by the organization. Then add metrics that have the lowest cost and effort to collect and can provide the highest potential benefit.

Link Team Performance with Individual Performance (2.7.2T.P4)

This old adage about “what gets measured gets done” is true on projects. If you are using the metric information to drive decisions on performance reviews or compensation, people will generally do what they need to do to hit the targets. For example, if communication is important on your project, build some metrics around communication. You can survey the clients and stakeholders on a quarterly basis to see how effective they think your team communication is.

You still may not drive the behaviors you need if the results of the metrics do not have a corresponding personal impact on the team members. The key is to collect metrics that give a quantifiable indication of overall team performance and make sure there is a connection between team performance and individual performance.

An example of where these are not linked is the classic case of the project that is seen as a failure, yet all the team members are evaluated highly on their performance reviews. This does not seem to make sense. Make sure that project team success is reflected appropriately in the individual performance reviews. If the team was successful, team members should be rewarded. If the team was not successful, team member reviews should be impacted accordingly. Remember that project team performance doesn’t need to drive 100% of the review process. There will be other factors to consider for overall performance. However, it is also not right that project team performance counts zero percent. There does need to be some tie between the success of a project and the success of the people working on the project.

Beware Unintended Consequences When Establishing Metrics (2.7.2T.P5)

Collecting metrics will drive certain behaviors, if the results of the metrics are used to as input into a person’s performance. In fact, this is exactly what you want to have happen or you would not be collecting the metrics to begin with. Therefore it is critical that the metrics drive the correct behaviors and not drive unintended behaviors.

Look at the following example. A team was being measured on the length of time that it took to close customer inquiries. The target was set to close all open customer inquiries within two business days. Team members realized that they needed to close the tickets quickly or they would be viewed as unsuccessful. The team members also realized that if they could not close the inquiry quickly, they should simply guess at the cause and then close the open ticket. If the guess did not solve the problem, a new problem ticket was opened and quickly resolved again. The result was repetitive thrashing of problem tickets, wasting much more time than required. In other words, the target was being met by driving bad behavior.

In this example, one of the problems was the metric. This team looked good on paper, since they were closing problem inquiries quickly. However, in reality they were performing poorly, generating extra work and causing the client to be dissatisfied. The idea was fine, but perhaps a better metric would have been to consider the inquiry “closed” when the client agreed and approved the resolution.

When you are considering a metric for your project, think about how the metric might drive unintended consequences. Be sure that you set up the metric in a way that clearly drives the desired behaviors.

Gather a Baseline if no Target is Available (2.7.2T.P6)

The collection of metrics information by itself provides only limited value. Most of the value comes when you can compare your metrics against some type of standard or target. For many metrics, there is an implicit target, even if you don't always specifically state it. For instance, the collection of actual effort, duration and cost information is used to compare against the estimated effort, deadline and budget to see how the project stands.

For many metrics, however, there is not necessarily an implicit target to attain. For instance, if you track your client's satisfaction with the project, they may rate your team an average of 3.8 out of a 5.0 scale. However, is that a good number or a bad number? Since you have nothing to compare it with, it's hard to say. The way to gain more value from the metric is to use the first measurement as a baseline - that is, a reflection of where you are today. In the survey example, you would want to consider your first results to be the baseline, and then you would strive to improve upon your baseline numbers. For instance, after collecting a baseline of 3.8 out of 5.0, you may choose a target of achieving a 4.2 out of 5.0 before the project is over. Another option is to look for a 10% improvement in the baseline, once the original metrics have been collected.

Keep in mind that the baseline metric may not be easy to define. In some cases, you may want to accumulate a number of initial metrics before declaring your baseline. For instance, you may want to track defects on a system for three months and then take the average number per month as the initial baseline. This would then be the number which you would try to improve upon in the future.

Be Creative in Looking for Ways to Measure Business Value (2.7.2T.P7)

One of the holy grails of metrics is to be able to accurately capture the business value produced by a project. In some cases the value is obvious. For example, sales could increase, inventory levels could be reduced or fewer people may be needed in a process. However, in many projects, the business value can be difficult or impossible to quantify exactly. Some common problems include:

The project produces soft benefits, such as improving client satisfaction or product quality.
The project involves infrastructure that is used by large groups of people. For instance, how much more productive will people be if you double the memory in their desktop computers? What is the quantifiable value provided with a new internal phone system? The fuzziness of these questions makes it hard to quantify a business benefit.
The project results in an increase in the amount of information people have available. It's hard to know exactly how the information gets leveraged to produce better decisions.
Things improve as a result of multiple projects over a period of time, but it is hard to know exactly how much value each project delivered.
The results are improvements at a low level that are hard to meaningfully roll-up. For instance, eliminating steps in a process. That process takes less time, but the time gets taken up by other work.

The best approach to determining the business value of a project is usually to look at a value proposition or Business Case that was completed before a project began. Look at how the benefits were quantified in this document to see if you can actually measure similar results when the project completes. If there are hard benefits identified, the metrics should be able to show how much value was delivered. If there were soft benefits identified, you will probably need to stick with anecdotal, survey and indirect evidence of the value provided.

If the project was sponsored by a client organization, they should take the lead on capturing the follow-up metrics. However, there may be follow-up required to make sure that the metrics are captured. The specific metrics to capture are based on the metrics that were used to justify the project to begin with. Hopefully, that cost can be easily compared with the estimated cost on the Business Case. However, the benefits of the project typically do not start until the project is completed, so these must be captured after the fact. In many instances, the project team may have disbanded, so the tracking, or helping the client organization to track the benefit, will usually be a part of the support organization.

Gather Subjective Metrics with Client Satisfaction Surveys (2.7.2T.P8)

Gathering metrics is important because it allows you to see how you are performing against the expectations of your clients. If the world were perfect, all of the metrics you collect would be factual, relevant and accurate. However, in many cases it is impractical or cost-prohibitive to try to gather exact and quantitative numbers. One way to supplement any quantifiable metrics is with client satisfaction surveys. For more information on surveys, see 2.7.3T Collecting Metrics Using Surveys.

Collecting Project Metrics and Demographics Can Assist on Future Projects (2.7.2T.P9)

Gathering and reporting a consistent set of metrics at the end of a project can help your organization see the trends for delivering projects over a period of time. The metrics should show how well project teams are meeting their commitments in terms of quality, cost and duration. As more and more projects report the metrics, a baseline will be established that will allow your organization to see how it is improving, or slipping, over time.

If you collect metrics on an organization-wide basis, you should collect some project demographics in addition to the actual metrics. Project demographics are just predefined project characteristics that provide a description on each particular project. If you store the demographics and metrics in a file or database, they can be analyzed to show the overall trends in a more discreet and granular way.

If you do not collect demographics, you can still compare actual cost to deliver versus the estimated cost. The advantage of gathering some project demographics is that you can compare similar projects. For instance, you can compare the cost and effort associated with delivering mainframe development projects versus web development projects. Or you could compare how well projects from the Sales organization meet their deadlines and budgets versus projects from the Finance area.

The other benefit of gathering project demographics is that you can use the information as input into future project estimates. For instance, let’s say you have marketing campaigns for different products. Let’s also say that you have been collecting project estimations and the actual results for cost and duration. If you had this information, you could estimate the cost and duration of a current product marketing campaign by comparing your project with similar projects that were completed in the past. This may be of help for estimating your project. The demographics you capture from completed projects can be used to search on later for new projects. (A sample worksheet to capture project demographics and metrics is available in the Template Library.)

Make Sure Your Metrics Tell a Complete Story (2.7.2T.P10)

In many instances the project team publishes the results of a metric in a way that does not allow the reader to fully understand whether the results are good or bad. This is because the reader sees a metric, but they do not understand the target and they do not always understand what the metric is trying to achieve. The project manager and project team may know what a given metric is telling them, but other readers of the information may not. One way to help is to always report the metric along with the target. For instance, if you report your current expenditures to date, also include your expected expenditures at this point in the project. If you report that your project has spent $100,000 so far and your total budget is $150,000, the reader still does not have the context to know whether this is a good or bad situation. Sure you are under budget, but the work is not complete either. The better way to report this information is to state that you have spent $100,000 to date and that according to your estimate you should have spent $110,000 at this point in the project. If the trend continued, you estimate the final cost of the project to be $135,000 versus your budget of $150,000. If you report the metrics with this context, your readers can understand what the numbers are saying.

Communicate and Train the Team in the Purpose and Value of Metrics (2.7.2.P11)

If you talk to your team about the importance of building a quality product, they will typically understand what you are saying. Quality is a term and has a connotation that everyone is familiar with. The general definition of “metrics” is not so obvious. The project manager may be trying to create a metrics program for a large project, while the project team doesn’t make the connection between gathering metrics and the business value received. This disconnect may affect the client as well. The client may not intuitively see that you need to measure to gain a factual sense for whether deliverables and processes are meeting expectations. The client may also not see the connection between gathering metrics and the ability to improve a process.

The project manager should take the time to explain why metrics are needed and how the information collected will help drive improvements. Likewise, the team should understand how to look for metrics that will provide an indication as to the state of a process or a deliverable. Educating the team and the client will help the project manager obtain better metrics with less work effort and less pushback. If your organization has an overall quality focus, this education should be done for the entire staff.

Use Critical Success Factors (CSF) and Key Performance Indicators (KPI) to Communicate Metrics with Your Clients (2.7.2.P12)

The terms critical success factors (CSF) and key performance indicator (KPI) are used to indicate success targets. (Key performance indicators are sometimes also called key business indicators (KBIs).) The terms normally indicate something very important to measure as it relates to your business. Many organizations use the two terms to mean the same thing, but there is a slight difference.

Key performance indicator (KPI). Usually key performance indicators focus on processes. For example, one of your KPIs might be to reduce the time it takes to ship an order to a client. This is a process-related success measure so it is a quality assurance metric.
Critical success factor (CSF). A CSF on the other hand, usually has to do with the characteristics of a deliverable. For example, you may have a CSF associated with reducing the number of defects in one of your products. This would be a quality control metric.

Metrics are designated as CSFs and KPIs if there is a sense that they are critical to the success of your company or your organization. The general idea is that your organization will not successfully achieve its objectives unless these CSFs and KPIs are met.

Identify your Work Products (Deliverables) as Success Outcomes (2.7.2T.P13)

When you defined the work, you established a set of project objectives. If your project achieves these objectives, you should be able to consider the project a success. Likewise, you defined a set of deliverables that you need to build to satisfy these objectives. This establishes a link between your deliverables and your project success.

When you create your Project Scorecard, the first thing you need to understand is your project success criteria. Part of your success criteria should be the completion of your project deliverables. If you successfully complete your deliverables to the appropriate level of quality, you will have at least one component of the overall success criteria on your scorecard.