To improve the performance of your business, measuring project success can be a helpful tool. But you must avoid pitfalls that might turn it into just another useless project management ritual.

 

Why Measure Project Success?


That might seem like a stupid question, but what’s the main point of measuring project success?

After all, once a project is over, nothing more can be done about it, whether it was a success or a failure.

That was the prevailing philosophy for a long time, but as the concept of learning organizations developed, the value of recording information about the past of an organization to drive its future strategy became more prevalent.

Today, the value of continuous improvement based on past data is acknowledged, and it is through rituals such as post-mortem meetings that companies can understand what went wrong during a given project so that they can fix it for the next projets.

By incorporating new learnings — particularly about successes and failures — into new processes, fewer mistakes are made, and higher performance can be achieved.

That’s why measuring project success has become a major problem for all practitioners of project management. To such an extent, in fact, that the Project Management Institute has devoted one of their Annual Seminars to that question.

Measuring project success is now seen as a staple of high-performing organizations, as Elizabeth Harrin puts it:

“Successful organisations take the guesswork out of this process: they define what success looks like, so they know when they have achieved it.”

The issue with measuring project success is that it’s not as easy as it sounds.

What’s so Hard about Measuring Project Success?

It seems reasonable that by measuring project success, you will gather data about what went particularly well or badly with your projects, and that you’ll be able to identify success factors to apply to all your projects.

Let’s say you’re convinced you should try measuring the success of your projects: Is it all that hard? Isn’t a project delivered “on time and on budget” a successful project?

Well, maybe. But maybe not.

What if the project is delivered on time and on budget… but 3 months later, the client turns out not to get any benefit from the project? Or they’re even dissatisfied with it?

Conversely, some clients might be entirely happy with the result of a project that was a near disaster in terms of project management (unforeseen delays, overworked resources, unprofitable project for the company, and so on).

Since the 1960s, project managers have been trying to get to a more precise notion of “project success” and have tried to identify what it takes to guarantee project success.

A lot of research has been done, and to write this article, we ourselves went through dozens of academic papers on the topic — from the first few papers written on the topic to state-of-the-art research done on how to optimize project success with genetic algorithms.

Sadly, the results have been more than disappointing, as Terry Cooke-Davies stresses in The “real” success factors on projects :

In spite of these well-known research results and despite column-miles of words that have been written about project management [4], despite decades of individual and collective experience of managing projects [5], despite the rapid growth in membership of project management professional bodies and despite a dramatic increase in the amount of project working in industry, project results continue to disappoint stakeholders.

Some have even started to question the value of project management itself, leading to a host of studies on the benefits of good project management.

It doesn’t help that project success is eminently subjective, and that in situations that involve many stakeholders with different or conflicting interest, everyone might have a different perception of the project and its result.

Luckily, some best practices have still emerged that should help you measure the success of your projects using simple processes.

How to Start Measuring Your Project Success

To start off on the right foot, we need to understand what we’re trying to do here:

We simply want to understand when we can classify a project as being a success. In other words, we want to understand what to measure to be able to say conclusively that a project was a success rather than a failure.

We’re looking for success criteria that will help us verify a project has been successful when it’s over. Success criteria are “a definition in measurable terms of what must be done for the project to be acceptable to the client, stakeholders and end-users who will be affected by the project.”

That concept must be contrasted with success factors, which are the causes that explain how success criteria can be met.

Over the years, measuring project success has gone from an informal, unimportant process, to a more formalized and fundamental aspect of project management.

Starting from the 1980s, the development of Critical Success Factors (CSFs), which started as simply checklists to verify customers and other project stakeholders were happy, to more intricate lists of criteria encompassing time to deliver, costs, quality, and gradually started to add “softer” criteria: clear communication, clear objectives and scope, etc.

From the growing lists of CSF came frameworks, which gradually expanded to include aspects that went beyond pure “operational success”, for example by taking into account business success (economic value created, etc.).

Although the complexity of the theory around measuring project success has increased, the fundamentals remain simple.

To start measuring your project success, we recommend you pick a selection of criteria, and ensure that they cover all aspects of project success by picking them from all important categories and stakeholders.

For example, as a way to remedy the issue highlighted above, you will want to include both project management and product success criteria.

Project management success criteria

All criteria that pertain to the project itself and its accomplishment. They help you measure the internal efficiency of your organization to deliver projects. Here, you’ll generally want to cover the project management “classics” and ensure that :

  • Project is completed on time
  • Project is completed within budget
  • Project meets quality targets

You can add more project management criteria such as :

  • Project includes all items within scope
  • Project satisfies commercial objectives in terms of revenue and profit

Or include criteria about different stakeholders:

  • Project team satisfaction
  • End-user satisfaction
  • Supplier satisfaction

Product success criteria

All criteria that deal with the effects of the project’s final product. They’re more of an external metric that helps you gauge your effectiveness at getting customer satisfaction.

Product success criteria can include such things as :

  • Product meets all end-user requirements
  • Client satisfaction (measured for example with NPS)

If the product delivers several benefits, it helps to have precise criteria that measure the attainment of each of those benefits.

Some project management consultants also advocate separating project controls (on time, on budget) from project success criteria, which must be specific to the benefits and the outcome you're looking for. 

An example of a success criteria as given in the preceding video would be :

  • We accurately invoice all service calls within 5 days of completion

To do the job of picking your criteria properly, it helps to avoid some classic pitfalls. Here are the 5 main pitfalls, along with strategies to circumvent them.

Pitfalls of Project Success Measurement

As with all measurements, project success can fall prey to the dreaded Goodhart Law:

“When a measure becomes a target, it ceases to be a good measure.”

As we just saw, aiming for perfection in terms of project management shouldn’t be done at all costs. Having balanced criteria for project success will help you keep, but you must be mindful of some challenges you might face while trying to define the right criteria for your project.

To do that, it’s helpful to check against some of the following pitfalls.

Pitfall 1: Conflicting criteria

Some of your criteria might be directly in conflict with one another in certain situations. When those conflicts arise as you try to evaluate the success of your project, things may turn sour as each stakeholder tries to defend the criteria that matter most to them.

Solution: Prioritize your criteria to resolve conflicts rapidly and take into account the power of all stakeholders.

Pitfall 2: Too few criteria

When you focus on a small set of success criteria, you might overlook other criteria that are just as important. Worse, you might not just overlook them, but negatively impact them.

The classic example of a narrow focus for success is the dreaded “Cobra effect”:

The term cobra effect originated in an anecdote set at the time of British rule of colonial India. The British government was concerned about the number of venomous cobra snakes in Delhi. The government therefore offered bounty for every dead cobra. Initially this was a successful strategy as large numbers of snakes were killed for the reward. Eventually, however, enterprising people began to breed cobras for the income. When the government became aware of this, the reward program was scrapped, causing the cobra breeders to set the now-worthless snakes free. As a result, the wild cobra population further increased. The apparent solution for the problem made the situation even worse.

Solution: Ensure you’re picking success criteria that cover the full definition of success and that balance each other to guarantee you avoid creating the wrong incentives.

Pitfall 3: Too many criteria

Conversely, too many success criteria could be a mistake. The more criteria, the more possibilities for conflicts.

While you must ensure that your criteria cover enough aspects of your projects to give a true picture of its success or failure, evaluating project success should be an activity that takes you a reasonable amount of time.

It could be integrated in a post-mortem meeting, although you want to ensure you do not fall prey to group think and keep everyone honest about their rating.

Solution: Limit yourself to a manageable amount of success criteria to keep the process manageable, and ensure you agree on the criteria and their meaning with all stakeholders.

Pitfall 4: Ambiguous criteria

If you’re going to have the stakeholders fill in project success information, you need to ensure that the data is reliable for it to be worth the trouble. Unfortunately, some success criteria often remain at too high a level to be evaluated precisely.

However, pairing each success criteria with a quantitative metric and a precisely defined way to measure that metric can help you overcome that ambiguity.

Solution: Disambiguate criteria by being clear about how you're going to measure criteria attainment and by using quantitative metrics.

Pitfall 5: Static criteria

To evaluate your product success, you might have to use different criteria for each project. Defining success shouldn’t be seen as a static thing, and you should ensure for each project that you're using the best set of criteria.

For example, some of your projects might have an essential environmental component that would warrant including it as a criteria on its own (e.g. "Project had minimal impact on environment", "Project is sustainable", and so on).

Solution: Re-evaluate success criteria for each project (but don’t fall for Pitfall 3!).

Pitfall 6: Evaluating success in a binary way

Success isn’t a yes/no question. Since projects are complex entities with many stakeholders, it’s hard to give a black or white answer about them.

Similarly, you could look at two projects that are successes, but success of different magnitudes: a small and a huge successful project can’t really be considered equivalent, and your measure of success should take that into account somehow.

Solution: Grade projects on a more continuous spectrum, for example by giving each criteria a score on a scale of 1 to 10, or the number of days the project is delivered late.