To improve the performance of your business, measuring project success can be a helpful tool. But you must avoid pitfalls that might turn it into just another useless project management ritual.
Pitfalls of Project Success Measurement
As with all measurements, project success can fall prey to the dreaded Goodhart Law:
“When a measure becomes a target, it ceases to be a good measure.”
As we just saw, aiming for perfection in terms of project management shouldn’t be done at all costs. Having balanced criteria for project success will help you keep, but you must be mindful of some challenges you might face while trying to define the right criteria for your project.
To do that, it’s helpful to check against some of the following pitfalls.
Pitfall 1: Conflicting criteria
Some of your criteria might be directly in conflict with one another in certain situations. When those conflicts arise as you try to evaluate the success of your project, things may turn sour as each stakeholder tries to defend the criteria that matter most to them.
Solution: Prioritize your criteria to resolve conflicts rapidly and take into account the power of all stakeholders.
Pitfall 2: Too few criteria
When you focus on a small set of success criteria, you might overlook other criteria that are just as important. Worse, you might not just overlook them, but negatively impact them.
The classic example of a narrow focus for success is the dreaded “Cobra effect”:
The term cobra effect originated in an anecdote set at the time of British rule of colonial India. The British government was concerned about the number of venomous cobra snakes in Delhi. The government therefore offered bounty for every dead cobra. Initially this was a successful strategy as large numbers of snakes were killed for the reward. Eventually, however, enterprising people began to breed cobras for the income. When the government became aware of this, the reward program was scrapped, causing the cobra breeders to set the now-worthless snakes free. As a result, the wild cobra population further increased. The apparent solution for the problem made the situation even worse.
Solution: Ensure you’re picking success criteria that cover the full definition of success and that balance each other to guarantee you avoid creating the wrong incentives.
Pitfall 3: Too many criteria
Conversely, too many success criteria could be a mistake. The more criteria, the more possibilities for conflicts.
While you must ensure that your criteria cover enough aspects of your projects to give a true picture of its success or failure, evaluating project success should be an activity that takes you a reasonable amount of time.
It could be integrated in a post-mortem meeting, although you want to ensure you do not fall prey to group think and keep everyone honest about their rating.
Solution: Limit yourself to a manageable amount of success criteria to keep the process manageable, and ensure you agree on the criteria and their meaning with all stakeholders.
Pitfall 4: Ambiguous criteria
If you’re going to have the stakeholders fill in project success information, you need to ensure that the data is reliable for it to be worth the trouble. Unfortunately, some success criteria often remain at too high a level to be evaluated precisely.
However, pairing each success criteria with a quantitative metric and a precisely defined way to measure that metric can help you overcome that ambiguity.
Solution: Disambiguate criteria by being clear about how you're going to measure criteria attainment and by using quantitative metrics.
Pitfall 5: Static criteria
To evaluate your product success, you might have to use different criteria for each project. Defining success shouldn’t be seen as a static thing, and you should ensure for each project that you're using the best set of criteria.
For example, some of your projects might have an essential environmental component that would warrant including it as a criteria on its own (e.g. "Project had minimal impact on environment", "Project is sustainable", and so on).
Solution: Re-evaluate success criteria for each project (but don’t fall for Pitfall 3!).
Pitfall 6: Evaluating success in a binary way
Success isn’t a yes/no question. Since projects are complex entities with many stakeholders, it’s hard to give a black or white answer about them.
Similarly, you could look at two projects that are successes, but success of different magnitudes: a small and a huge successful project can’t really be considered equivalent, and your measure of success should take that into account somehow.
Solution: Grade projects on a more continuous spectrum, for example by giving each criteria a score on a scale of 1 to 10, or the number of days the project is delivered late.