Meaningful Metrics

Where is the line that separates process for the sake of process and process that adds value for most of the people most of the time?

How does one develop technology metrics that are meaningful to the majority of efforts without misrepresenting those that don't fit the measure?

I've run software development projects in a wide variety of environments and with a wide variety of methodologies. I've been a scrum master and I've guided traditional waterfall teams. And I've become increasingly convinced over the years that the only really meaningful metrics for teams are customer satisfaction-based. Metrics like agile velocity and traditional on time/scope/budget metrics can be useful as a guide as you progress through your efforts, but at the end of each project those metrics were not nearly as meaningful as how happy my customer was with the end product.

As you start any development effort take some time to identify how you know whether or not you succeeded. Then, choose the methodology that best supports that end game. Sometimes, speed is everything and other times predictability is everything. You can set yourself up best to succeed if you understand what is most important to each of the stakeholders you impact, and that includes your development team.

Then, you have to develop three sets of metrics.

The first set of metrics is in support of your chosen methodology. Are you measuring velocity accurately in an agile world and are you controlling scope and timelines in a traditional model? How are you controlling for quality? It matters little what methodology you chose as long as you ensure that you control for and measure the things about that effort that matter to your stakeholders and introduce risk to your efforts. Use this set of metrics to drive your effort to completion and success.

The second set of metrics is in support of measuring a project's success. Assuming that on scope, on time, and on budget is the definition of success for every project is just as bad as assuming waterfall or Scrum is the right software development methodology for every project. If your stakeholders indicate that revenue increases are the definition of success, then that had better be your success metric. And you had better find a way to measure that metric, even if it means more development work. There is always a way to isolate and measure those success factors if you build specifically to support those metrics. Let's use our increased revenue example as a test. How could you isolate revenue increases due to a web site change when there are a million other factors that could also impact revenue? Split A/B deployment and measurement is a potential solution that jumps to mind. But if you don't build specifically to support the measurement of that goal, then you'll never be able to quantify your effort.

Finally, the last set of metrics you need are ones that give you visibility across efforts. I recommend you develop a satisfaction scale that is independent of any specific success factor. Then, for each effort you undertake, map the project's success factors to degrees of satisfaction. Then include a subjective component to the satisfaction measure for an aggregate metric system that can measure different types of projects similarly independent of their methodology or definitions of success.

The bottom line is that measurement of your efforts is critical to your success. Just don't confuse metrics that facilitate the successful development of your software with the set of metrics that measure its success. They often are very different sets of things to measure. Give them both the attention they deserve and you set yourself up for success.