Engineering IX Perspectives

Using Metrics to Improve Project Quality

Any agile workplace (IX Engineering included) can and should regularly make a conscious effort to strive for improvement, as constant advancement is key to having an agile mindset a practice that metrics can both help and hinder. The right metrics can showcase and localize problems ripe for improvement or reassure teams their work is effective, while the wrong metrics can incentivize poor behavior…

How so? Think of the old anecdote (which later made its way onto a Dilbert comic strip) about the company with a product quality issue. Their metric and incentive for fixing this issue was based on ensuring Quality Assurance (QA) flagged every bug — they’d get a cash bounty for reporting problems, and engineers would reap the same reward for creating a fix. It’s a solid idea in theory, but in practice, it led to engineers coding in bugs, telling QA where to find them, and then fixing the problem they’d manufactured for the promise of a cash reward.

In other words, poorly selected metrics might push teams to game the system. And even if your team is free of bad actors, they might provide an incentive to prioritize the metric (i.e. bugs fixed) over the goal (quality improvement). This is one of the many reasons why it’s paramount not only to collect the right metrics, but to collect those metrics in the right way.

Why collect metrics at all?

To quote Lord Kelvin, “If you can’t measure it, you can’t improve it” a phrase succinctly capturing the importance of metrics. Simply put, metrics have the capacity to improve every aspect of a team and business, from communication channels to processes. They empower teams to identify challenges and problems in the pipes, allowing them to track the effectiveness of any changes or solutions implemented.

How can teams evaluate metrics?

Regardless of which metrics we’re using here at Index, our process generally remains the same:

  • Determine the goal
    • Metrics should always support the goal, though they shouldn’t be the goal. When told “measure X,” always ask “why?” A well-thought out goal allows us to select the right metrics. For instance, one might say, “We want to be confident that our code performs as expected and that all existing code will continue to do so as we integrate bug fixes and new features.”
  • Determine which metrics support that goal
    • Select 2-4 metrics that support your goal, selecting those with the lowest risk and cost for the highest value. When evaluating whether or not a metric is right for your team, a few questions to ask yourself might be:
      • What is the value of measuring that metric?
      • What risks are associate with measuring this?
      • How can it be gamed?
  • Collect the metric for an appropriate period of time
    • Consider the nature of the goals you’re hoping to reach and the improvements you’re hoping to make, then set appropriate timelines for collecting these metrics. For example, measuring “field reported bugs over the lifetime of a release” for a single sprint is unlikely to be helpful or impactful. Similarly, an improvement from a team process-related change is usually visible within 2-3 sprints.
  • Check in on the team’s progress
    • If your goal has not been met, make changes and continue measuring your metric. If it has been met, stop measuring and take a step back. While it may be tempting to grow your list of metrics, it could cost your team more time and fatigue (and there will always be an opportunity to revisit these metrics later). Instead, select a new goal and restart the process.

By and large, selecting the right metrics (and then executing correctly against them) allows teams at Index Exchange to set aggressive goals and track their improvements and success along the way. We trust that a strong measurement strategy is the key to driving improvement and growth, no matter the project or circumstance.

Leave a Reply

Your email address will not be published. Required fields are marked *