Skip to main content

Know When to Say When; How Not to Get In Over Your Head With Metrics

KnowWhenToSayWhen1As business analysts we can agree that benchmarking is important, but from that point on we’re likely to find the conversation diverges. Differences abound in approaches of how and what to benchmark in order to prove value. Organizations become overzealous in what they want to benchmark and scorecard in their drive to create greater efficiencies. However, a key component for developing and monitoring successful metrics is ensuring that they are in alignment with the maturity of an organization. Knowing when to say when can enable less mature organizations to develop metrics that are both useful and appropriate at the developmental level.

Benchmark People and Process First

In a previous BA Times article on managing metrics, the author and my esteemed colleague, Keith Ellis, made the case for creating multiple point metrics for people, techniques, process, technology, organization and documentation standards. While creating multiple-point metrics may be the best possible scenario, it is an enormous undertaking and, given the maturity of an organization, the least likely possible scenario the organization can accomplish successfully. An organization that lacks BA maturity would do well to start with a simple approach to metrics that considers people, process and tools, with a greater emphasis on the first two.

I highly recommend forming a team of individuals to work together, ideally in a requirements workshop setting, to determine the metrics to be monitored. Select a maximum of five things to measure for a benchmark. Keep in mind that the metrics don’t have to be exact. A “swagged” or estimated number calculated with simple mathematics and a plus/minus degree of accuracy can be used when there are no specific data elements readily available and should be acceptable until a level of maturity is reached. As the organization begins to mature, so too can the details being measured, the formulas used to calculate those measurements and the accuracy of the metrics.

Processes are interdependent and complex, so restrict yourself to three to five processes. Basic measureable process-related activities could include such items as time to complete an iteration, number of change requests and total number of iterations. Also, look at the solution development lifecycle and pick one or two that are bleeding the most. Finally, schedule benchmarking on a monthly basis. One year out is too late for a remedy, and with fewer metrics to benchmark, you will find the task is quite manageable.

Tools really shouldn’t play a role in developing and monitoring metrics, but they invariably do when an organization begins to cloud and confuse what level of metrics should be assessed. It’s important when using this simple approach to remember that knowledgeable people and refined processes are needed before you can select a tool. A tool is only as good as the people and process using it. A simple SWOT (strengths, weaknesses, opportunities, threats) analysis will help a team to quickly ascertain appropriate people and process metrics.

Scorecarding Metrics

Scorecarding is very important for measuring how well activities are being executed and individuals are performing. It should be related to goals and objectives and show a demonstrated relationship with benchmarking to people and process (and tools, if they’ve somehow gotten into the mix). However, scorecards are often put in place and not used because they’re so complicated or they don’t relate back to benchmarks.

Even in singular projects, scorecarding should focus on one to three things. Take, for example, two or three benchmarks that would be measured for running a requirements workshop. One workshop iteration takes 40 hours of effort for planning, execution, validation, etc. The next workshop is conducted after training people and reduces the workshop effort to 20 hours. The scorecard would indicate a 50 percent increase in efficiency.

Project Comparison

Side-by-side paired project execution has the potential to be a valuable benchmarking tool, as long as it doesn’t become too burdensome or complex. Keep in mind that no two projects are identical so give yourself some leeway in finding comparable projects. For instance, a $100,000 project could be benchmarked against a $150,000 project. Then, follow two or three things from project to project and be prepared to evaluate five to ten projects that are similar in nature.

Ongoing Reviews

Peer reviews, contract reviews and structured walkthroughs can increase efficiencies more effectively than look-back reviews conducted after the fact. The reviews should be done through the entire process as iterative functions so that by the time you’ve gotten through a project, you should be confident in your scorecard data and benchmark comparison.

Toward Positive Financial Impact

As an organization matures, overall project costs, return on investment, internal rate of return and the time spent on validating output of interviews, requirements workshops and other BA functions will build toward performance improvement, not only for individual projects but also for project portfolios. This will enhance the value of business analysis and the positive financial impact that can be realized. An organization that lacks in BA maturity should take a simple approach to measure its performance. When it comes to requirements metrics, don’t try to run before you can walk.

Don’t forget to leave your comments below


Glenn R. Brûlé, CBAP, CSM, Executive Director of Client Solutions, ESI International brings more than two decades of focused business analysis experience to every ESI client engagement. As one of ESI’s subject matter experts, Glenn works directly with clients to build and mature their business analysis capabilities by drawing from the broad range of learning resources ESI offers. ESI, a subsidiary of Informa plc (LSE:INF), helps people around the world improve the way they manage projects, contracts, requirements and vendors through innovative learning. For more information, visit www.esi-intl.com.