Three BA Metrics You Should be Tracking

What do you know about healthcare.gov, the United States’ new online healthcare marketplace? No matter your political affiliation, you have to admit the launch of the website has been a giant fiasco.

When you visit the site, here are a few of the interesting things you might experience:

1. A waiting line. Yes, just like the Department of Motor Vehicles —If the system is busy, you will be put into a queue and have to wait before you can begin the application process.
2. No mobile capabilities, you need to use a desktop or a laptop for best results.
3. Great advice like: “If you’ve created an account previously and have had problems with it, create a new account with a new user name, email address, and password. Be sure to clear your cookies and cache before you create your new account.”

I am fascinated by this very public debacle. My BA curiosity kicks in and I want to know:

• Is it possible to have a smooth launch of a large, complex project with dozens of vendors trying to integrate project plans, requirements, designs, systems, etc.?
• Who decided to launch a broken system and what was their rationale? Did anyone advocate for prototyping, a pilot program or a phased approach?
• Is it faster and cheaper to launch a flawed system that needs several months of repairs or spend years trying to get the system right before launch?
• What skills or tactics could we use to prevent similar problems with our projects?

As our friends at Healthcare.gov repair, update, and maintain the healthcare marketplace, it seems likely they are tracking hundreds of standard metrics like number of applicants, number enrolled, number of defects, system downtime, system error rates and response times.

These metrics are important and offer a great snapshot of the current state of the system. Trend lines for these metrics might help you with system troubleshooting or system risk management strategies.

However, these system-focused metrics don’t offer insight to improve project management or business analysis processes. They don’t help you assess the value of the BA role or track stakeholder satisfaction.

Though you may not work on complex, high-profile projects like healthcare.gov all BAs, need to evaluate and improve their processes. Here are three metrics that help BAs gauge their effectiveness, value and efficiency:

1) Layers of Defects

Can you imagine the size of the defect log for a system like healthcare.gov?

As time passes, most large systems develop layers of defects. A project goes live with 10 defects. The fixes for those defects result in more defects, and so on. How many layers deep are your defects and missed requirements issues?

When I work with organizations and BAs, I establish this metric by asking, “What originated this work?” for every project/enhancement/bug fix. I often hear one of the following:

• “It’s from another enhancement that did not work out right.”
• “It’s a new requirement for a project that went live last month.”
• “This fixes a defect from a project that went live last year.”
• “This addresses issues from another defect that we misunderstood.”

Then I ask again, “What originated this work?” I get the same responses. In many cases, I ask this question multiple times and find more than FIVE layers of defects hiding under each project.

I am finding that many organizations spend 80% of their time and money working on projects that are essentially rework—features and functionality that originated from missed requirements, misunderstood requirements, poorly tested requirements, etc.

This never-ending maintenance work indicates weakness in requirements processes. The rework hogs resources that would be better spent on innovative and strategic capabilities.

Tracking the layer of defects or the percentage of work being done to fix something broken or missed will help organizations gauge the health of their BA and even testing processes. BAs should use this metric to determine how to “get it right the first time” and free up resources that will offer more strategic value to stakeholders.

The balance to this metric is if you are expecting rework that is due to an agile approach where the rework is really learning and discovery and part of the process. In this case, rework for these projects should be removed from the metric.

2) Business Sponsor’s Percent of Successful Projects

The definition of success varies from project to project. When I think about healthcare.gov I can imagine many different definitions of success:

• Project team: “The system will be delivered on time and under budget.”
• Consumers: “I want the system to be like the amazon.com of healthcare.”
• Democratic Party Leaders: “I want the system to help us secure the presidency for 8 more years.”
• Republican Party Leaders: “I want the system to fail miserably to help us secure the house, senate and presidency.”
• Media: “I want the system to cause controversy so I have news to fill my 24-hour cycle.”
• People who already have good healthcare insurance: “What system?”

BAs need to consider the definition of success from multiple perspectives. But whose perspective is most important?

When I ask leaders of PMO offices what percent of their projects are successful, they typically reply with an answer between 80-90 percent. When I ask the business sponsors the same question, time and time again I hear that only 20-30 percent of their projects are successful. Funny isn’t it?

PMO: 80%-90% of our projects are successful
Business Sponsors: 20%-30% of our projects are successful

So, I ask how they measure it?

• PMOs often measure by cost and schedule
• Business sponsors measure by: “Did I get the value I expected based on the money I paid?” or “Was the project worth doing—is my life better?”

The BAs primary job is to identify and protect the sponsor’s value proposition, so the success metric needs to be driven by the sponsor’s perception of value. It’s critical to the requirements and BA role.

In many cases, BAs need to set time and budget aside and work based on value.

In some cases value will conflict with time and budget (Ahhh, yes, healthcare.gov!). When you recognize this conflict, you need to partner with the PM and/or sponsor on your concerns.

How can BAs drive this success percentage higher?

• Understand what outcomes the sponsor values most and why.
• Look for inconsistences between the sponsor’s expectations and the project team’s expectations?
• Determine how to prevent inconsistencies.
• Determine how to front-load the value—give your sponsor value ASAP.

3) Rework Caused by Diving Into Details Too Soon

OK. I am dragging you back to healthcare.gov one more time. I can’t begin to imagine the pressure analysts in this environment felt as they began to outline requirements! I can only assume that SME resources were extremely limited and tight deadlines caused BAs to choose a path: all details or no details.

Based on my experience, most analysts on intense, complex projects jump directly from a high-level business need or scope document to the nitty-gritty details. They feel crunched for time or have limited access to resources, so they gather and document as much detail as possible without leading stakeholders through the big picture.

This nose-dive to details seems efficient, but usually results in and unmanageable amount of rework as the project progresses:

• You may have trouble completing deliverables as you need to re-write when you are still eliciting.
• Traceability becomes complicated and gaps arise when detailed requirements do not have connections to higher-level features or processes.
• Change control becomes a nightmare if you add design details to requirements documents. The design will change after your requirements document is base-lined, creating a change control headache for you with each update.

So ask yourself:

• How much time do I spend rewriting or updating requirements?
• Do I rewrite because of new things I learn during the elicitation process?
• Do I rewrite because of enhancements and defects?

Then, improve your rework metrics:

• Complete higher level modeling and requirements ahead of details.
• Capture details informally, but don’t add them to official project documents until higher level models are approved.
• Avoid managing traceability and change to a level of detailthat will change (i.e. screen designs)

Great BAs know how to guide their stakeholders through a progressive elicitation process that moves from concepts to details. They manage rework and change control by understanding when and how to offer details.

Which metrics do you find most valuable and why?

Don’t forget to leave your comments below.