The BA Practice Lead Handbook 9 – Measuring the Effectiveness of your Business Analysis Practice

Written by Kathleen B. Hass, PMP on July 16, 2013. Posted in Articles.

In previous articles, we discussed ultimate measures of business analysis success: value to the customer and wealth to the bottom line of your organization. We stated that the real work lies ahead; the work to ensure sustainability, continuous improvement, and delivery of real business benefits. We focused on:

Continually increasing the capabilities of your BA team and the maturity of your BA practice,
Measuring the business benefits of your BA practice and of projects in terms of value to your customers and/or wealth to the bottom line, and
Creating and executing a strategic communication plan to demonstrate the value of business analysis to all stakeholders.

That all seems well and good, but there must be more measures of BA practice success. In article #7, Tom T. posted a comment that asked these important questions about metrics and measurements of business analysis performance, which are essential to the sustainability of your BA Practice (see BA Practice Framework below). The commenter asked:

How can I measure and manage the productivity and quality of my BA team?
Should there be a “standard” timeline for producing requirements? I doubt it, but there should probably be some sort of guideline in estimating how long the process should take.
Are there specific quality measures which can and should be implemented? Probably, but they all “add overhead” to an already cumbersome and time-consuming process.
Without solid historical evidence (of our own), how do I justify the need for additional reviews and inspections?
How do I even get people to accept the need for compiling solid historical evidence?
For me, it’s not about the requirements tools, it’s about the oversight. Tools and templates for that don’t seem to be nearly as abundant.

The BA Practice Framework

Sustainability. Demonstrate value through performance measures

So let’s examine the questions posed by Tom T. His first question: ‘How can I measure and manage the productivity and quality of my BA team?’ is really two different questions, one about productivity of the team and the other about quality of the business analysis products and services.

Productivity

According to BusinessDictionary.com, productivity is a measure of the efficiency of a person, machine, factory, system, etc., in converting inputs into useful outputs. Productivity is computed by dividing average output per period by the total costs incurred or resources (capital, energy, material, personnel) consumed in that period. Productivity is a critical determinant of cost efficiency.

According to wordnetweb.princeton.edu productivity is:

The quality of being productive or having the power to produce
In economics, productivity is the ratio of the quantity and quality of units produced to the labor per unit of time

So, productivity measures the cost to construct or manufacture outputs in a production environment when the same product or service is delivered repeatedly. I submit that this measure does not apply to business analysis deliverables. Each business analysis output is unique and tailored to the needs of projects. The question is not: ‘How many business analysis products can our BAs produce in a week?’ Indeed, in this world of scarce resources and time to market demands, the fewer and lighter the BA artifact the better.

Having said this, you as BA Practice Lead should begin to capture historical data on how long it takes to develop, manage changes, and validate requirements for projects of varying complexity. Refer to prior articles for information about the four levels of project complexity. Historical data will enable you to plan new efforts more accurately, and determine what types of projects require longer timeframes.

Quality

Quality, however, is another matter altogether. The quality of business analysis products and services is paramount to establishing and sustaining a successful BA Practice. According to Merriam-Wesbster.com, quality is:

A peculiar and essential characteristic
An inherent feature
Capacity
A degree of excellence
Superiority in kind
A distinguishing attribute

Therefore, the essential characteristics of high quality requirements artifacts must be identified and assessed to determine the level of quality. To do so, the BA Practice Lead needs to establish a quality assurance program for the systematic monitoring and evaluation of the various aspects of business analysis products and services to ensure that standards of quality are being met.

Characteristics of High Quality Requirements

Are there specific quality measures which can and should be implemented? Some say for requirements to be useable, they must consist a set of characteristics and attributes. It may be helpful to use a checklist similar to the one presented here to help quickly validate the quality of a set of requirements. Note: the checklist should be different for projects of low, moderate, and high complexity – obviously, the higher the complexity, the more rigorous the validation sessions. See former articles to determine the complexity level.

The characteristics of good requirements can also vary based on the specific business and technology domain being addressed. The following checklist provides the characteristics that are generally acknowledged.

Requirement Validation Checklist

Date:
Attendees:

Technical team members:
Business team members:
Author/BA:
Others:
Original initiating documents to be used to ensure completeness of requirements:
o Business Case
o Project Charter
o Statement of Work

ID Number	Quality Characteristic	Explanation	Criteria Met (Y/N)	If No, Defect and Rework Required Defined	Another Review Needed (Y/N)
	Clear	Requirement is clear and concise so it can be used by virtually everyone in the project. Selected types of requirements are expressed formally using technical language, e.g., legal, safety, and security requirement, and they are mapped back to the requirements that are more easily understood. However, in most cases the language used to document requirements is as non-technical as possible.
	Unambiguous	The requirement is concisely stated without recourse to technical jargon, acronyms (unless defined elsewhere in the Requirements document), or other esoteric verbiage. It expresses objective facts, not subjective opinions. It is subject to one and only one interpretation. Vague subjects, adjectives, prepositions, verbs and subjective phrases are avoided. Negative statements and compound statements are avoided.
	Visible	A diagram can express structure and relationships more clearly than text, whereas for precise definition of concepts, clearly articulated language is superior to diagrams. Therefore, both textual and graphical representations are essential for a complete set of requirements. Transforming graphical requirements into textual form can make them more understandable to non-technical members of the team.
	Unique	The requirement addresses one and only one thing. Each requirement is unique, describing an exclusive need to be met by the solution. Each requirement has an identifier that does not change. The reference is not to be reused if the requirement is moved, changed, or deleted.
	Complete	The requirement is fully stated in one place with no missing information.
	Consistent	The requirement does not contradict any other requirement and is fully consistent with all authoritative external documentation.
	Traceable	The requirement meets all or part of a business need as stated by stakeholders and authoritatively documented.
	Current	The requirement has not been made obsolete by the passage of time.
	Verifiable	The implementation of the requirement can be determined through basic possible methods: inspection, demonstration, test (instrumented) or analysis (to include validated modeling & simulation). Acceptance criteria describe the nature of the test that would demonstrate to customers, end users, and stakeholders that the requirement has been met. Acceptance criteria are usually captured from the end users by asking the question, “What kind of assessment would satisfy you that this requirement has been met?”

Attributes of High Quality Requirements

Attributes are used for a variety of purposes including explanation, selection, filtering, validating and assuring quality. Attributes allow the BA team to associate information with individual or related groups of requirements, and often facilitate the requirements analysis process by filtering and sorting. Assessing these attributes during quality reviews is one way to focus on the quality of business analysis requirements artifacts. BA Practice Leads should build a checklist of the attributes that are most important to their organization to build in the desired quality of BA products. The checklist may look something like this.

ID Number	Quality Attributes	Explanation	Criteria Met (Y/N)	If No, Correction / Refinement Needed	Another Review Needed (Y/N)
	Complexity Indicator	Complexity indicates how difficult the requirement will be to implement. Highly complex requirements have numerous interdependencies and interrelationships with other requirements. Complex requirements necessitate more rigor to define and model, and more verification to demonstrate their validity.
	Owner	Ownership specifies the individual or group that needs the requirement. The absence of ownership indicates the requirement may not be valid.
	Performance	Performance addresses how the requirement must be met; how fast the process must be executed.
	Priority	Priority of the requirement rates its relative importance based on business value. Low priority requirements typically have a low return on investment, and are likely not produced.
	Source	Source of the requirement identifies who requested it. Every requirement should originate from a source that has the authority to specify requirements.
	Stability	Stability is used to indicate how mature the requirement is. This is used to determine whether the requirement is firm enough to prioritize it and to begin work on it.
	Status	Status of the requirement denotes whether it is proposed, accepted, verified with the users, or implemented.
	Urgency	Urgency refers to how soon the requirement is needed to meet the business objectives, the market window.
	Functionality	The requirement can be implemented so that it performs the functions and features needed, and complies with the relevant standards. Issues related to data security are considered.
	Usability	The requirement is intuitive, easy to understand and to use. The customers/users must be able to perform their tasks in a consistent and efficient manner. The solution appears to be simple, and hides the complex technology from the customers/users.
	Reliability	The requirements have a high probability of failure-free operation of a business process in a specified environment for a specified time.
	Efficiency and Performance	The requirement can be implemented efficiently in the target environment, performing the tasks in an appropriate time frame while utilizing a reasonable amount of resources.
	Maintainability	The requirement is easy to maintain, enhance, and refine as the business need changes.

Characteristics of High Quality Requirements Development Processes

Another measure of quality is the process used to develop the requirements. If you don’t have a defined process, it is important for you to implement one for your BA team. The process should include the minimal types and numbers of requirements artifacts for projects of differing complexity. Remember, ‘just enough’ process is good enough. Perhaps your process should include the following steps. Again, it may be helpful to use a checklist during validation sessions to ensure your standard process is followed. A set of attributes are presented below for your consideration.

Planning requirements activities. Planning the number and type of elicitation sessions to be conducted and requirements artifacts to be produced. Plans should follow your organizational defined standard process. So, Should there be a “standard” timeline for producing requirements? If you plan well, and begin to capture actual data on how long it takes to develop, manage changes, and validate requirements for projects of varying complexity, you will have an idea of the timeline that is appropriate. How do I even get people to accept the need for compiling solid historical evidence? You simply need to require your BAs to develop plans for their activities, capture actuals, and begin to build historical data and a target timeline for projects of varying complexity. It is your job.
Studying requirements feasibility to determine if the requirement is viable technically, operationally, and economically
Trading off requirements to determine the most feasible requirement alternatives
Assessing requirements feasibility by analyzing requirement risks and constraints and modifying requirements to mitigate identified risks. The goal is to reduce requirement risks through early validation prototyping techniques
Modeling requirements to restate and clarify them. Modeling is accomplished at the appropriate usage, process, or detailed structural level
Deriving additional requirements as more is learned about the business need
Prioritizing requirements to reflect the fact that not all requirements are of equal value to the business. Prioritization may be delineated in terms of critical, high, average, and low priority. Prioritization is essential to determine the level of effort, budget, and time required to provide the highest priority functionality first. Then, perhaps, lower priority needs can be addressed in a later release of the system.

Characteristics of High Quality Requirements Validation Processes

Without solid historical evidence (of our own), how do I justify the need for additional reviews and inspections? Requirements validation is the process of evaluating requirement documents and models to determine whether they satisfy the business needs and are complete enough that the technical team can commence work to finalize solution design and begin development. Validation is not an option. It is essential to catch any omissions, errors or defects before further investment is made to convert the requirements into working processes and systems. Build a fast and efficient requirements validation process into your standard BA processes for the systematic monitoring and evaluation of business analysis products and services to ensure that standards of quality are being met. Justify it by assuring that defects found early are vastly less expensive to fix than those found in the test phase, or in production. IBM Systems Science Institute estimates that it is 100X more costly to fix a defect after deployment of the solution than in the design phase, and 15X more expensive in the testing phase.

When using the checklists, the validation team compares the set of requirements to the original initiating documents (business case, project charter, or statement of work) to ensure completeness. Include both business and technical representatives in the review process.

Business representatives focus on the clarity and accuracy of the requirements, and
Technical representatives focus on whether the requirements are sufficient to finalize design and begin construction of the business solution.

Beyond establishing completeness, validation activities include evaluating requirements to ensure that design risks associated with the requirements are minimized before further investment is made in solution development. An often-used analysis technique to help validate and understand requirements is prototyping to make the proposed solution to the requirements visible. Bring developers into the requirements process; use them to build validation prototypes during the requirements elicitation and validation process.

A word about validation and reviews: These sessions are designed to catch errors, omissions, and defects. We want to catch and eliminate any defects prior to moving into the development process. Catching and eliminating defects early is a good thing. It is also a learning process for all participants, contributing to continually improving the requirements development process and outputs.

Putting it all Together

So what does this mean for the Business Analyst?

For the individual BA, consider these strategies:

In the absence of organizational requirements validation checklists, build and use your own
Work with your PM to plan BA activities and deliverables, and capture actual time and resources used
Work with your BA Practice lead and BA team to improve the quality of your BA deliverables.

So what does this mean for the BA Practice Lead?

For the BA Practice Lead, consider these strategies:

Develop organizational requirements validation checklists for low complexity, moderately complex and highly complex projects and programs
Develop your standard BA practices for low complexity, moderately complex and highly complex projects and programs. Conduct quality assurance activities to ensure all BAs follow the standards
Begin to collect basic data on the time, resources, and quality of your BA Practice using the completed checklists and BA plans.

Don’t forget to leave your comments below.