Skip to main content

Putting the User Back into User Acceptance Testing

Why is getting users involved in User Acceptance Testing (UAT) so challenging? Isn’t it called UAT because the users are the main participants? My experience has shown that involving users in all phases of the project, especially UAT, is the best way to ensure project success.

I recently worked on a project in which a major defect was found after the software application moved to production. This defect caused the users to perform three days of manual processes. Users on the IT project team worked countless overtime hours. The defect also resulted in a frustrated user group and business sponsor. The project team’s morale was low and the business users lost a great deal of confidence in the project team’s ability to deliver quality software solutions. To reduce the risk of making this crucial mistake in the future the project team improved the UAT approach by getting users more involved.

Traditional Approach

Too often, User Acceptance Testing is not taken seriously. For many reasons UAT gets shortened, is not conducted in a way that ensures a successful project or, the worst scenario, is not done at all.

An approach I have used in the past consisted of the project team members—Business Analysts and QA Analysts—writing test scripts and providing demos of the new application to the users. The users would then walk through test scripts step-by-step. In some organizations the BAs write and execute the UAT tests and then present the results to the users for sign-off.

Traditional approaches are often not effective because they are missing a key ingredient—the right amount of user involvement. In the project previously mentioned, there were five major issues relating to UAT that we had to address. These are common problems in many organizations related to lack of user involvement:

• Users may not be fully vested in UAT. In the traditional approach the users are directed by the BA during UAT and are brought in too late in the project to have an impact on the test plans. This results in a lack of ownership by the users and less responsibility on their part for the success or failure of the project.

• Users do not fully understand how the new functions should work when they are asked to test them. Just seeing a demo is typically not enough. This can result in the UAT session becoming a training opportunity and not a true test.

• Tests are often generic and are not all based on real-life scenarios. If the test scripts are written by the IT project team, there is a greater risk for missing real-life scenarios. This is because, unlike the user, the IT project team does not use the application every day.

• Project team members are usually pressed for time. Often a BA has already been assigned to perform requirements activities on another project during UAT. Balancing multiple projects means that BAs have a hard time focusing on UAT, while meeting their other project deadlines.

• High pass rate of UAT test plans. Ironically, this is not a positive thing. Often a BA writes test scripts and tests them himself prior to UAT to ensure the scripts pass. When a BA writes the test scripts the users are not given an opportunity to interject enough real-life scenarios to validate the system.

A Recommended Approach

To address these common issues and increase the chance for project success we need to take a new approach to UAT.

1. Involve key users early

Once Quality Assurance (QA) begins testing, UAT planning should start. Identify users who have a deep understanding of the business requirements and are change agents for the group. Identify all of the tasks that need to be accomplished, the owners of each task, and a high-level timeline. This will help ensure that all the right people are involved.

2. Provide hands-on training of the system for the UAT participants

Providing a demo is not good enough. Once QA feels the application is stable enough, give hands-on training to the UAT participants. It is critical to explain to the users that issues may arise because QA testing is not complete. Ask the users to stay focused on how the application works and not so much on the fact that it is not fully operational.

3. Use facilitation sessions to create test plans

Have the users write their own test plans. This may sound far-fetched, but it is key to getting UAT as close to real life as possible. The BA’s primary role is to facilitate the UAT test plan creation process, but not to write a single test script. Using process workflow diagrams and Use Case documentation from the requirements package, ask the users to determine what processes and system functions need to be tested. Provide the UAT participants with examples of test scripts and explain the need to capture the goal of each test, the necessary steps, and the expected results. The steps become second nature to the users of the system, and they often find it difficult to document each step they take to accomplish a goal. Help them think through their processes in detail to ensure they have documented each task completely.

Review the test plan. Once the test plans are written, the BA reviews the test plans to ensure all the necessary functions and processes impacted will be tested.

Determine necessary inputs and outputs. Once all the test plans are written, ask the users to document the inputs they need to complete each of their test scripts and the outputs that will be generated. Make sure all UAT participants have the necessary inputs to complete their tests based on all of the outputs. If some are missing, enlist other users to create those inputs during testing execution.

Make it as close to real life as possible. To enhance the real life feel, the BAs work with the users to determine a testing schedule. Make sure the schedule follows their daily process. Again, use process workflow and Use Case documentation to ensure the test plans are executed in the order the activities would be done in real life.

4. Ensure users execute the test

The BA’s role is to ensure the test environment is set up and to assist the users as they execute the tests. The user’s role is to execute their tests and document the results.

The Results

Recommended Approach to UAT

1. Involve key users early

2. Provide hands-on system training for the UAT participants

3. Use facilitation sessions to create test plans

4. Ensure users execute the test

Using this approach can help reduce the risk of major defects making it to production and ensures the users are satisfied that the solution meets the objectives of each project. Here are some of the key results from using the improved approach:

• The UAT participants take responsibility for the success of the release. They feel part of the team due to their collaboration with the IT project team and involvement in UAT planning, and test creation and execution. They also help champion the benefits of each release to the larger user base.

• Due to the pre-test training, users are comfortable with new applications. This allows the users to develop real life test scenarios, and the time allotted for testing is not used for training.

• Since the users create and execute test plans, the tests are very close to real life scenarios and the users are more comfortable running the tests.

In addition there are tangible benefits to future projects:

• QA incorporates the scenarios documented by users in the QA test plans for future releases.

• Over time there is decreased use of the BA’s time for UAT. With the BA facilitating the UAT process and not doing most of the work, the BA can focus on other necessary tasks – like launching the next project.

Implementing the Approach

A lack of user involvement in UAT is not uncommon. I urge you to try this approach even if you have not yet experienced a drastic wake-up call, such as major defects in production.

As we are called upon to deliver solutions faster and faster, it is just a matter of time before major defects make it to production. Here are some tips for getting started:

• Start small. To help manage the changes with the new approach, identify a release with a low number of users and/or new functions. This will allow you to test the new process, discover lessons learned, and make the necessary adjustments.

• Plan for additional time. Using this new approach will initially require more time. Work with your Project Manager to plan more time into the UAT phase for your first two or three projects. As you get accustomed to this approach, it will require less time.

• Identify power users and champions of application. They are your best testers and have the most interest in the project’s success.

• Sell the benefits of the new approach to your users. As with any new approach, BAs need to help the users understand what the approach is, and how it will ultimately improve their business.

• Save the user test plans for future releases. Reuse of test plans will help speed up the time dedicated to UAT in future releases and can be used to update your QA test plans. Successful implementation of this approach helps ensure projects meet the user needs. The collaboration of users with the project team leads to a shared responsibility for the success and failure of the project.

What is UAT and Why We Do It?

UAT is the final approval by customers signaling the new system or enhancements can be deployed. UAT is unlike other types of software testing (e.g., unit testing, system testing, integration testing), because during UAT we are looking for conformity. We need to validate that the solution meets the business objectives and works correctly with real-life scenarios. UAT is typically conducted by users with assistance from the BA and other project team members.
UAT is most often conducted before a system is deployed into a production environment. For higher risk projects UAT may continue for a period of time while running the old system and new system in production. This gives the users ample time to become comfortable that the new system meets their needs. For commercial software companies UAT is also know as “Beta” testing. Here the system is launched into a production environment, but only to a subset of customers who will provide feedback on defects and necessary improvements. 

Comments (2)