Measuring Usability with the System Usability Scale

Beauty, it is said, is in the eye of the beholder. As proof, ask a group of friends or colleagues to verbally explain how to measure beauty, then sit back and watch the entertainment as people struggle to express verbally what is visually obvious to them.

Business Analysts face a similar challenge when we are asked to measure non-functional requirements. Measuring functional requirements is obvious, like recognizing the difference between black and white. Did the function achieve its desired outcome or not? When performing the function, were there any steps missing or extra actions required? Non-functional requirements, on the other hand, can be like measuring the various shades of gray that exist on the spectrum between black and white. Usability is a non-functional requirement that is associated with user’s impressions of the system design and impressions, like beauty, are often difficult to quantify. I am going to introduce you to the advantages of the System Usability Scale (SUS), which I have used with success on projects to help quantify the abstract concept of usability.

The System Usability Scale was developed over 30 years ago by John Brooke at Digital Equipment Corporation (DEC). It is a survey that contains 10 statements:

1. I think that I would like to use this application frequently
2. I found the application unnecessarily complex
3. I thought the application was easy to use
4. I think I would need the support of a technical person to be able to use this application
5. I found the various functions in this application were well integrated
6. I thought there was too much inconsistency in this application
7. I would imagine that most people would learn to use this application very quickly
8. I found the application very cumbersome to use
9. I would feel very confident when using the application
10. I would need to learn a lot of things before I could start using this application

Each statement has five potential responses based on a 5-point Likert scale – strongly disagree (1), disagree (2), neutral (3), agree (4), strongly agree (5) – where the numbers in parenthesis are the point values earned by each response. There is a 3-step process to convert the raw responses onto a 100 point scale:

---

---

1. Responses to odd statements: subtract one from each point value
2. Responses to even statements: subtract each point value from 5
3. The results are now normalized from 0 to 4. Add up the total normalized points and multiply by 2.5 to convert from a 0 to 40 point scale to a 0 to 100 point scale.

A SUS score of 68 is considered to be average. The “Measuring Usability with the System Usability Scale” article has more information about how to evaluate scores.

There are a couple of things to keep in mind when using the System Usability Scale. First, I would be cautious when using the SUS to compare something that is familiar with something that is unfamiliar. In my experience, users that were critical of a new application, a new process or some other change to their way of organizing their work adapt to the change over time. Eventually, years later, when faced with another change to the item in question, the critics have transformed to champions, much to the bemusement of the Business Analyst whose long memory recalls the earlier resistance to change. An axiom I state to project sponsors or leaders of a change initiative is “users hate change”. SUS practitioners have found that users are indeed more favorable towards the known quantity, skewing the usability results more positively towards the old and familiar as opposed to the new and different. When presenting your analysis in this situation, you need to make your audience aware of this inherent bias.

Second, while the SUS is a good, broad measure of usability, you may need to apply other analysis techniques from your Business Analyst toolbox to help evaluate its results. For example, I have had good success applying the SUS when a team is evaluating different Commercial-off-the-Shelf applications or getting initial feedback from user training. However, even if one application is clearly considered more usable than another or trainees are more critical than you had hoped, using the results without further analysis may lead to wrong conclusions. Are there characteristics of your survey group (age, gender, company experience, etc.) that can help explain the preference? Is a training group more negative because the implementation of their department’s requirements were deferred due to cost or schedule constraints? As discussed before, are users negative about the usability of an application because it requires a completely new, unfamiliar device or process than other options? This more detailed analysis will help you recommend potential mitigation actions to address the specific needs of your situation.

If not handled properly evaluating usability can turn into a debate based on subjective views that are impossible to analyze. The System Usability Scale is a tool that Business Analysts can use to help make usability a more objective quantity, opening up options to apply further analytic techniques to provide a more complete assessment to our stakeholders.