The ROI of Usability Testing
This is a reduced version of a paper published in Information Age, a journal of The Australian Computer Society in 2006, reproduced with permission of the Author and publisher. The full version contains a case study. The reference is Eklund J (2006) Usability Testing: Is it really worth it? Information Age Oct/Nov 2006
John Eklund, PhD MACS
Usability Testing as a means of ensuring the quality of software or websites is a process that is gaining mainstream acceptance as an essential component of the software development lifecycle, though it’s sometimes misunderstood by development teams. In this article, Dr John Eklund, a senior usability consultant, discusses usability testing and its ROI.
What is Usability Testing?
The objective of usability testing is to make computer systems easier to use, allowing users to undertake their tasks in a manner that is effective, efficient and satisfying. Usability as a goal is achieved when, “the people who use the product can do so quickly and easily to accomplish their tasks.” (Dumas and Redish, 1999). Usability focuses on the interaction between user and system, which typically takes place in some form of interface. In order to understand how it is done, it necessary to look more broadly at notions of Human Computer Interaction (HCI). There are four core elements in any human computer interaction: The USER, The TASKS the user has to perform, the SYSTEM (software and resources) that enable them to perform it, and the CONTEXT in which it is performed. These four elements or variables together define the ENVIRONMENT in which the particular Human-Computer Interaction takes place.
ENVIRONMENT = USER + TASKS + SYSTEM + CONTEXT
The aim of a usability testing is to provide feedback on system designs from a human factors perspective to improve those designs. This is done by creating a modelled environment and attempting to replicate the elements of user, tasks and context while examining the system variable. In other words, we fix the user, tasks and context variables and examine the system, and measure its performance in a qualitative sense. The output of a usability evaluation is a description of how various characteristics of the system perform, and how they might be altered in order to improve the performance of the Human-Computer Interaction in the environment. Usability is a measure of the quality of the system as it is used. It is a component of the broader set of criteria termed Quality in Use (ISO 14598-1).
Why we need usability testing
Usability testing as an activity is an important component of User Centred Design, an overarching methodology that involves consulting users at various stages of the software development lifecycle, typically through gathering user requirements, testing visual designs and partial prototypes, and validating final designs against user expectations and tasks.
Usability Testing determines the extent to which a design meets user requirements, whether they are explicit or not
Usability testing is necessary because even if user requirements are very carefully gathered and documented at the start of a development project, the manner in which those requirements are expressed in an interface design, and a user group’s reaction to that design, can both vary significantly, affecting both user perceptions and usage of the application or website. This can cause a lot of surprise and angst for the developers of the system. Fundamentally, developers of software systems are too close to a project to be able to objectively assess whether what they have created is a faithful interpretation of requirements, or one that is likely to meet the needs of the users.
The independence of testing is paramount – you just can’t test your own designs
The methods of usability testing
Usability testing can be undertaken through a variety of methods such as questionnaires, focus groups, paper prototype walkthroughs, task based functional testing, eyetracking, expert reviews and heuristic evaluation. There are many others and an infinite array of variants on approaches, and often it is achieved by a combination of methods that most efficiently attains the testing outcome. All these methods hope to achieve one goal: to assess the quality of designs against a range of agreed standards and make recommendations for improvements. Those ‘standards’ are selected at the start of the test process, and form part of defining the test objectives. Standards can be what users expect, what they like, how they perform on task completion, as well as rules for good interface design and best practice.
The place of usability testing in the software development lifecycle
An iterative software development process involves specifying what is to be built, creating successively more tangible artifacts that represent that specification, and evaluating them from multiple perspectives. Each evaluation gauges software quality and results in a modification to the specifications before entering the next build phase. The number of iterations depends on project attributes such as business goals, size, budget and uniqueness, the skills of the development team, and the adopted development methodology.
One of the most common errors is to leave usability testing too late. It should be introduced quite early in the development lifecycle, typically at the point where an early partly functioning prototype is developed. Earlier testing is more exploratory when changes to designs are acceptable. The later the testing, the more it will become a validation exercise that focuses on identifying problems with minor interface elements. Major changes become more expensive and difficult to implement.
A usual mistake is to leave usability testing too late in the development cycle, when design changes are more difficult to make
When usability testing is done properly, at the right time, with an appropriate scope, and a carefully expressed set of test objectives, it will add direct value to the development process, identifying issues and making recommendations for improvements. It provides the development team with a set of actions: Items in an interface to fix, edit or implement, as well as an objective assessment of the quality (i.e. quality in use) of what has been created so far. As a validation exercise, it provides project sponsors with assurance before launch that the developed system will be usable by the target market and support the business goals of the sponsors.
But, do I really need to do Usability?
So, if you are one of those pragmatic project manager types in charge of the budget of an applications or website development project, I can guess what you might be thinking while reading this. You probably already know user centred design is good, and that usability testing is an important part of it, but wonder if it is worth the cost of hiring an external party to do it. You may believe that your team is taking into account usability principles while developing. And, while you would like your users to have a good experience with your application or website, you don’t necessarily want to spend an already stretched budget just to make your users happier. You might wonder if there some quantifiable benefit from usability testing from a business perspective.
Obviously usability testing is going to cost – it’s a specialist activity completed by consultants. The question is, what return on the cost can you expect if you invest in independent usability testing as part of a commitment to user centred design? In order to answer this, we can revisit at a few commonly held misconceptions about Usability Testing:
- Usability is obvious, my development team is experienced and know their users very well
- Usability testing is like paying for someone to be critical rather than getting on with creating usable designs in the first place
- Usability testing is just another process that will impact delivery timelines
- Usability is not related to the business goals of the system, only how nice the system is to use.
I need to acknowledge that all of these concerns can be real and will occur if the testing is not well-integrated into the development lifecycle or if the testing is poorly scoped or implemented. However, carefully planned testing can adequately address them all – and easily. It is the relationship between usability testing and the business goals of the system that I would like to focus on in the remainder of this article, using a case study. We look first at the potential benefits for usability testing (see Bias & Mayhew, 2005 for a thorough analysis), and see how these worked in the example.
Potential Benefits
Firstly, there are potential business benefits in developing the product, by:
1. Producing a product that has only relevant functionality
2. Detecting and fixing usability problems early
3. Reducing amount of major redesign
4. Minimising documentation
5. Maximising product design to work well not just look good
6. Reducing the risk of project failure.
Secondly there are potentially direct business benefits. In particular e-commerce sites can be improved by increasing the number of customers who will:
- Find products they want
- Find information easily
- Be satisfied and make repeat purchases
- Trust the site
- Not require support
- Recommend the site to others
- Support and increase sales by other channels.
Product sales can be increased as a result of the usability of the product, by:
- Improving the competitive edge by marketing the product as easier to use
- Increasing numbers of satisfied customers
- Obtaining higher ratings in product reviews.
Employers can benefit from easier to use systems, through:
- Faster learning and better retention
- Reducing task time and increasing productivity
- Reducing errors that require correction
- Reducing staff turnover as a result of higher motivation and satisfaction
Reducing time spent by other staff providing assistance when users encounter difficulties.
Suppliers can also benefit from:
- Reduced maintenance costs
- Reducing support and help line costs
- Reducing training costs.
The ROI for Usability Testing can be found in development, sales, use and support of the product. Usually it’s a combination of these.
References
Bias R & Mayhew D (2005) Cost-Justifying Usability – An Update for the Internet Age. Morgan Kaufmann, CA.
Dumas, J & Redish, J (1999) A practical guide to usability testing. 2nd Ed. Intellect.
