A Client Guide to Usability - Frequently Asked Questions
John Eklund, UX Research ©2008
Having attending dozens of meetings with potential clients interested in engaging the services of a usability specialist, I thought it might be helpful to list the most common questions and observations that I have encountered, along with my usual answers. Maybe this will answer some of your questions before a meeting is required.
Question 1. What will your services provide and what is the value of usability?
Usability gives an independent, expert view of the extent to which designs meet the requirements of the user groups, with structured recommendations for improvement in line with best practice.
Question 2. We are on tight timeframes, and I’m worried that incorporating usability will cause a delay in development
If well-integrated usability testing can be conducted in parallel with other development activities. The outputs are recommendations that add direct value to development and can save on redevelopment effort. It also doesn’t have to be a protracted effort, and can be conducted in one week, with one week notice for recruiting users.
Independent usability testing is a fundamental part of user centered design – specifying the context of use, creating designs, and evaluating designs. An independent analyst provides totally objective advice, they test so many different sites that they are aware of best practice. Usability testing is not meant to be a substitute for user centred development.
Question 4. I’m concerned that bringing in a third party to provide independent usability testing will put my developers offside, as their designs will be criticised.
Most developers understand the place of usability testing in development, and recognise that it provides valuable insights into customer behaviour that will improve their designs. Test results are also presented in a constructive, positive manner in reports and debriefs.
Question 5. If I use usability services, what control do I have over the process?
You would need to be involved at every step of the planning process, helping to define the demographic and agree the test objectives, approve the test scripts and questionnaires. You would be encouraged to observe the test sessions as well.
Question 6. We are in the early stages of development, so we really need more direct design input rather than a critique at a detail level. The position of radio buttons and details like that can be worked out later. Given that’s the case, what is the value of independent usability services early on?
Early usability sessions can be quite exploratory and focus on the big picture aspects of designs such as look and feel and information architecture. Later testing is more validation and can focus on detail aspects of an interface to be improved. So, don’t think of usability as only testing. Important in usability testing is to uncover those aspects that can be changed at that stage of development.
Question 7. There are a lot of stakeholders involved in this project with different opinions on aspects of the designs, so we really need independent advice. How can this be effectively communicated to our business?
An independent report by an expert who presents results of user testing in a formal debrief is usually quite convincing. Involving them in the test process, by observing sessions, is also very useful in obtaining their buy-in.
Question 8. Why are we talking about usability testing with 8 users? Surely that wont be enough to give credibility to the results?
You might be surprised just how much information comes from just 8 users and how conclusive the results can be. The main reason 8 is used, and it can be 6 or 10 or more) is that you can run about 4 sessions of 90 minutes in a day, and two days of testing is usually enough. Also with 8, you may get one or two less useful sessions, and you can still base sound results on 6 users. Ultimately the number of user depends on how many different demographics there are, budget, and time.
Question 9. I’d like to include one representative from the business in the sample of customers for usability testing. Would you recommend that?
No, because one representative of any user or stakeholder group wouldn’t be enough to base any useful results on. It would also be a totally different test script. If you wanted to include this group it might be best to interview at least 5 stakeholders separately from user testing.
Question 10. One of the aspects of our website we would like to investigate is why people would visit it, what they expect of it and how they like it, and not just how simple it is to use. Can this be considered in a usability study?
Yes, there is ample opportunity to gather information on user preferences, motivations, attitudes, influences, etc as well as how they respond to the details of an interface. Much of the results can inform your marketing strategy and positioning and not just the interface design.
Question 11. How do you recruit the participants for a study?
We normally outsource this to a professional recruiter who takes the demographic profile and creates a screening questionnaire, then filters out from their own database of candidates the people who fit the required profiles. You can undertake to do this yourself.
Question 12. Should we pay the participants in a usability study?
Yes, to ensure they are motivated to attend. Participant incentives are usually about $100 for a 90 minute session and this goes up for more unusual profiles.
Question 13. Our online application supports several different user groups, with some tasks in common. Should we test one or two from each group to make up the sample?
It is better to have at least 4 from each sub-demographic, otherwise the findings are erratic. You can have multiple demographics in a study, as long as there are common tasks.
Question 14. Why would you choose to run a focus group rather than one to one testing?
Focus groups are useful for testing early prototypes, as you can get individual and group feedback from more numbers of users with less time, although there is less contextual information about individual behaviours. Focus groups are designed to gather preferences rather than observe actions, and usually the objective is to reach a group concensus.
Question 15. What sort of physical setup do you need to have for testing?
You can run usability sessions in a quiet room with only a laptop and a notepad. We have had a lot of success running a portable setup in two adjoining rooms on client premises, conducting interviews in one room and videoing them. The adjoining room has a Video monitor and a monitor showing the user’s desktop. The traditional lab setup with one-way glass is a “nice to have” but certainly not needed. When a formal setup is required, this can be hired on a day basis.
Question 16. Can users in remote or regional locations be tested?
Sessions can be run anywhere with a remote lab setup, or phone interviews and surveys also work quite well with users sitting at their desks. An alternative to testing overseas is to recruit users of that profile who are visiting the city in which testing is conducted.
Question 17. What’s the best time to for usability testing – how often should it be done?
Normally I would recommend once at the stage when wireframes and visuals have been developed, and again on a functioning prototype, but it depends on many factors.
Question 18. When is an expert review (testing without users) better than user testing?
An expert review provides a judgement of the usability of a design against a set of standards. You typically get a lot of detail without the more contextual information about user behaviours. It is useful for comparative evaluations, and also in identifying potential issues before user testing. It’s not really any quicker to do, although there’s no need to allow time for recruiting participants (ie it has less lead time)
Question 19. What we are testing is quite complex and large, so how will be know where to focus the effort in user testing?
Go back to the business objectives of the application and find the areas of the system that are most critical or where there is some uncertainty in designs, or those areas that are representative or most commonly used, identify the core user groups and create tasks around these.
Question 20. What is the cost of usability testing?
It varies depending who is asked and sometimes I think according to who is doing the asking! It is curious that there is almost no information anywhere on the cost of these services. Prices these days vary a lot, and it depends on things like the length of the sessions, the complexity of the system, the extent and coverage of the testing, and the amount of detail required in reporting. Expect to pay between $1,000 and $1,500 per user. This might sound expensive but remember usability testing is very effective at gathering information from small numbers of users. You can conduct usability testing on 8 users for less than $10K and that should include a detailed report and debrief. This would be 8 sessions of about 75 minutes each. It can be less with a summary report and no debrief, which is sometimes more suitable. If you are paying more than this it is time to ask why.
Question 21. How can the cost of usability testing or other user centred design services be justified?
By increased conversation rates, improved customer satisfaction, reduced redevelopment effort post launch. Key is to use it at a time where it add value to development and provides a validation of designs. It’s a bit like an insurance policy.
Question 22. There are quite a few providers of usability services in the marketplace now. What should I look for in a provider?
I’ve asked a few people who regularly commission these sorts of services from external providers, and of course I’ve responded to many requests for services where the criteria have been explicitly stated, and I think the answer is:
1. Expertise – you need a consultant with at least 5 years experience in your industry, someone senior with the right qualifications and a background in commercial research
3. Reliability – you should be able to rely on the provider to deliver the services in a timely manner and drive the whole process
4. Extent of offering – you need someone with a broad knowledge of development practices and user centred design who can recommend other experts or offer related services
5. Price – Like with everything, the provider should offer value for money.
