Test in Multiple Geographic Locations?

You may feel some pressure from your marketing or market research partners within your company to conduct usability testing in multiple locations. There is a romantic quality to traveling to different markets to conduct tests, and anyone who has traveled knows that people are different in different regions of the world. But do the people in different places find different usability problems from the people who are close by your offices?

The answer is: Occasionally. Some research shows that testing in multiple locations will reveal many of the same problems. As much as 80 percent of usability problems may be the same across locations. But the other 20 percent may be location or context specific and that can make being on location well worthwhile.

So, just as you take into account other factors that make people different from one another, such as age or education level, you probably should look at location as well. Needs of users may be different from city to city (San Franciscans generally don't need snow tires for their cars; drivers in Boston are much more likely to use them). Terminology may be different from region to region ("purse" or "pocketbook"? "pop"'or "soda"?). Motivations and experiences will be different, too, possibly from floor to floor inside a corporation (the mail room versus "mahogany row"). These things and others make up what we'll call the user's "context.'

When considering whether to test in multiple locations or in a location outside your immediate area, start by thinking about the user's context (as well as budget and time availability). If the context of one type of users is significantly different from the context of another type of users in a way that will impact your product, you should try to go into those different contexts to test. The context might be geographical (for example, weather-driven buying decisions in the south versus the north) or hierarchical (for example, administrative assistants versus directors of a company) or both (or something else).

In a Lab or at the User's Site?

Determine the appropriate location of usability tests by considering these
factors:

■■ Your test design and measures " Is the test exploratory/formative or validating/summative?

■ Must you sit next to the participant because you will collect qualitative data, need to prompt or give hints, or manage the product being tested?

Decide on a Location and Space

In the early days of usability testing, the main assumption was that nearly all usability tests were highly structured and quite formal. In addition, most usability testing was done on products used in offices. Technology has expanded greatly in the last 15 years and so has the application of usability testing. As we have learned through our own experiences, not all testing should be done in a lab setting, and sometimes doing testing in a lab simply is not practical. The location of the test sessions is intricately linked with the design of the study and who the users are. There are other considerations as well, as we discuss in the next section.

The discussion in the rest of this chapter assumes that you will want people to observe the sessions who are not employed in moderating the test. Consider carefully from what location you want the observers to observe. You, as the moderator, get to decide. Are you comfortable having observers in the room 

with you? (Remember, you'll have to train them on how to be in the session.) How do you feel about observers possibly talking among themselves while observing from outside the testing room?

Also, especially for beginner moderators, we recommend that you concentrate on moderating and get some one else to take notes or log data.

Set Up a Testing Environment

For many of those contemplating the implementation of a usability testing program, the discipline has become synonymous with a high-powered, well-appointed, well-equipped, expensive laboratory. For some organizations, the usability lab (and by that we mean the physical plant) has become more prominent and more important than the testing process itself. Some organizations, in their zeal to impress customers and competitors alike with their

commitment to usability, have created awe-inspiring palaces of high-tech wizardry prior to laying the foundation for an ongoing testing program. Not realizing that instituting a program of user-centered design requires a significant shift in the culture of the organization, these organizations have put the proverbial

cart before the horse, in their attempt to create instant programs, rather than building programs over time.

This approach to usability testing is rather superficial and snort-sighted, and has a high risk of failure. It approaches user-centered design as a fad to be embraced rather than as a program that requires effort, commitment, and time in order to have lasting effects on the organization and its products. It isn t

uncommon for newly built, sophisticated usability laboratories to become the world's most elaborate storage rooms. Having succumbed to the nrusperception that equates the laboratory with the process itself, these organizations have discovered onlv too late that usability testing is much more than11icoUection of cameras and recorders. Rather, a commitment to user-centered des.gn and usability must be embedded in the very philosophy and underpinning of organization itself in order to guarantee success.

Sample Test Plan

If we roll up the parts of the plan that we have included as samples above, the test plan deliverable comes to about 10 pages. To see the full plan, go to the web site that accompanies this book  (www.Wiley.com/go/usabilitytesting).
The product being tested is a hotel room reservations system on a web site. There, you can view more examples of test plans, as well as download templates for test plans and other related deliverables in the file

Ch05 Hdotcom_test_plan.doc.

Describe How the Results Will Be Reported

This section provides a summary of the main sections of your test report and tlie way in which you intend to communicate the results to the development team. For the report contents section, simply list the sections that will appear in your test report, as shown in the example in Figure 5-9.
For the presentation section, describe how you wilt communicate results to the development team both prior to and following the report. For example, you might hold an informal meeting with those on the critical path of the project just after the test is completed and prior to analyzing all the data. Then,

Sample Performance Measures - II

Qualitative Data

■ Think aloud verbal protocol
■ Quotable quotes: (for example)

■ "I loved it — when can I get one?"

■ "You guys have done it again — you're still not listening to customers."

■ Wow, I'm very, very impressed."

■ "Can I please leave now — keep my money and the product."

Sample Preference Measures

Ratings and rationale concerning:

■ Usefulness of the product « How well product matched expectations « Appropriateness of product functions to user's tasks

■ Ease of use overall

■ Ease of learning overall

■ Ease of setup and installation

■ Ease of accessibility

■ Usefulness of the index, table of contents, help, graphics, and so on

■ Help desk replies to inquiries

■ Ease of reading text on the screen Preference and rationale for:

■ One prototype vs. another prototype

■ This product vs. a competitor's product

■ This product's conceptual model vs. the old model