Disadvantages

■ Depending on the skill of the testing staff, the control room setup can create a very impersonal environment. This is sometimes referred to as the "guinea pig" syndrome, with the participant feeling overly self-conscious during the test. (This effect can be exacerbated by the type of intercom system used, some of which make the test moderator's instructions sound like the "voice of God.") This self-consciousness is worse for less sophisticated (in terms of exposure to high-tech equip- ment) participants who are insecure about their abilities. The test moderator may need to mitigate the effects of this setup by staying in the room with a less sophisticated participant.

Advantages

■ It allows unparalleled unobtrusive data collection. Because the moderator is not in the room, this eliminates almost entirely any possible biasing effects due to inadvertent nonverbal communication or mannerisms.
■ Being in a (nearly) soundproof room, the testing team and observers can easily communicate among themselves about the proceedings and discuss possible product improvements.
■ This environment can accommodate many observers (typically 4 to 12) viewing the test at one time.

Classic Testing Laboratory Setup

This setup (see Figure 6-8) consists of one room designated as the testing room and a second room designated as an observation and control room. The only individual inside the testing room is the participant. All other test personnel, including the test moderator, observers, camera operator, and so forth are

stationed inside the control room viewing the proceedings through a one-way mirror. All communication between the test moderator and participant occurs through an intercom and speaker arrangement, and, in the more elaborate laboratories, the test is extensively monitored with multiple video cameras, audio tape recorders, data loggers, and other modern electronic equipment. However, as an alternative, depending on the type of test, the test moderator could also be in the test room with the participant.

This setup requires a large capital outlay and commitment to testing by  management. However, if you are interested in trying out such a setup for a test that you are conducting, you may do so for a modest rental fee. Simply locate a tab facility (there are many throughout the country; wwrf.Quirks.com has a directory of such facilities). Some facilities do none of their own research but only make the space and equipment available for a per diem rate. Re- search firms, market researchers, universities, and other corporations may
have lab spaces that you can rent. Most come with a receptionist and other niceties. Many also offer participant recruiting.

Disadvantages

As with the simple single-room setup, the test moderator's behavior can adversely affect the test.

« Unless you have a permanent setup, you will need to tie up two conference rooms or office spaces for up to a week.

Advantages

- All the advantages of the simple single-room setup also apply to this setup.
- Observers get to view the test as much as they like without having to worry about interfering. They are free to come and go as they please without distracting anyone. This is especially crucial for very lengthy tests, where observers may be interested only in certain portions or cer tain types of participants. More important, they are able to talk among themselves, discussing test observations.

Electronic Observation Room Setup

The electronic observation room setup (see Figure 6-7) enables the observers to be physically separated from the testing activities. From a separate observation room, they can watch but are not set up to communicate directly with the test moderator.

The signal from the video camera(s) as well as the screen image, if the test involves software, web sites, or web-based applications, are both directed to  video monitor(s) in the observation room. The room could be in a completely different part of the building or adjacent to the testing area.

When the testing area is adjacent to the observation room, you typically use speaker phones dialed into each other to provide observers with audio for the session.

Communication between the test moderator and observers can occur face to face or via notes during breaks or at the end of sessions, or by instant messaging (if you are taking notes on a computer) during sessions. If you use Morae from TechSmith and all of your observers are on the same local area network (LAN) as the testing machine, observers can use one module of Morae to see and hear a feed from the testing room from wherever they are. Likewise, you can set up WebEx, NetMeeting, or GoToMeeting under which to run the participant's session so that others outside the immediate area can "observe" remotely-

Disadvantages

- It takes training and practice to manage a large number of observers.

- The moderator must be steadfast in enforcing proper behavior of observers to prevent them from biasing the test, which can be distractine to the moderator.                  

- This setup requires additional equipment: microphones and a projector.

- The observers probably will not be able to see the participant's subtle behaviors or facial expressions.

Advantages

- All the advantages of the simple single-room setup also apply to this one.

■ It accommodates a large number of observers.

■ Because so many observ ers can attend, debriefings between or after sessions can be more useful than when a smaller number attend.

- You may not have to record sessions because everyone who wants to observe can.

- With present, engaged observers, you can gather notes and insights from them more easily.

Large Single-Room Setup - II

Large Single-Room Setup - I

Similar to the setup found in the section "Simple Single-Room Setup/ the moderator and participant are situated near each other (see Figure 6-6). In this case, though, by using a much larger room — such as a training room — you can invite many more observers to each session. (Some usability practitioners routinely have up to 30 observers per session.) Here, the participant and moderator are in front of the room. Both wear microphones so that the observers can hear without strain. The image of the participant's screen is

projected onto the wall above the participant or above the observers, who could even be facing awmj from the participant,

Disadvantages

Loss of proximity to the participant limits what the test moderator can see directly of the proceedings. It is possible to miss subtle behaviors.
It is crucial to have a good angle via the video camera on what the participant is seeing and doing, because you may be blocked from seeing directly.

Because the test moderator is not within the participant's peripheral vision, if the test moderator is directly behind the participant and within 10 feet or so, the test moderator could make the participant feel very uneasy and overly self-conscious. The guideline here is to remember when you are behind the participant but within ten feet, you must stay within the participant's peripheral vision. Farther back than approximately ten feet should not cause any discomfort to the participant. 

As with the simple single-room setup, there is limited space for observers.

Advantages

The test moderator is more free to move about, take notes use data l„. gmg so ware while .he .est is goin, on, and yet is still mn ^T proximity of the participant. Being somewhat removed from L pa" ic
.pant, you need not be as concerned about controlling bodyt— manner,sms and so forth In addition, you may use a%ompu e" wXui distracting the participant.
The participant does not have a complete sense of isolation, because the participant is still in the room with the test moderator. This setup is also more likely to encourage the participant to think aloud, than if left alone in the test room.

Modified Single-Room Setup

This setup (Figure 6-5) enables you to use a room large enough to position yourself at a workstation behind or to the side of the participant without

Disadvantages

- The test moderator's behavior can affect the behavior of the participant If you are not mindful of your speech, mannerisms, and so on you will inadvertently and subtly react to what the participant is doing. Even
if you do not make any outward remarks, the participant may pick up a sigh or a shift in your posture in response to what he or she is doing.
That in turn can cause the participant to veer off from the direction in which he or she was going or look to another page. It is especially crucial to avoid making notes when it appears the participant has finished
a task. This can cue the participant that he or she has finished a task successfully even if the participant is not sure. Likewise, your writing or typing may suggest to the participant that he has done something' incorrectly. Even if this is true, you do not want to telegraph that,

Simple Single-Room Setup - II

Simple Single-Room Setup - I

The simple single-room setup, shown in Figure 6-4, is the most basic type of testing setup, both in terms of resources and the amount of space required. It represents the minimum environment required for testing that can accommo- date observers, essentially a quiet secluded room. Within the room, the test moderator is located a few feet from the participant at about a 45-degree angle. 

As you can see from Figure 6-1, the idea is for the test moderator to be close but not too close. It is important to remain within the peripheral vision of the participant, so that the participant can sense where you are at all times, but not so close that your presence is distracting or anxiety-provoking.

The main concern with this type of positioning is that the test moderator might inadvertently bias the participant through subtle (or not so subtle) nonverbal cues. However, if such biasing can be minimized, there are real advantages to being close to the participant, and for very interactive testing, such as exploratory testing, this degree of intimacy is required.

Setting up a Permanent or Fixed Test Lab

Here we describe different testing setups/environments for office settines Each setup has advantages and disadvantages, and each lends tseU Sf a particular style or philosophy of testing, as well as to the degree of ell hecaur iS a book for "to*™* " we wS recommend what we feel is the setup that provides the best value (e.g . objectives vs cost) for a start-up testing enterprise.

Minimalist Portable Test Lab

Arranging Sessions at a User's Site

Taking your usability test on the road requires some special planning because the logistics are not trivial- The steps to planning are shown in Figure 6-1. If you are doing testing in the user's workplace, you must get permission of the management there to do the testing (as well as for installing anything on their systems, if that is necessary). Once you do that, you may be able to get help with recruiting participants from the manager who gave you permission. Exercise caution with this approach, however, keeping in mind that the manager probably will be biased in her selections. There may be company policies about whether participants can take part in a test during the regular workday and whether they can accept honoraria, incentives, or compensation for the time they spend with you.

Next, you must get permission from the participant, make an appointment and ensure that the participant understands the purpose and format for the session. You should have a short conversation with the participant by phone before you visit the person to confirm the appointment, explain the study allay any fears she might have, and answer any questions. Be sure to tell the participant generally what you will be doing during the session, whether there will be others with you, and whatever else you might need at the participant's location to make the session worthwhile for both you and the participant Remember that you are a stranger entering this person's personal space] whether m a workplace or a home. For that reason, you should be organized

Test in Multiple Geographic Locations?

You may feel some pressure from your marketing or market research partners within your company to conduct usability testing in multiple locations. There is a romantic quality to traveling to different markets to conduct tests, and anyone who has traveled knows that people are different in different regions of the world. But do the people in different places find different usability problems from the people who are close by your offices?

The answer is: Occasionally. Some research shows that testing in multiple locations will reveal many of the same problems. As much as 80 percent of usability problems may be the same across locations. But the other 20 percent may be location or context specific and that can make being on location well worthwhile.

So, just as you take into account other factors that make people different from one another, such as age or education level, you probably should look at location as well. Needs of users may be different from city to city (San Franciscans generally don't need snow tires for their cars; drivers in Boston are much more likely to use them). Terminology may be different from region to region ("purse" or "pocketbook"? "pop"'or "soda"?). Motivations and experiences will be different, too, possibly from floor to floor inside a corporation (the mail room versus "mahogany row"). These things and others make up what we'll call the user's "context.'

When considering whether to test in multiple locations or in a location outside your immediate area, start by thinking about the user's context (as well as budget and time availability). If the context of one type of users is significantly different from the context of another type of users in a way that will impact your product, you should try to go into those different contexts to test. The context might be geographical (for example, weather-driven buying decisions in the south versus the north) or hierarchical (for example, administrative assistants versus directors of a company) or both (or something else).

In a Lab or at the User's Site?

Determine the appropriate location of usability tests by considering these
factors:

■■ Your test design and measures " Is the test exploratory/formative or validating/summative?

■ Must you sit next to the participant because you will collect qualitative data, need to prompt or give hints, or manage the product being tested?

Decide on a Location and Space

In the early days of usability testing, the main assumption was that nearly all usability tests were highly structured and quite formal. In addition, most usability testing was done on products used in offices. Technology has expanded greatly in the last 15 years and so has the application of usability testing. As we have learned through our own experiences, not all testing should be done in a lab setting, and sometimes doing testing in a lab simply is not practical. The location of the test sessions is intricately linked with the design of the study and who the users are. There are other considerations as well, as we discuss in the next section.

The discussion in the rest of this chapter assumes that you will want people to observe the sessions who are not employed in moderating the test. Consider carefully from what location you want the observers to observe. You, as the moderator, get to decide. Are you comfortable having observers in the room 

with you? (Remember, you'll have to train them on how to be in the session.) How do you feel about observers possibly talking among themselves while observing from outside the testing room?

Also, especially for beginner moderators, we recommend that you concentrate on moderating and get some one else to take notes or log data.

Set Up a Testing Environment

For many of those contemplating the implementation of a usability testing program, the discipline has become synonymous with a high-powered, well-appointed, well-equipped, expensive laboratory. For some organizations, the usability lab (and by that we mean the physical plant) has become more prominent and more important than the testing process itself. Some organizations, in their zeal to impress customers and competitors alike with their

commitment to usability, have created awe-inspiring palaces of high-tech wizardry prior to laying the foundation for an ongoing testing program. Not realizing that instituting a program of user-centered design requires a significant shift in the culture of the organization, these organizations have put the proverbial

cart before the horse, in their attempt to create instant programs, rather than building programs over time.

This approach to usability testing is rather superficial and snort-sighted, and has a high risk of failure. It approaches user-centered design as a fad to be embraced rather than as a program that requires effort, commitment, and time in order to have lasting effects on the organization and its products. It isn t

uncommon for newly built, sophisticated usability laboratories to become the world's most elaborate storage rooms. Having succumbed to the nrusperception that equates the laboratory with the process itself, these organizations have discovered onlv too late that usability testing is much more than11icoUection of cameras and recorders. Rather, a commitment to user-centered des.gn and usability must be embedded in the very philosophy and underpinning of organization itself in order to guarantee success.

Sample Test Plan

If we roll up the parts of the plan that we have included as samples above, the test plan deliverable comes to about 10 pages. To see the full plan, go to the web site that accompanies this book  (www.Wiley.com/go/usabilitytesting).
The product being tested is a hotel room reservations system on a web site. There, you can view more examples of test plans, as well as download templates for test plans and other related deliverables in the file

Ch05 Hdotcom_test_plan.doc.

Describe How the Results Will Be Reported

This section provides a summary of the main sections of your test report and tlie way in which you intend to communicate the results to the development team. For the report contents section, simply list the sections that will appear in your test report, as shown in the example in Figure 5-9.
For the presentation section, describe how you wilt communicate results to the development team both prior to and following the report. For example, you might hold an informal meeting with those on the critical path of the project just after the test is completed and prior to analyzing all the data. Then,

Sample Performance Measures - II

Qualitative Data

■ Think aloud verbal protocol
■ Quotable quotes: (for example)

■ "I loved it — when can I get one?"

■ "You guys have done it again — you're still not listening to customers."

■ Wow, I'm very, very impressed."

■ "Can I please leave now — keep my money and the product."

Sample Preference Measures

Ratings and rationale concerning:

■ Usefulness of the product « How well product matched expectations « Appropriateness of product functions to user's tasks

■ Ease of use overall

■ Ease of learning overall

■ Ease of setup and installation

■ Ease of accessibility

■ Usefulness of the index, table of contents, help, graphics, and so on

■ Help desk replies to inquiries

■ Ease of reading text on the screen Preference and rationale for:

■ One prototype vs. another prototype

■ This product vs. a competitor's product

■ This product's conceptual model vs. the old model