Watch Yourself on Tape

One of the benefits of recording your idea) medium for reviewing your own per{"^J^^ddl* technology by reviewing your on^^aviors that need improvement. Take notes on what you do well and on behavio ^ time you That way you will remember to work on those aspects moderate a test.

Learn the Basic Principles of Human Factors/Ergonomics

Learn the basic principles of human information processing, experimental psychology, cognitive psychology, statistics, interface design, and usabthtv engineering. Subscribe to and read the proceedings from the major soaehes (listed in Chapter 14). Attend seminars and study basic psychology courses.
Manv universities and community colleges offer certificate programs m u«b, - ity testing or human factors. Attend conferences hosted by professional societies. (See Chapter 14 for a list.)

How to Improve Your Session-Moderating Skills

Conducting a usability test is an extremely challenging and worthwhile endeavor on a variety of levels. On the most ordinary level, you are working very closely with people in an attempt to design a product for maximum utility and benefit. On a deeper lexel, it is a very profound experience that forces you to confront your own mind and its tendency to be biased, distracted, and flighty.

Jumping to Conclusions

Inexperienced test moderators tend to overreact to early results. This can cause other members of the test team to act on the data prematurely. It is up to the test moderator to maintain a cool, steady demeanor and remind everyone to avoid forming conclusions until all of the results are in. One of the reasons for testing

multiple participants is just for that purpose: to get a rounded, comprehensive view of the product through the eyes of different types of people with different types of backgrounds. While it is important for the test moderator to pick up patterns in the behavior of participants as early as possible, this does not
necessarily mean reacting to that behavior. Avoiding premature conclusions helps to keep members of the test team from making major product changes before all the data is in.

Not Relating Well to Each Participant

Participants come in all shapes, sizes, and demeanors. Regardless of whether a particular participant is shy, arrogant moody, intimidated, self-conscious,  or whatever, the test moderator needs to adjust his or her style in order to allow the participant to be comfortable and at ease The test moderator should not get involved with battling, belittling, or in any way making a participant feel like anything but a guest. As far as the test moderator is concerned, the participant is always right.

Too Rigid with the Test Plan

An experienced test moderator will know when to deviate from the test plan. It is important for the moderator to sense when the test design is not achieving its objectives and is not exposing the product's deficiencies to the fullest extent. At tho^e times, it is up to the moderator to make the appropriate changes so that a participant's time and efforts are not wasted. Sometimes a participant with a different background than what was expected will appear. Sometimes the tasks are the wrong ones for addressing the research questions. Whatever the case, it is up to the moderator to revise the plan accordingly.

Acting Too Knowledgeable

This problem occurs when the test moderator and participant are in the same room during the test. Participants will tend to defer to the test moderator and ask many questions if they feel that the test moderator knows all the answers. Being too knowledgeable can also intimidate a participant who may

be somewhat nervous and self-conscious about his or her abilities.

One simple way to counteract the problem of acting too knowledgeable is for the test moderator to "play dumb." That is, the test moderator downplays any knowledge of the product and takes on the role of a research technician who is simply collecting data. Participants then change their entire demeanor 

when it becomes clear that they were not going to receive any assistance Tom the test moderator. They begin to try harder and to behave as if they were alone in their own home or office.

Too Involved with the Act of Data Collection

While the purpose of the test is to collect as much information as possible, the act of collecting that information should not interfere with direct observation of what is occurring. The test moderator needs to stay aware of what the participant is doing at all times, even if that means that every aspect of the participant's behavior is not written down. That's one purpose of record- ing the test, of developing coding categories, and of having others help with the more rote collection of such things as timings and number of references to an index or online help. All of these aids help ensure that the test moderator does not become so engrossed in the collection process that he or she misses important behaviors.

Leading Rather Than Enabling

Behavior that leads rather than enables is usually caused by the test moderator being too close to the product and unintentionally providing cues to £ participant about correct performance. The test moderator's tone of voice a nod of the head, cutting short tasks when the participant struggles, even the type of question the test moderator asks can all influence the participant and the test results. This potential problem is the main reason why assessment and validation testing is often conducted from a control room, although subtle hints are still possible even from there. To avoid the problem of leading rather than enabling, remember that you are there to collect data and to ensure that the test environment remains as neutral as possible.

Troubleshooting Typical Moderating Problems

Now that you have reviewed some of the important characteristics that a test moderator should possess, as well as some pointers for moderating the smoothest possible sessions with test participants, let's review some of the behaviors that test moderators should avoid. We describe the most common "errors" that test moderators make while conducting a test, many of which we have learned from experience. Consider it a list of "what not to do." Even experienced test moderators can benefit from taking a few moments just prior 

to testing to review this list. As with the previous list of characteristics, you can use this list to evaluate and improve your own performance or to evaluate the performance of someone you hire to conduct usability testing for you.

Offer Appropriate Encouragement

Another reason for encouraging the participant to keep trying is to show designers and developers watching the usability test the dire consequences of certain difficulties experienced by participants. It is important to let them see just how painful and frustrating the process of using their product can be.

Actually seeing the participant struggle and get very frustrated and observing the serious consequences firsthand will do more to convince a designer to  revise a product than weeks of discussions and negotiations. Sometimes just that extra amount of human struggle will convince a product designer that a change is needed and that the product should not be released as is.

Give Participants Time to Work through Hindrances

Keeping in mind that the participant may be very nervous is the first step.Skillfully working with the person's sense of frustration is the next step. There will be times when the participant becomes exceedingly frustrated during the course of a test. When you see that this is occurring, recognize this as a critical point that can be advantageous. Many test moderators, at the first sign of user frustration, will immediately tell the participant to give up and go on to the next task. They do this in order to avoid confrontation, keep things on an even keel, and maintain the participant's interest. However, because the frustration is usually related to a critical deficiency in the product, moving on too quickly misses an important opportunity. The participant's behavior at this point can he more revealing than at any other lime and can help the test team to understand how the participant learns to use the product It is important to encourage the participant to continue rather than cutting the task short too quickly. The trick is to find just that point when the participant is frustrated but is still willing to try.

Retrospective Review

An alternative to having participants think aloud is to replay the test with the participant after all of the tasks are done. As the moderator, you will have noted where during tasks the participant had issues question, or

After the tasks are done, you can to/g.e to the point you want the parttcipant to discuss what issues he or she was having wha^e bought processes were, and how the workarounds were arrived at Revi^w can be a good tool to use with participants for whom talking and working as smaU children'very old adu,te'^ s

A senous drawback to performing a review is the time it takes to do You have completed a fult slate of tasks and now must spend more time review™ he session from the beginning. It can take as much time as or more timf than the mam part of the session actually took. Some practitioners also avoid retrospective review because it gives some participants opportunity to revise and rationalize their behavior rather than simply reporting on what happened

Testing Multiple Product Versions

Now let's look at another common situation. Suppose that you want to compare two different versions of a product, Version A and Version B to see which one shows more promise as your ultimate design. (These are known as different cond.t.ons,") Additionally, you want to see whether performance vanes for either of two user groups, call them supervisors and technicians

This will result in a 2 x 2 matrix design as shown in the following table

Within-Subjects Design

Perhaps testing 15 participants is simply out of the question. Instead of 15, you could get by with only five participants by having each one perform all three modules as shown in the table below. This is called a icithin subjects design. However, you have the same problem of transfer of learning effects to consider.
To mitigate these effects, you must use a technique called counterbalancing, whereby the order of tasks is either randomized or balanced out. By varying the order of the presentation of tasks, you can limit the effects of learning transfer.

'Think-Aloud"Advantages and Disadvantages

Some participants will naturally verbalize what they are going through as they work through a usability test session. These people are gold; having a running commentary (something researchers call a "verbal protocol") from participants as they "think aloud" while they perform tasks offers many insights to why a problem exists and how someone tries to work around it Asking participants to think aloud during their sessions also reveals important clues about how they are thinking about (lie product or system they are using

and whether the way it works matches up with the way it was designed Farticipants always filter to some extent, however, so they may consciously or unconsciously leave things out as they talk. Likewise, it is impossible to know wrythmg that is going through a participants' mind during the session (it is of course likely that you don't want to know everything, just what pertains to your product, anyway). Thinking aloud can also help participants think through the design problem and form ideas for recovering While this J'n * a wa>'s vvhat they would do in real life, their doing so will giv e you ideas about how to remedy design problems.

One important reason to avoid asking participants to think aloud is significant]3re measuring ,ime 0n tasks" Thinking aloud slows performance

Sit-By Sessions versus Observing from Elsewhere

When the practice of usability testing started, most sessions were conducted in a highly clinical style, with the moderator seated in a control room to observe and collect data. Over the years, a less formal "sit-by" style of conducting sessions has developed for circumstances under which one would want to be near participants during usability testing, including most of the types of tests we describe in this book. For example, being with the participant in the testing room is advantageous for exploratory situations in which one wants to be able to gather the first impressions of participants about the design or functionality of a product. Doing so allows the moderator to be able to ask follow-up questions easily in a more cooperative session.
Additionally, if there is very quick interaction that must be observ ed closely or if you are conducting paper prototyping sessions, the moderator must be with the participant to mimic the computer actions.

However, if there is a chance in an assessment or validation test that the moderator's presence may bias the data, or if the emphasis is on gathering quantitative rather than qualitative data, set the test up to let the participant work by him- or herself while you manage the sessions from another room. For a deeper discussion of the physical arrangements for testing and observation rooms, see Chapter 6.

Choose the Right Format

Keeping in mind the research questions you are working with, the location of the test sessions, the time you have available, and the types of participants involved, you must consider the best way to get data from the people you have brought in to help you evaluate the product.

A Day in the Life of the Participant

Well, todays the day. I received a confirmation email yesterday and I am suppsed to report to some building over on Montgomery Stret, come sort of research facility I feel kind ol intimidated when someone watches me do things over my shoulder 

Technology can be really betuddling It s always been that way ever smce l cln remember. I can't louch-type, which makes using a computer slow (or m" And now this person wanls me to perform with some people watching me Weil it s an easy 75 bucks. I guess, and seeing as how I don't know these peoDle it'can't

be too embarrassing. 

So, I head over to the building where l¹m supposed to report and a man greets me at the front door. He ushers me into a room, hands me a questionnaire, and begins asking me the same questions I answered on the phone. \ guess they want to make sure I'm really not some kind of ringer, rm starting to get that

nervous feeling in the middle of my stomach. That same feeking I used to get when I took those tests in fifth grade with Mrs, Harmer, That kind of feeling that I'm going to be put on the spot and I'm not quite sure how this is going to pan out. My palms are really starting to sweat. Maybe ! should have stayed home and

painted the garage floor.

I'm probably being silly, so I complete the questionnaire and begin making small talk with some of the people in the room. I'm just starling to relax when I'm ushered into another room. The room is empty, and I begin to wonder where everybody is. I just assumed there would be a lot ot people watching. The room is large, with a table, a desk, computer, and a piece of equipment I've never seen before. I'm totd to sit down. That's when somebody else comes into the room and begins to explain that I'll be working in this room by myself and some people on the other side of the glass will be watching me.

Getting the Most out of Your Participants

Let s explore some things that the test moderator can do to enharce the process. One of the best things a test moderator can do is to develop in-eased sensitivity to the plight of the participants. What does it feel like to b= in the testing hot seat? Figure 4-1 shows a hypothetical example of one parti^ant's point of view compiled from many of the participants observed over thf vears While it is written tongue in cheek, its point is not. Participants are ofterr laced in awkward, stressful situations where they have little control over everts The 

more you put them at ease, the greater are your chances for accurate results that are applicable to real-world situations. You should become family with the codes of ethics of the Usability Professionals' Association and the Human Factors and Ergonomics Society. The links for each are available on re web site that accompanies this book (www.wiley.co_m/go/usabilit_ytesti^

Good Organizer and Coordinator

A usability test is a project within a project. Even a simple test require the management of an astonishing number of small details, events, and mil^rones.  Ensuring that equipment is in running order, getting all participants tc re site on time, and making sure that the product is ready for testing are ultimaidy the responsibility of the test moderator, In addition, the test moderator is te focal point for the other test team members, and must coordinate their actirries as well as those of any outside consultants into a unified effort. Thereto, the test moderator should be a good organizer and coordinator.

Good Communicator

Good communication skills are essential tn usability testing. The test moderator must communicate with individual members of the development team, participants, observers, and other individuals who may be helping to ^minister the test in one way or another. The test moderator must be sklrul at persuading others to make changes that are in their best interest, an: he or she must be able to explain the implications behind the test results Good writing skills are also essential because the test report is often the sole neans of communicating test results to those who did not attend the test s~iions. The written report is also the important historical document that is relie: upon months or years later to review or revisit the test results.

"Big Picture" Thinker

There is so much data collected during a usability test, and there is so much data that could be collected during a test that it is very easy to lose sight of the forest for the trees. The test moderator must be able to weed out the significant from the insignificant and this ability takes two concrete forms.
- The ability to draw together all of the various inputs, comments, and data from a single test to form a cohesive picture of a participant's performance.
- The ability to draw together the varied inputs from different test sessions and focus on the most important and critical findings,
1, is very easy to get lost in the details and focus on ^v^lobservations.lHs also easy to be influenced by the last participant and forget all that has before.
An effective test moderator, however, avoids these difficulties by saying focused on the big picture.

Empathic "People Person"

Participants will relate more readily to a test moderator who is an empathic individual. This may not be all that critical during the test session itself, especially if the session requires little probing or exploration on the part of the test moderator. However, empathy can play a major part during the debriefing session when the test moderator is trying to elicit a participant's innermost thoughts and feelings about the previous two hours of work. Participants will tend to hold back if they feel that the test moderator cannot relate to their
particular situation, this being especially true if the session was unusually frustrating or difficult.

Long Attention Span

Experienced test moderators share a secret: Usability testing can be tedious and bonng. There are long stretches when seemingly nothing is happening when participants are reading and absorbing, thinking, and sometimes jusi resting. The moderator cannot possess the type of personality that needs new stimulation every 5 to 10 minutes. The moderator must be able to pay attention for long periods of time because there is no predicting when a gem of a discovery wilt arise during a test session. In addition, because the moderator

may view up to 10,15, or 20 sessions, all of which involve observing the same or similar tasks, the ability to stay focused is extremely vital.

Flexibility

Another related characteristic of an effective test moderator is flexibility, which has to do with knowing when to deviate from the test plan (we discuss this further in Chapter 9). There are times when a particular participant does not have the expected skills or simply views the task in a completely different way than was originally intended. Jeff conducted a test when the entire high-level design of an interface became questionable after testing only two participants. 

He could see immediately that the premise for the design was flawed. At that time, he recommended that the company halt testing and go back to the drawing board. To continue ferreting out minor problems with the product would have been a waste of everyone's time. While this is an extreme case, the point is that one needs to be prepared for the unexpected, even if that ha, serious consequences.

Comfortable with Ambiguity

Usability is not a precise science consisting of formulas and black and white answers. Even if a usability test is conducted under the most rigorous conditions, which is atypical, you are still not assured that all of the results are valid and generalizable to your entire user population. Instead, usability testing can often be an imprecise, ambiguous enterprise, with varying and sometimes conflicting observations, not surprising for any venture that has human beings as its focus. A test moderator, then, must understand and be comfortable with

ambiguity.

For example, prior to testing you may think that there are only two ways to perform a particular task. During testing though, you discover that the participants have found four other ways to perform the same task. Or, you discover that you are no closer to a clear-cut resolution of a product's problems after a week of testing than you were before you began. Or, when testing multiple versions of a product, no clear winner emerges. The versions are all equally bad or, if you are lucky, equally good. These situations require patience, perseverance, and very often skill at negotiation. Without tolerance for ambiguity and the patience to persevere, the test moderator tends to rationalize and to blame the participants for making unplanned choices during the test

Good Listener

Listening skills involve the test moderator's ability to hear with "new ears'" during each session and to lay aside personal biases and strong opinions about what he or she is seeing and hearing. The test moderator needs to understand both the content and the implication of a participant's comments, as there are often mixed messages of all kinds during testing. The test moderator must pick up on the subtle nuances of speech and emphasis, as a participant's comments are often indirect and less than forthcoming. It is so important to understand the rationale behind the participant's behavior, because the rationale often signals whether a change in the product is required or not.

Excellent Memory

Some might believe that because usability test sessions are recorded, the test moderator need not rely on memory for conducting and evaluating a test session. Actually, memory is called Into play well before a test session has ended. Because a test session can be rather long, the test moderator needs to remember behaviors or comments that took place earlier in the session in order to cross-check and probe those behaviors later in the session. For example, a participant may attempt to perform the same task in two or three different ways, and the test moderator may want to probe to understand why the participant performed the task differently each time. 

Memory is also required to recall the results of a test session after its completion. Because there is often very little time to devote to searching the videotapes after a test, except as insurance against missing some point entirely, the test moderator often must rely heavily on memory and notes.

Instant Rapport with Participants

Bringing in participants to evaluate your product is an auspicious and very opportune point in the development cycle that should not be squandered If for some reason a participant is not at ease and is not able to function as he or she normally would, it represents a lost opportunity and potentially misleading results. If you are able to test only five participants, one uneasy participant represents a potential loss of 20 percent of your test data. The test moderator's ability to quickly size up each participant's personality, make

friends, and put the person at ease is essential to getting the most from the testing process. Some participants need coddling, some need stroking, and some are businesslike and require a more formal approach. Whichever the case, the test moderator must make each person feel comfortable and secure.

Quick Learner

An effective test moderator need not be expert in the intricacies of the specific product being tested. For example, if the product is a database management system, the moderator need not be an expert in database management.