Explain Why the Participant Is Here

You may think that they already know this information, but you would be amazed at what external recruiters tell participants about what they will be doing in the sessions. Provide enough detail and context about the product for them to perform the tasks. Do not feel you need to provide product history, number of participants being tested, and so forth. Express appreciation tor their willingness to participate and how much their input helps Pr0^ a better product, regardless of how they perform. In no way, shape, or should the performance of the participants be tied to their compensation for participating, even as a joke. Vou can be sure that the thought has crossed their mind. No need to reinforce it.

Offer Refreshments

Being offered refreshments will help the participants to relax and feel at home They are more apt to indulge if you already have a cup of coffee, soda, or water in your hand. "Breaking bread" together is a wonderful icebreaker. Do not downplay the fact that your participants may be nervous. It is very common and you need to address it.

Make Introductions

Introduce yourself, of course, and anyone else whom the participant is likely to encounter during the usability test. You need not go into great detail about people's backgrounds; just a passing reference to the function of each person is fine. For example, you might say, "That person over there will be managing the recordings," or "That person over there will be keeping track of time to ensure that I let you go on time." New volunteer the fact that any person associated with the test or observing in the same room has worked on the product that the participants will be testing. Of course, if a participant asks (hardly anyone ever does), then by all means tell the participant the truth, but do not volunteer that information. The reason for this is simple; you want the

participants to feel absolutely unencumbered about providing any negative feedback. Associating real people with the product only makes it that much more difficult for participants to be honest.

As test moderator, include yourself in this "non-association" guideline if you are affiliated with the product. (Ideally, you will not be affiliated with the product.) Explain your role as the person who will be moderating the test, observing, and taking notes. If you are not affiliated with the product, then play up the fact that you are a neutral observer. If you will be moderating the usability test in the test room, then explain exactly what you will be doing during the test.

Write the Orientation Script Out

Now that you know the important reasons for creating a script, you need to actually create one. The following sections list the typical contents of an orientation/introduction script. However, do not feel that you need to include every category in your own script. If you do, you might end up writing an essay.

Plan to Read the Script to Each Participant Verbatim - II

. People are easily influenced by past events. Imagine that you have just tested the first four participants and are about to test the fifth one. Three of the first four have performed miserably. Not only did they have great

difficulty with the product, but they hated using it. You may be feeling down and discouraged, and are unknowingly about to project your feeling of frustration to the fifth participant by the manner in which you

introduce the session. However, reading the script forces you to use the same language, which in turn makes it easier to control the nonverbal aspects of vour communication, such as your mannerisms, expressions,

and voice modulation. In short, it is harder to express your own frustration when you read the script.

More than one test moderator may be conducting the test over a period of time. If that is so, there is a need to minimize the differences in the test moderators as much as possible. At the very least, reading

the script controls the initial information that is communicated to the participants.

Those members of the development team not present at the sessions will want to know precisely what was said to the participants. Showing the script to interested parties who were unable to attend the usability test

accomplishes this objective. It also communicates professionalism and rigorousness on your part.

You may forget an important point. There are so many details involved with moderating a test, why make it hard on yourself? Use the script and cross "what to say" off your list of things to remember and worry
about.

Plan to Read the Script to Each Participant Verbatim - I

Do not attempt to memorize the script, paraphrase it, or simply "wing it" from session to session. Here is why this is so important: 

- You want to present the exact same information to each participant so that all the participants are exposed to identical conditions prior to the test. By paraphrasing the script, you may change what you say in very

subtle ways.

NOTE If you feel self-conscious reading the script or feel that it sets an overly formal tone, simply tell the participant why you are reading it For example. I'm going to read this script to you now so that I provide the exact Instructions to you that I provide to everybody else, and so that I do not

forget anything of importance."

Keep the Speech Short - II

Keep the Speech Short - I

Unless you have an extremely complex test, limit the orientation script to a few paragraphs. Anything longer than that will not be retained by the participants in any case. Any instructions beyond two pages probably means that you are including actual test materials, such as task scenarios, as part of your script. We show two examples in Figure 8-2 and 8-3. Figure 8-2 shows a script you might use for a simple, exploratory test. Figure 8-3 has very specific instructions for a benchmark comparison test.

Keep the Tone of the Script Professional but Friendly

It should not be chummy or overly familiar as if the participants were your buddies. For example, in the food service industry, the waiter who asks for your name and continues to use it ad nauseam during the meal comes to mind as an instance of this lack of professionalism in the guise of improved service.

Orientation Script

The orientation script (also known as the introduction script) is a communications tool meant to be read verbatim to participants. It describes what will happen during a test session, sets the tone for that session in the minds of the participants, and is intended to put them at ease. It achieves this by informing the participants of what they will be doing, and reinforcing the fact that the product, not the participant, is being tested. Remember that participants often have only the vaguest idea of what they will be doing, possibly having been

presented with only some ambiguous reference to "participation in market research" or the like. For a particularly nervous participant, introducing the session this way can provide reassurance that he or she is actually the right person in the "right" place.

The script may be read to the participants in the testing area or in a "waiting or meeting area" prior to moving to the testing space. Our own preference is to read it just before beginning testing activities, which occur in the main testing area.

When developing an orientation script, there are three major guidelines to remember.

Guidelines for Observers

Observers can be important to your test for a variety of reasons, not the least of which is being able to use their notes later. Vou should encourage people to observe testing sessions regardless of whether you have a laboratory setup or are using a less formal arrangement.
Whether your observers will be in the room with you and the participant or in a separate observation area from the test participant, it is always a good idea to give observers some direction, because this may be an unusual activity for them. Vou should at least develop a list of pointers about what to do and not do while observing.

Prepare Test Materials - II

■ Post-test questionnaire
■ Debriefing topics guide

All the parts can be rolled up into what we sometimes call a session script or J a session checklist. Vou don't have to do alJ of the materials for every test. Create materials that support the goals of your test and that will deliver the data needed to answer your research questions. For example, for small usability tests you can typically get what you need about participants from the screener, then interview participants briefly at the beginning of sessions — so there are no background or pre-test questionnaires. There usually is no training

required for participants to take part, and enough preference data comes out organically during the session that having a post-test questionnaire is overkill or there is no need to compare impressions between the beginning and end of sessions. Let your research questions and test design dictate the combination of the items you need to make your test work.

In this chapter, we also cover considerations for preparing yourself to test a prototype or an incomplete product. There is a special section on common question formats, as well. Although the information about question formats follows the discussion about the post-test questionnaire, you can use these formats for any of the questionnaires included in your usability test design.

Prepare Test Materials - I

One of the more labor-intensive activities required to conduct a usability test is developing the test materials that will be used to communicate with the participants, collect the data, and satisfy legal requirements. It is important to develop all required test materials well in advance of the time you need them. Apart from the obvious benefit of not having to scurry around at the last minute, developing materials early on helps to explicitly structure and organize the test. In fact, if you have difficulty developing one particular type of test material, this can be a sign that there are flaws in your test objectives and test design.

While the specific content of the materials will vary from test to test, the general categories required will hardly vary at all. This chapter contains a list of the most common materials you need to develop a test, as well as examples of the various types of test materials. As you develop them, think of these materials as aids to the testing process. Once they are developed, their natural flow will guide the test for you. Be sure to leave enough time to include the materials in your pilot test. The test materials reviewed in this chapter are as

follows:

■ Orientation script
■ Background questionnaire
■ Data collection instruments
- Nondisclosure agreement and recording consent form
■ Pre-test questionnaire
— Task scenarios

Protect Participants' Privacy and Personal Information

If you collect personal information about participants, you have a responsibility to protect that information and keep it private. For example, if you must collect information such as participants' Social Security number because your accounting department asks for it, ensure that it is only used for that purpose.

Don't include it in materials distributed to others, such as a schedule of sessions or the final report. Do tell participants why you are collecting the personal information and what you are going to do with it. You should also tell them how long you are going to keep it.

Keep the identities of your participants anonymous. During the sessions, try to minimize how much you say their names, especially if the session is being recorded (and even more if the recordings will go into highlights videos that will be seen widely in your company). Try to strike a balance between being personable during the time participants are with you and protecting their privacy later.

Don t use names to laoei recordings or rues. iou win neeu pui naura session schedules to ensure that the right people show up and that they match the selection criteria. However, you should never use names — not even just first names — in reports. Refer to the participants by some other identifier, such as Ul, U2, etc. (for user one, user two, etc.) or PI, P2, and so on (for participant one, participant two, etc.).

Absolutely do not share video or audio recordings of participants for any reason other than the reasons that participants agreed to. You should have participants sign recording permissions that spell out that you are using the recordings onlv for data analysis and internal reports. Once the sessions are complete, store the recordings securely, either on media that can be locked away or digitally protected with passwords.

Compensate Participants - II

Even in the case where you have an ongoing research relationship with your customer, if participants must be imported to your location your company should offer to foot all the expenses such as travel and meals associated with having the person participate. If no strong relationship exists, you could prov ide anything from a small honorarium to paying the person's salary for the time he or she participated. Some other compensation choices are: free or discounted products from your company's sales catalog, gift certificates, or a simple memento, such as a T-shirt or calculator. ver you like. While vou should always provide something of value in appreciation for participation, it is of the utmost importance that you do not imply in any way that compensation is tied to the person's performance. Participants simply get paid for showing up and giving it their best, and that's it. Even joking about this with the person is not a good idea, as it plants a seed in the person's mind that he or she needs to be "positive," which may prevent the person from being critical.
Some workplaces, unions, or professions don't allow people to take incen- tives, honoraria, or gifts. If someone wants to take part in your study but declines the incentive, find another way to thank them for their participation.

Compensate Participants - I

It is customary to compensate those who serve as participants, although^ need not always be monetary- compensation. Participants acquired off the street" or via an agency for a study where your company has been kep anonymous should all receive monetary compensation and possibly a small token of appreciation such as a pen or a T-shirt. Expect to pay the going rate on the open market for the skill level you are requesting. If your product will be used by administrative personnel, expect to compensate them by paying

the hourly rate for administrative assistants. If you will be testing lawyers or doctors or other professionals, you may need to pay close to the hourly rate for that profession. If you will simply be testing with typical "consumers," with no specific profession required, expect to pay an honorarium of about $50 per hour, depending on location (participants on the coasts or in large cities may be more expensive than people in smaller cities or rural places in the middle of the country). Participants are always happy to accept cash; checks or gift checks will also do, but try to make sure participants get the check at the end of the session rather than by mail later. You may need to test in the evening or on weekends if your user profile indicates end users who are not available during the regular work day. If money is no object, however, you can get

people to come in wheneYour company may require participants to sign receipts showing that they

have receiv ed a check or cash in exchange for their participation. Check with your accounting and/or legal department to determine what your company requires.

Participants sent from one of your company's large clients, however, should be compensated differently depending on the relationship of your two companies. If their participation is part of an ongoing research relationship where thev benefit from being privy to your company's future product line as much as vou benefit from their input, then a token gift alone might suffice. 

Schedule and Confirm Participants - II

Then confirm, confirm, and confirm again. The day before the participant is scheduled, do another reminder with the same information that was in the scheduling communication. On the afternoon or evening before the partici- pant's session, phone him or her. Try very hard to get the person rather than leaving a message; this increases the chances that the person will show up and do it on time.

Schedule and Confirm Participants - I

As you interview participants to see whether they meet the selection criteria, we recommend that you collect their top three available dates and times as well but not make any immediate commitments. This gives you greater flexibility in moving participants around on your personal scheduling Ouija board before finalizing the session schedule and mix of participants.

When you have the availability information, you can also set priorities among the pool of participants. If there are people in the pool whom you definitely want to be in the study, but they have limited or special times

available, schedule them first and then schedule other people with more flexible schedules around them. Availability may be tied to the incentive offered. If you're having difficulty finding and scheduling qualified participants, you may need to increase the amount that participants will be compensated for their time, or flex your session schedule to nonworking hours.

Once you have deeded who should be in which sessions, send an email (or if email ,s not available, phone or send a letter if there is time) to the participant with the information about the appointment time, date, and location Include any special instructions (such as where visitor parking is or requirements for checking in at reception), along with a reminder about the amount and conditions for getting the incentive. In your follow-up communication ask participants to acknowledge that they have received it by replying to you

Expect to A/lake Tradeoffs

Because to some extent your participant characteristics are a projection of who you and the team are hoping the participants should be, you may find that the pool of respondents you have to select from meet some of

the requirements you want, but not all. This is when it is useful to have interviewed participants conversationally rather than by going through the multiple choice questionnaire by rote. The conversational screening allows you to ask follow-up questions that could help you decide whether a respondent is close enough to the visualized profile to give you valid data. If you or your agency use the straight questionnaire and are finding that it is difficult to schedule enough suitable participants, you and your agency should

review where the respondents are not meeting the requirements and adjust accordingly. The agency may have learned something useful that will help you eventually secure an adequate number of participants and so could go back to already interviewed respondents to ask follow-up questions. These last minute trade offs are not uncommon, and as long as you document them and account for any biases they may have introduced, it is fine to make them.

Beware of Inadvertently Testing Only the "Best" People - II

« When you are acquiring people from an established customer through your own sales force. In this case, similar to the previous one, there is also a hidden agenda. Your sales rep very often has strong relationships

that he or she would like to further by using the testing as a perk. These relationships are often with the best performers and most influential people in your customer's organization. You never see the average or poor performers.

We mention these situations as warnings because "best" people are hard to predict. We have learned this the hard way, having had people show up as participants who were so accomplished that they should have been designing the product, never mind using it. Such users were able to foresee and work around the most troublesome areas, almost as if they knew they were there.

These participants also downplayed whatever problems they encountered, although they were critical ones. They can also be the most critical. One or at most two of these people during a test is reasonable, but if you see three or four, it plays havoc with your test results.

What makes the situation even worse is when the development team attending the test does not agree that these people are "ringers." The team is more than haPPv to bask in the glow of receiving unexpected but excellent results. Now, if you could just get all "ringers" to buy the product in sufficient numbers, everyone would be happy.

Beware of Inadvertently Testing Only the "Best" People - I

Very often when acquiring participants, and especially when you do not have primary control of participant selection, you will be sent only the very "best" people by your contact or recruiter. By "best," we mean that regardless of the category of end user, whether it is novice, expert, or whatever, you are sent the cream of the crop, the high achievers. Your first reaction to this might be very positive, but on close inspection, it is easy to see the potential problem here. The "best" end users typically possess the skills to plow through even the most hard-to-use products and perform admirably. Consequently, the product "tests out" much better than it should, and provides a false sense of confidence to the design and marketing team. Later, after release, when average and poor performers use the product, many of the design flaws that exist, but did not
come out during the test, are exposed.

This situation is especially apt to occur when you are acquiring participants under the following conditions:
■ When you will be testing end users from within your own company, usually from a department with which you are not familiar.

■ When you are acquiring people directly from an established customer and participation is seen as an enviable perk. In this case, the manager responsible for providing people sends participants to your company as a reward for a job well done at their company. Invariably, these are their best performers.

Include a Few Least Competent Users in Every Testing Sample

We have found that we learn an extraordinary amount by including one or more least competent users (LCUs) among participants, even if they do not make up a significant percentage of the eventual end users. An LCU is defined as an end user who represents the least skilled person who could potentially use

your product. In the example of the user profile for the chemical engineeringmarket, the LCU is a person with no computer experience, who has never used even a word processor, who is a high-school graduate, and so on. The LCU need not fall at the bottom of all the scales, but the LCU should be at the bottom of the majority of them. Why include LCUs even if the user profile is projected at mostly expert users? Simply this: If your least expert group can successfully use the product, if the]/ can muddle their way through the usability test, then you can assume that most other groups will also be able to use the product.

Of course, there are exceptions to every rule, but by and large, we have found the LCUs to be excellent indicators of a product's overall ease of learning. On the other hand, if the LCUs cannot get through the test, that is not necessarily an indictment. It does, however, reveal clues on how to fix fundamental problems of intuitiveness, orientation, or organization through redesign, more or better information in the instructions, and so forth. In addition, during early product development and exploratory tests, you learn much about the end

user's mental models through the eyes of the LCU, before the LCU has been

As You Eliminate or Accept People, Mark Them Off on Your List

Tin ere is nothing more embarrassing than calling the same person and getting them out of the shower two days in a row. Gather responses to the questions either on copies of the screener or in a spreadsheet like the one in Figure 7-5, and use it to maintain the data you gather throughout the recruiting and screening process. (For an expanded sample of such a spreadsheet, go to the web site that accompanies this book: www.wiley.com/usabilitytesting.com.)

Co through the Questions in the Questionnaire

Based on your test plan's user profile and the questions for the screener, question the person about his or her qualifications, such computer experience, job responsibilities, and equipment used.
■ NOTE if you are using an outside agency, have the agency first try the questionnaire on you as if you were a potential participant See how the person who calls you responds to different and/or ambiguous answers to the same question. It is your money and your study. Make sure that the agency does it right.
If possible, use the questionnaire as a checklist for managing a more conversational interview, turning the questions into open-ended queries so as to get the respondents to volunteer information about themselves. This way, the respondent is less likely to be able to guess the "right" answers to get into the test, and you may learn additional things that could be useful to know.

Explain Why You are Calling and How You Cot the Contact Information

Briefly explain the nature of the research. For example: Michael Reel at the Franklin County Clerk's Office suggested I contact you about a research study I am working on. I am working with him and the Office of the Secretary of State to evaluate how usable a new web site is for registered voters. To do this, my colleague Dana would like to observe you using the web site and then ask you a couple of questions. We're doing individual 30-mimite sessions all day on Tuesday, August 28. Does this sound like something you might be interested in? 

By all means, be sure to mention that the session requires the participant to be recorded, because you want participants to raise any objections to being recorded now, and not when they arrive for testing. Ask the potential participant if he or she is interested in hearing more, and, if so, explain how long (worst case) it will take you to ask all the questions on the screening questionnaire. (If it is a really long questionnaire, fudge a little bit about how long it will take. Many professionals simply say that it depends on the answers, which is true, we suppose.) If the person is not interested, then express your thanks, mention how much fun he or she will be missing, and, if appropriate, ask for recommendations of other interested parties.

Inform the Potential Participant Who You Are

If the usability test is being run as an anonymous one (the name of the sponsoring company remaining confidential), then of course the research firm or person calling should mention this fact and give the reasons for anonymity. 

For example, "Hi, my name is Sandy. I'm calling from a research firm called UsabilityWorks." If the study is anonymous, add something like this: "Because my client is a household name, I can't tell you the sponsor of the research. You will learn it at the end of your session, if you are selected for the study," if that is indeed the case.

Conduct Screening Interviews

Mow that you have the questionnaire developed (see the section "Write the Screening Questionnaire/' earlier in this chapter), here are some guidelines for administering it over the phone or having someone do it for you. The guidelines work for email communication, too.

Use the Questionnaire or Open-Ended Interview Questions?

We have found that the quality of participants and the "show rate" is higher if you recruit in an apparently more casual and personable way, using a screener as a checklist during a more conversational interview. Here you can ask open-ended questions to get the respondents to volunteer information about themselves that will reveal whether they qualify. There are several advantages to this approach. First, you build rapport with the candidate, he rapport results in a stronger feeling of responsibility and commitment in the participants. They are more likely to show up for the session and be on time. Second, it is less likely that the candidate will give an answer just because he or she thinks it is the correct one to get into the study. Third, you can as follow-up questions that may expand or clarify something important study. Finally, you can make better on-the-flv adjustments to your selection criteria. Tins doesn't mean that you're biasing the sample, but instead it helps you incorporate things you learn as you interview respondents and make intelligent tradeoffs if some requirement or other becomes difficult to meet For example, Dana worked on a study of a web site for registered voters 

She wanted people from both rural and urban places (a classifier), but the study took place at harvest time. Many of the people in rural places would only be available after sundown (and a very, v ery hard day) or if it rained If Dana had used only a scripted questionnaire, she might not have found out Poople weren't available. But in a casual conversation, she found out whv and what the options were for getting these types of participants.

Screening Considerations

We highly recommend that the interviewer not read off the answer choices but instead simply check off the choice that is closest to the respondent's answer. A screener is an excellent tool to use if you have never recruited participants before or if you plan to use an agency to do recruiting.