Select a Data Collection Method

Once you are clear about the type of data you want to collect and how it will help you to achieve the test objectives, the next challenge is to develop the means for collecting that data. In terms of data collection instruments, you are limited only by your imagination, resources, and the time required to develop

the instruments. Will you have help with the collection? Will you have help reducing and analyzing the data once it is collected? It makes no sense at all to design a data collection method that requires extensive analysis of 20 hours of video recordings if you only have 2 weeks after the test in which to provide a

test report.

Envision yourself creating the test report and even making a presentation to members of the team. Visualize the type of findings you will want to report if not the actual content. Then, given the amount of time and resources at your disposal, plan how you will get to that point once the test has ended. Your data col echon effort should be bounded by that constraint, unless you realistically feel that you or someone else will be able to analyze the additional data later

Review the Research Question(s) Outlined in Your Test Plan

If after reviewing these, you have a difficult time ascertaining what data to collect, regard that as an important message. More often than not, it means that you need to clarify the research question(s) to make them more specific This may require reinterviewing the designers and developers and educating them as well. 6
Decide What Type of Information to Collect
Match the type of data you'll collect to a problem statement of the test plan. Figure 8-5 shows several matchups of problem statements with data collected.

Data Collection Tools - II

For simplicity's sake, data collected during a test falls into two major categories:

- Performance data: This consists of objective measures of behavior, such as error rates, time, and counts of observed behavior elements. This type of data comes from observation of either the live test or review of the

video recording after the test has been completed. The number of errors made on the way to completing a task is an example of a performance measure.

. Preference data: Preference data consists of the more subjective data that measures a participant's feelings or opinions of the product. This data is tvpically collected via written, oral, or even online questionnaires or

through the debriefing session after the test. A rating scale that measures how a participant feels about the product is an example of a preference measure.

Both performance and preference data can be analyzed quantitatively or qualitatively. For example, on the performance side, you can analyze errors quantitatively simply by counting the number of errors made on a task. You can also analyze errors qualitatively to expose places where the user does not understand the product's conceptual model.

On the preference side, a quantitative measure would be the number of unsolicited negative comments a participant makes. Or, qualitatively, you can analyze each negative comment to discover what aspect of the product's design the comment refers to.

In terms of the product development lifecycle, exploratory (or formative) tests usually favor qualitative research, because of the emphasis on the user's understanding of high-level concepts. Validation (or summative) tests favor quantitative research, because of the emphasis on adherence to standards or measuring against benchmarks.

Following are examples of performance data.

Data Collection Tools - I

Taking notes during the typical usability testing session can be incredibly difficult. If you are moderating the test and taking notes yourself, your attention will be divided between recording what you observe and observing what is happening now. We strongly encourage you to enlist someone else to take notes or record data if at all possible. If it isn't possible, you should give even greater consideration to designing the most efficient, effective data collection tools (keeping in mind that by "data collection tool" we mean anything from a basic Word document with space for notes to sophisticated tracking software).
The purpose of the data collection instruments is to expedite the collection of all data pertinent to the test objectives. The intent is to collect data during the test as simply, concisely, and reliably as possible. Having a good data collection tool will assist analysis and reporting as well.
There are many data measures from which to choose, and these should be tied back to the test objectives and research questions. Let us not get ahead of ourselves though. Before simply collecting data, you need to consider the following six basic questions:
- What data will address the problem statement(s) in your test plan?
■ How will you collect the data?
■ How will you record the data?
■ How do you plan to reduce and analyze the data?
■ How and to whom will you report the data?
- What resources are available to help with the entire process?
The answers to these questions will drive the development of the instruments, tools, and even the number of people required to collect the data.
Data collection should never just be a hunting expedition, where you collect information first, and worry about what to do with it later. This holds true even for the most preliminary type of exploratory testing. If you take that approach, you run the risk of matching the data to hoped-for results.
Also, an imprecise shotgun approach typically results in an unwieldy amount of data to reduce and analyze, and tends to confuse more than enlighten. The type of data you collect should be as clear in your mind as
possible before the test and should be tied directly to the questions and issues you are trying to resolve.

Test the Questionnaire

Try the questionnaire out on someone who fits the user profile or even on a colleague. It is amazing how easy it is for ambiguity to sneak in. Piloting the background questionnaire is just as important as pilot test.ng he other materials for the test, such as the screening questions (see Chapter 7), ana session script (discussed later in this chapter).

Make the Questionnaire Easy to Fill Out and Compile

Design the questionnaire for the ease of both yourself (in analyzing the responses) and the participants (in having to remember their history), by avoiding open-ended questions. Have the participants check off boxes or circle answers. This will also minimize their time filling out the questionnaire (important if they will be filling it out the day of the test) and will decrease the number of unintelligible answers. You may want to automate the questionnaire by using a survey tool or other online form maker.

Focus on Characteristics That May Influence Performance

Ascertain all background information that you feel may affect the performance of the participants. This could expand on the classifiers you specified in the screening process. Similarly, to develop screening questions when you are recruiting participants, form questions that focus on behaviors that vou are interested in exploring. For example, in a study for an entertainment news web site, you might collect information about the last time the participant downloaded shows or movies from similar web sites. However, unlike screening, now you can ask more questions about participants that could set a context in which to analyze the performance data from the session. For example, for the test of the entertainment news web site, you could ask about other, similar interests or habits such as magazine purchases or what the last five shows or movies were that participants watched and in what venue.

Background Questionnaire - II

To confirm that the "right" people show up. It is amazing how often mix-ups occur when there are so many details to manage. If you did not make the phone calls or write the emails yourself to screen and select
participants, it is important to verify that the people who show up actually possess the skills and knowledge you expected. It is not that unusual for agencies to misunderstand what you are doing and to send unqual-

ified people. If you do get the wrong people showing up, you will need to decide on the spot whether to use them or release them. You will also need to communicate to the person or organization supplying your par-

ticipants that they need to do a better job of qualifying the participants.

To provide a synopsis of each participant for the test moderator and for product team members who observe the test. If you anticipate that the usability tests will be observed by a design team or other inter-

ested parties, it is important that they know the background of each person while they observe the test. It is both confusing and misleading to observe a test without a sense of the skills, knowledge, and experi-

ence of the specific participant. There is no basis on which to judge how participants are doing or why they are performing as they are without this knowledge. To avoid this potential misunderstanding, make the
data from the screening questionnaire and the background questionnaire available to the observers after the participant fills it out. The observers can reference it while the test proceeds.

Background Questionnaire - I

The background questionnaire provides historical information about the participants that will help you to understand their behavior and performance during a test. It is composed of questions that reveal the participant's experience, attitudes, and preferences in all areas that might affect how they perform.

For example, if you are testing a database management system (DBMS), it will be helpful to know if the participants have used a DBMS before, and, if so, which one(s) and for how long. While you will not know if that experience will affect their performance negatively or positively, you almost certainly know that it will affect their performance differently than a person without DBMS experience.

The background questionnaire is typically filled out just prior to the test. 

Sometimes, particularly if it is lengthy, you might mail it to the participant ahead of time.

The information you include in the background questionnaire is initially culled from the participant profile in your test plan. The background questionnaire is similar to a phone screener, although more detailed. The phone screen need only determine if a potential participant meets the selection requirements and can be classified in a user group. The background questionnaire, however, goes further by exploring previous training and experience. This more specific information can help explain a participant's behavior during the test. Perhaps the participant is choosing buttons or menu selections based on expectations formed from using a competitive product that no other participant used.

In addition to the previously stated reasons for acquiring the correct crosssection of participants and providing insight about each person's performance from a historical perspective, there are two more purposes for the background questionnaire. Both come into play on the day of the test, just prior to its

beginning.

Refer to Any Forms That Need to Be Completed and Pass Them Out

This includes background questionnaires, pre-test questionnaires, permissions, and so on.
Figures 8-2 and 8-3 show examples of orientation scripts for different types of tests.

Ask for Any Questions

Before you begin, be absolutely sure that the participants understood your instructions. Due to being nervous and/or poor acoustics in the room, the participants may not have fully heard or understood your instructions. If you are not sure, ask them to parrot back a particular point by inquiring, for example, "Do you remember how to use the thinking aloud protocol

Mention That It Is Okay to Ask Questions at Any Time

Of course, explain that you may not answer those questions in order to simulate the situation of their being alone and having to rely on their own resources and materials at hand. Make that aspect of your role very clear. You are not there to solve problems participants encounter.

Explain Any Unusual Requirements

Demonstrate and practice how these special situations work and reassure the participant that you will be available to remind him or her how to do it, if need be.

Explain Any Unusual Requirements

Demonstrate and practice how these special situations work and reassure the participant that you will be available to remind him or her how to do it, if need be.

Assure the Participant That He or She Is Not Being Tested

This is probably the most familiar adage of testing, and you should certainly say this. However, do not hold out hope that they will necessarily belie e youjust because you say it. This slogan has become the -it's for your own good slogan of our youth. It is often repeated but never believed' Only the manner in which the test is conducted, the way in which you react to the person's behavior, and the behavior of the observers will cause the truth of this adage to sink in. Your manner, body language, and voice modulation during difficulties all communicate much more than just the words. In sum, do not be surprised if at the first sign of difficulty, you hear the participant mutter that familiar refrain, "Oh, I'm an idiot. That wasn't the program's fault," or "1

just need more time to learn how to use it."

Explain What Is Expected of the Participant

Describe how the usability test will proceed without providing every last detail. Broach the subject of nondisclosure, if you have not already done so, and how that wiJl be handled. Encourage them to perform as they normally would (e.g., same speed and attention to detail, given the fact that it is an artificial situation). Encourage the participants to ask questions and to take breaks if they need to.
Avoid any reference whatsoever to your expectations of their behavior or performance. Remain absolutely neutral in terms of their expected performance.
For example, do not say any of the following in order to make participants less nervous:
■ "Most people find this extremely easy."
■ "We brought you in for an extremely simple test."
■ "I'm sure you'll have no difficulty with this product, so don't be nervous."
While well intentioned, these are exactly the wrong things to say. By making those references, you have essentially put the participants on the defensive if things do get difficult. At the slightest hint of adversity, they may begin to hurry and try harder in order to fulfill your expectations. After all, if it is simple and they are having a hard time, then by definition thev must  No one likes to think of himself or herself that way.

Describe the Testing Setup

Point out and describe the equipment. Let the participants know whether they will be staying where they are, moving to another room, and so on. Locate the restrooms. Let them know if:
- People are watching from behind a one-way mirror or from another room via cameras. Do not get cute here and say "Oh that old thing. It's just so we can see all sides of the equipment."
■ The session is being recorded. It is never a good idea to lie to participants about being observed or recorded in order twt to make them nervous.
First of all, it is not ethical. Second, once the test starts, almost all participants forget their concerns about being watched and recorded, depending on the testing environment.

Explain Why the Participant Is Here

You may think that they already know this information, but you would be amazed at what external recruiters tell participants about what they will be doing in the sessions. Provide enough detail and context about the product for them to perform the tasks. Do not feel you need to provide product history, number of participants being tested, and so forth. Express appreciation tor their willingness to participate and how much their input helps Pr0^ a better product, regardless of how they perform. In no way, shape, or should the performance of the participants be tied to their compensation for participating, even as a joke. Vou can be sure that the thought has crossed their mind. No need to reinforce it.

Offer Refreshments

Being offered refreshments will help the participants to relax and feel at home They are more apt to indulge if you already have a cup of coffee, soda, or water in your hand. "Breaking bread" together is a wonderful icebreaker. Do not downplay the fact that your participants may be nervous. It is very common and you need to address it.

Make Introductions

Introduce yourself, of course, and anyone else whom the participant is likely to encounter during the usability test. You need not go into great detail about people's backgrounds; just a passing reference to the function of each person is fine. For example, you might say, "That person over there will be managing the recordings," or "That person over there will be keeping track of time to ensure that I let you go on time." New volunteer the fact that any person associated with the test or observing in the same room has worked on the product that the participants will be testing. Of course, if a participant asks (hardly anyone ever does), then by all means tell the participant the truth, but do not volunteer that information. The reason for this is simple; you want the

participants to feel absolutely unencumbered about providing any negative feedback. Associating real people with the product only makes it that much more difficult for participants to be honest.

As test moderator, include yourself in this "non-association" guideline if you are affiliated with the product. (Ideally, you will not be affiliated with the product.) Explain your role as the person who will be moderating the test, observing, and taking notes. If you are not affiliated with the product, then play up the fact that you are a neutral observer. If you will be moderating the usability test in the test room, then explain exactly what you will be doing during the test.

Write the Orientation Script Out

Now that you know the important reasons for creating a script, you need to actually create one. The following sections list the typical contents of an orientation/introduction script. However, do not feel that you need to include every category in your own script. If you do, you might end up writing an essay.

Plan to Read the Script to Each Participant Verbatim - II

. People are easily influenced by past events. Imagine that you have just tested the first four participants and are about to test the fifth one. Three of the first four have performed miserably. Not only did they have great

difficulty with the product, but they hated using it. You may be feeling down and discouraged, and are unknowingly about to project your feeling of frustration to the fifth participant by the manner in which you

introduce the session. However, reading the script forces you to use the same language, which in turn makes it easier to control the nonverbal aspects of vour communication, such as your mannerisms, expressions,

and voice modulation. In short, it is harder to express your own frustration when you read the script.

More than one test moderator may be conducting the test over a period of time. If that is so, there is a need to minimize the differences in the test moderators as much as possible. At the very least, reading

the script controls the initial information that is communicated to the participants.

Those members of the development team not present at the sessions will want to know precisely what was said to the participants. Showing the script to interested parties who were unable to attend the usability test

accomplishes this objective. It also communicates professionalism and rigorousness on your part.

You may forget an important point. There are so many details involved with moderating a test, why make it hard on yourself? Use the script and cross "what to say" off your list of things to remember and worry
about.

Plan to Read the Script to Each Participant Verbatim - I

Do not attempt to memorize the script, paraphrase it, or simply "wing it" from session to session. Here is why this is so important: 

- You want to present the exact same information to each participant so that all the participants are exposed to identical conditions prior to the test. By paraphrasing the script, you may change what you say in very

subtle ways.

NOTE If you feel self-conscious reading the script or feel that it sets an overly formal tone, simply tell the participant why you are reading it For example. I'm going to read this script to you now so that I provide the exact Instructions to you that I provide to everybody else, and so that I do not

forget anything of importance."

Keep the Speech Short - II

Keep the Speech Short - I

Unless you have an extremely complex test, limit the orientation script to a few paragraphs. Anything longer than that will not be retained by the participants in any case. Any instructions beyond two pages probably means that you are including actual test materials, such as task scenarios, as part of your script. We show two examples in Figure 8-2 and 8-3. Figure 8-2 shows a script you might use for a simple, exploratory test. Figure 8-3 has very specific instructions for a benchmark comparison test.

Keep the Tone of the Script Professional but Friendly

It should not be chummy or overly familiar as if the participants were your buddies. For example, in the food service industry, the waiter who asks for your name and continues to use it ad nauseam during the meal comes to mind as an instance of this lack of professionalism in the guise of improved service.

Orientation Script

The orientation script (also known as the introduction script) is a communications tool meant to be read verbatim to participants. It describes what will happen during a test session, sets the tone for that session in the minds of the participants, and is intended to put them at ease. It achieves this by informing the participants of what they will be doing, and reinforcing the fact that the product, not the participant, is being tested. Remember that participants often have only the vaguest idea of what they will be doing, possibly having been

presented with only some ambiguous reference to "participation in market research" or the like. For a particularly nervous participant, introducing the session this way can provide reassurance that he or she is actually the right person in the "right" place.

The script may be read to the participants in the testing area or in a "waiting or meeting area" prior to moving to the testing space. Our own preference is to read it just before beginning testing activities, which occur in the main testing area.

When developing an orientation script, there are three major guidelines to remember.

Guidelines for Observers

Observers can be important to your test for a variety of reasons, not the least of which is being able to use their notes later. Vou should encourage people to observe testing sessions regardless of whether you have a laboratory setup or are using a less formal arrangement.
Whether your observers will be in the room with you and the participant or in a separate observation area from the test participant, it is always a good idea to give observers some direction, because this may be an unusual activity for them. Vou should at least develop a list of pointers about what to do and not do while observing.

Prepare Test Materials - II

■ Post-test questionnaire
■ Debriefing topics guide

All the parts can be rolled up into what we sometimes call a session script or J a session checklist. Vou don't have to do alJ of the materials for every test. Create materials that support the goals of your test and that will deliver the data needed to answer your research questions. For example, for small usability tests you can typically get what you need about participants from the screener, then interview participants briefly at the beginning of sessions — so there are no background or pre-test questionnaires. There usually is no training

required for participants to take part, and enough preference data comes out organically during the session that having a post-test questionnaire is overkill or there is no need to compare impressions between the beginning and end of sessions. Let your research questions and test design dictate the combination of the items you need to make your test work.

In this chapter, we also cover considerations for preparing yourself to test a prototype or an incomplete product. There is a special section on common question formats, as well. Although the information about question formats follows the discussion about the post-test questionnaire, you can use these formats for any of the questionnaires included in your usability test design.

Prepare Test Materials - I

One of the more labor-intensive activities required to conduct a usability test is developing the test materials that will be used to communicate with the participants, collect the data, and satisfy legal requirements. It is important to develop all required test materials well in advance of the time you need them. Apart from the obvious benefit of not having to scurry around at the last minute, developing materials early on helps to explicitly structure and organize the test. In fact, if you have difficulty developing one particular type of test material, this can be a sign that there are flaws in your test objectives and test design.

While the specific content of the materials will vary from test to test, the general categories required will hardly vary at all. This chapter contains a list of the most common materials you need to develop a test, as well as examples of the various types of test materials. As you develop them, think of these materials as aids to the testing process. Once they are developed, their natural flow will guide the test for you. Be sure to leave enough time to include the materials in your pilot test. The test materials reviewed in this chapter are as

follows:

■ Orientation script
■ Background questionnaire
■ Data collection instruments
- Nondisclosure agreement and recording consent form
■ Pre-test questionnaire
— Task scenarios

Protect Participants' Privacy and Personal Information

If you collect personal information about participants, you have a responsibility to protect that information and keep it private. For example, if you must collect information such as participants' Social Security number because your accounting department asks for it, ensure that it is only used for that purpose.

Don't include it in materials distributed to others, such as a schedule of sessions or the final report. Do tell participants why you are collecting the personal information and what you are going to do with it. You should also tell them how long you are going to keep it.

Keep the identities of your participants anonymous. During the sessions, try to minimize how much you say their names, especially if the session is being recorded (and even more if the recordings will go into highlights videos that will be seen widely in your company). Try to strike a balance between being personable during the time participants are with you and protecting their privacy later.

Don t use names to laoei recordings or rues. iou win neeu pui naura session schedules to ensure that the right people show up and that they match the selection criteria. However, you should never use names — not even just first names — in reports. Refer to the participants by some other identifier, such as Ul, U2, etc. (for user one, user two, etc.) or PI, P2, and so on (for participant one, participant two, etc.).

Absolutely do not share video or audio recordings of participants for any reason other than the reasons that participants agreed to. You should have participants sign recording permissions that spell out that you are using the recordings onlv for data analysis and internal reports. Once the sessions are complete, store the recordings securely, either on media that can be locked away or digitally protected with passwords.

Compensate Participants - II

Even in the case where you have an ongoing research relationship with your customer, if participants must be imported to your location your company should offer to foot all the expenses such as travel and meals associated with having the person participate. If no strong relationship exists, you could prov ide anything from a small honorarium to paying the person's salary for the time he or she participated. Some other compensation choices are: free or discounted products from your company's sales catalog, gift certificates, or a simple memento, such as a T-shirt or calculator. ver you like. While vou should always provide something of value in appreciation for participation, it is of the utmost importance that you do not imply in any way that compensation is tied to the person's performance. Participants simply get paid for showing up and giving it their best, and that's it. Even joking about this with the person is not a good idea, as it plants a seed in the person's mind that he or she needs to be "positive," which may prevent the person from being critical.
Some workplaces, unions, or professions don't allow people to take incen- tives, honoraria, or gifts. If someone wants to take part in your study but declines the incentive, find another way to thank them for their participation.

Compensate Participants - I

It is customary to compensate those who serve as participants, although^ need not always be monetary- compensation. Participants acquired off the street" or via an agency for a study where your company has been kep anonymous should all receive monetary compensation and possibly a small token of appreciation such as a pen or a T-shirt. Expect to pay the going rate on the open market for the skill level you are requesting. If your product will be used by administrative personnel, expect to compensate them by paying

the hourly rate for administrative assistants. If you will be testing lawyers or doctors or other professionals, you may need to pay close to the hourly rate for that profession. If you will simply be testing with typical "consumers," with no specific profession required, expect to pay an honorarium of about $50 per hour, depending on location (participants on the coasts or in large cities may be more expensive than people in smaller cities or rural places in the middle of the country). Participants are always happy to accept cash; checks or gift checks will also do, but try to make sure participants get the check at the end of the session rather than by mail later. You may need to test in the evening or on weekends if your user profile indicates end users who are not available during the regular work day. If money is no object, however, you can get

people to come in wheneYour company may require participants to sign receipts showing that they

have receiv ed a check or cash in exchange for their participation. Check with your accounting and/or legal department to determine what your company requires.

Participants sent from one of your company's large clients, however, should be compensated differently depending on the relationship of your two companies. If their participation is part of an ongoing research relationship where thev benefit from being privy to your company's future product line as much as vou benefit from their input, then a token gift alone might suffice. 

Schedule and Confirm Participants - II

Then confirm, confirm, and confirm again. The day before the participant is scheduled, do another reminder with the same information that was in the scheduling communication. On the afternoon or evening before the partici- pant's session, phone him or her. Try very hard to get the person rather than leaving a message; this increases the chances that the person will show up and do it on time.

Schedule and Confirm Participants - I

As you interview participants to see whether they meet the selection criteria, we recommend that you collect their top three available dates and times as well but not make any immediate commitments. This gives you greater flexibility in moving participants around on your personal scheduling Ouija board before finalizing the session schedule and mix of participants.

When you have the availability information, you can also set priorities among the pool of participants. If there are people in the pool whom you definitely want to be in the study, but they have limited or special times

available, schedule them first and then schedule other people with more flexible schedules around them. Availability may be tied to the incentive offered. If you're having difficulty finding and scheduling qualified participants, you may need to increase the amount that participants will be compensated for their time, or flex your session schedule to nonworking hours.

Once you have deeded who should be in which sessions, send an email (or if email ,s not available, phone or send a letter if there is time) to the participant with the information about the appointment time, date, and location Include any special instructions (such as where visitor parking is or requirements for checking in at reception), along with a reminder about the amount and conditions for getting the incentive. In your follow-up communication ask participants to acknowledge that they have received it by replying to you

Expect to A/lake Tradeoffs

Because to some extent your participant characteristics are a projection of who you and the team are hoping the participants should be, you may find that the pool of respondents you have to select from meet some of

the requirements you want, but not all. This is when it is useful to have interviewed participants conversationally rather than by going through the multiple choice questionnaire by rote. The conversational screening allows you to ask follow-up questions that could help you decide whether a respondent is close enough to the visualized profile to give you valid data. If you or your agency use the straight questionnaire and are finding that it is difficult to schedule enough suitable participants, you and your agency should

review where the respondents are not meeting the requirements and adjust accordingly. The agency may have learned something useful that will help you eventually secure an adequate number of participants and so could go back to already interviewed respondents to ask follow-up questions. These last minute trade offs are not uncommon, and as long as you document them and account for any biases they may have introduced, it is fine to make them.

Beware of Inadvertently Testing Only the "Best" People - II

« When you are acquiring people from an established customer through your own sales force. In this case, similar to the previous one, there is also a hidden agenda. Your sales rep very often has strong relationships

that he or she would like to further by using the testing as a perk. These relationships are often with the best performers and most influential people in your customer's organization. You never see the average or poor performers.

We mention these situations as warnings because "best" people are hard to predict. We have learned this the hard way, having had people show up as participants who were so accomplished that they should have been designing the product, never mind using it. Such users were able to foresee and work around the most troublesome areas, almost as if they knew they were there.

These participants also downplayed whatever problems they encountered, although they were critical ones. They can also be the most critical. One or at most two of these people during a test is reasonable, but if you see three or four, it plays havoc with your test results.

What makes the situation even worse is when the development team attending the test does not agree that these people are "ringers." The team is more than haPPv to bask in the glow of receiving unexpected but excellent results. Now, if you could just get all "ringers" to buy the product in sufficient numbers, everyone would be happy.

Beware of Inadvertently Testing Only the "Best" People - I

Very often when acquiring participants, and especially when you do not have primary control of participant selection, you will be sent only the very "best" people by your contact or recruiter. By "best," we mean that regardless of the category of end user, whether it is novice, expert, or whatever, you are sent the cream of the crop, the high achievers. Your first reaction to this might be very positive, but on close inspection, it is easy to see the potential problem here. The "best" end users typically possess the skills to plow through even the most hard-to-use products and perform admirably. Consequently, the product "tests out" much better than it should, and provides a false sense of confidence to the design and marketing team. Later, after release, when average and poor performers use the product, many of the design flaws that exist, but did not
come out during the test, are exposed.

This situation is especially apt to occur when you are acquiring participants under the following conditions:
■ When you will be testing end users from within your own company, usually from a department with which you are not familiar.

■ When you are acquiring people directly from an established customer and participation is seen as an enviable perk. In this case, the manager responsible for providing people sends participants to your company as a reward for a job well done at their company. Invariably, these are their best performers.

Include a Few Least Competent Users in Every Testing Sample

We have found that we learn an extraordinary amount by including one or more least competent users (LCUs) among participants, even if they do not make up a significant percentage of the eventual end users. An LCU is defined as an end user who represents the least skilled person who could potentially use

your product. In the example of the user profile for the chemical engineeringmarket, the LCU is a person with no computer experience, who has never used even a word processor, who is a high-school graduate, and so on. The LCU need not fall at the bottom of all the scales, but the LCU should be at the bottom of the majority of them. Why include LCUs even if the user profile is projected at mostly expert users? Simply this: If your least expert group can successfully use the product, if the]/ can muddle their way through the usability test, then you can assume that most other groups will also be able to use the product.

Of course, there are exceptions to every rule, but by and large, we have found the LCUs to be excellent indicators of a product's overall ease of learning. On the other hand, if the LCUs cannot get through the test, that is not necessarily an indictment. It does, however, reveal clues on how to fix fundamental problems of intuitiveness, orientation, or organization through redesign, more or better information in the instructions, and so forth. In addition, during early product development and exploratory tests, you learn much about the end

user's mental models through the eyes of the LCU, before the LCU has been

As You Eliminate or Accept People, Mark Them Off on Your List

Tin ere is nothing more embarrassing than calling the same person and getting them out of the shower two days in a row. Gather responses to the questions either on copies of the screener or in a spreadsheet like the one in Figure 7-5, and use it to maintain the data you gather throughout the recruiting and screening process. (For an expanded sample of such a spreadsheet, go to the web site that accompanies this book: www.wiley.com/usabilitytesting.com.)

Co through the Questions in the Questionnaire

Based on your test plan's user profile and the questions for the screener, question the person about his or her qualifications, such computer experience, job responsibilities, and equipment used.
■ NOTE if you are using an outside agency, have the agency first try the questionnaire on you as if you were a potential participant See how the person who calls you responds to different and/or ambiguous answers to the same question. It is your money and your study. Make sure that the agency does it right.
If possible, use the questionnaire as a checklist for managing a more conversational interview, turning the questions into open-ended queries so as to get the respondents to volunteer information about themselves. This way, the respondent is less likely to be able to guess the "right" answers to get into the test, and you may learn additional things that could be useful to know.

Explain Why You are Calling and How You Cot the Contact Information

Briefly explain the nature of the research. For example: Michael Reel at the Franklin County Clerk's Office suggested I contact you about a research study I am working on. I am working with him and the Office of the Secretary of State to evaluate how usable a new web site is for registered voters. To do this, my colleague Dana would like to observe you using the web site and then ask you a couple of questions. We're doing individual 30-mimite sessions all day on Tuesday, August 28. Does this sound like something you might be interested in? 

By all means, be sure to mention that the session requires the participant to be recorded, because you want participants to raise any objections to being recorded now, and not when they arrive for testing. Ask the potential participant if he or she is interested in hearing more, and, if so, explain how long (worst case) it will take you to ask all the questions on the screening questionnaire. (If it is a really long questionnaire, fudge a little bit about how long it will take. Many professionals simply say that it depends on the answers, which is true, we suppose.) If the person is not interested, then express your thanks, mention how much fun he or she will be missing, and, if appropriate, ask for recommendations of other interested parties.

Inform the Potential Participant Who You Are

If the usability test is being run as an anonymous one (the name of the sponsoring company remaining confidential), then of course the research firm or person calling should mention this fact and give the reasons for anonymity. 

For example, "Hi, my name is Sandy. I'm calling from a research firm called UsabilityWorks." If the study is anonymous, add something like this: "Because my client is a household name, I can't tell you the sponsor of the research. You will learn it at the end of your session, if you are selected for the study," if that is indeed the case.

Conduct Screening Interviews

Mow that you have the questionnaire developed (see the section "Write the Screening Questionnaire/' earlier in this chapter), here are some guidelines for administering it over the phone or having someone do it for you. The guidelines work for email communication, too.

Use the Questionnaire or Open-Ended Interview Questions?

We have found that the quality of participants and the "show rate" is higher if you recruit in an apparently more casual and personable way, using a screener as a checklist during a more conversational interview. Here you can ask open-ended questions to get the respondents to volunteer information about themselves that will reveal whether they qualify. There are several advantages to this approach. First, you build rapport with the candidate, he rapport results in a stronger feeling of responsibility and commitment in the participants. They are more likely to show up for the session and be on time. Second, it is less likely that the candidate will give an answer just because he or she thinks it is the correct one to get into the study. Third, you can as follow-up questions that may expand or clarify something important study. Finally, you can make better on-the-flv adjustments to your selection criteria. Tins doesn't mean that you're biasing the sample, but instead it helps you incorporate things you learn as you interview respondents and make intelligent tradeoffs if some requirement or other becomes difficult to meet For example, Dana worked on a study of a web site for registered voters 

She wanted people from both rural and urban places (a classifier), but the study took place at harvest time. Many of the people in rural places would only be available after sundown (and a very, v ery hard day) or if it rained If Dana had used only a scripted questionnaire, she might not have found out Poople weren't available. But in a casual conversation, she found out whv and what the options were for getting these types of participants.

Screening Considerations

We highly recommend that the interviewer not read off the answer choices but instead simply check off the choice that is closest to the respondent's answer. A screener is an excellent tool to use if you have never recruited participants before or if you plan to use an agency to do recruiting.

Screen and Select Participants

As we have mentioned a couple of times, you'll need to put together some type of process to determine whether the candidates you have qualify for the study. This usually takes the form of a document that contains a table of qualifications (see the sections "Define the Criteria for Each User Group," and "Choose the Number of Participants to Test," earlier in this chapter) and a set of multiple choice questions (see the previous section "Write the Screening Questionnaire"). The table lists requirements and classifiers along with a range of the number of participants you would like to possess these attributes. The multiple choice questions operationalize the selection/elimination process. The document, typically called a screcner, includes scripted language for starting a phone interview process, getting through the questions, scheduling

the participant (if qualified), and closing the interview.

Newspaper Advertisements

If you have plenty of lead time and want to specifically recruit people who are less technology savvy, advertise in your local newspaper either in a display ad area or the classified ad section for "participants needed for research study."

(Response rates are likely to be higher for display ads than for classifieds.) You might also try online versions of the same newspaper; some newspapers offer special rates for advertising in both the print and online editions. 

In the advertisement, be vague. (Let's be precise about what we mean by vague. Just refer to market research or product testing without providing precise details about what the person will be doing.) Only state the qualifications necessary and the approximate amount of money you are willing to pay for qualified participants. You can have interested candidates either communicate to a web site address or respond by phone.

Responding by phone is much more labor-intensive on your part, especially if you get many more interested people than you have slots for, but it is faster if you are working against a deadline. If you can, use a phone line with an explanatory message on voice mail or a standard answering machine. That way you can return calls at your leisure.

Including a link in the ad to a web site with an online questionnaire not only allows you greater control over the process but also controls the number of responses. Either way, people have to be quite motivated to take part because they have one extra step to go through to respond to the ad. Make it clear in your ad that people must qualify to participate. Of course, this assumes that your participants are computer and Internet users.

Employment Agencies

Employment agencies can be a good source of some types of participants.

There are temporary and employment agencies for almost every position in the work force, from file clerk to programmers to system analysts.

If you would like to conduct international usability tests but are unable to travel to other countries, employment agencies might be able to assist you in finding people who have been in your country for a very short tune and have not completely assimilated yet. While having them in your study is not the same as testing in their home country, you can get some useful data by using foreign nationals from the target country. (For a d.scuss.on abou whether geography matters, see the section "Testing in Multiple Geograph.c

Locations" in Chapter 6.)

Pricing varies by location in the country and of course, by the s i the person you require. Expect to Pay for end users at an hourly "'^f includes the agency fee and the person minimum of time, usually 4 hours. Once you establish a relationship agency and use them on an ongoing basis, you can negotiate both the hourly

rate and the minimum time required for people.

Because agencies are used to placing people in jobs, some training may be necessary to help the person or agency you are working with to understand what you need. Be clear about what you are doing and what your needs are. 

Do not expect the agency to understand the discipline of usability testing. Having explained usability testing to a number of different agencies, we have had mixed results. We try to keep it very simple. Simplicity and brevity are also important, so that the agency cannot inadvertently describe the testing procedure to potential candidates. In short, tell agencies only as much as they need to know to acquire the right participants and no more.

Ln addition to the screening questionnaire that you give the agencies to use, give them a written description of the exact way in which you would like your study described to potential candidates. Ask them to read your description verbatim. Keep it simple, keep it short, and, if necessary, keep it vague, such as, "We're conducting research." Personally, we do not even like to use terms such as "We'll be looking at ease of use." We prefer to fill in the participant about the exact nature of the research once he or she arrives.

One last point. It may be necessary (and advantageous) for you to go through vour own human resources department in working with temporary agencies. Some companies prohibit their employees from contacting agencies directly. If that is the case, spend some time to educate the HR person who will work the agencies on all the potential pitfalls mentioned previously.

Market Research Firms or Recruiting Specialists - II

There is much less of a concern of a professional firm biasing the participant and saying the wrong thing than there is with other sources. However, it is critical that you provide a clear, concise screening questionnaire because the marketing firm still will not understand your product or possibly much about the person that you are looking for. So, be precise, clear in your communication, and specific in your screening questions.
Market research firms can also supply the facility for conducting the usability test. They typically are set up with focus group observation rooms complete with microphones, digital video/audio output, video conferencing, and one-way mirrors. This is definitely the way to go when it is important that you
conduct an anonymous study of your product (e.g., the company who made the product is not mentioned) and cannot have participants come to your own location.
Expect to pay a premium for acquiring participants from these firms, especially if you need people quickly. They may ask anywhere from 5125 to $300 and up per person, depending on the difficulty of obtaining the
person and the amount of contact required. This does not include the actual compensation that you pay to the participant. However, once again, if you use the firm on a regular basis, you can negotiate the price down considerably. Typically, the firm will want to over recruit by 20 percent in order to ensure that you get the right number of people. If you say you need 10 end users, the firm typically will want to line up 12 people and have 2 of them act as substitutes if another person cancels or is a "no-show." (The lingo is "recruit
12 for 10.")

Market Research Firms or Recruiting Specialists - I

Market research firms recruit people for their client's marketing studies. A few firms specialize in recruiting participants for usability studies as well. Consequently, they will know exactly what you are talking about when you tell them that you need particular end users for a research study. Some market research firms may not understand usability testing per se, but bringing in computer users, scientists, doctors, lawyers, chemists, general consumers, and so on, for the purpose of gathering opinions and preferences is their business.
The larger ones can get any type of person you ask for, for a price.
Almost all major cities now have at least one market research firm. Because they network with each other in order to conduct studies around the country and internationally, once you find a good one, you can usually find another agency in a different city if you need to. With so much communication being done electronically, a recruiting specialist based in one city can easily recruit participants in other places. These firms therefore are the vehicle to use when it is important that you conduct usability testing in different geographical
locations. They can also help with some of the other types of research tasks, such as surveys and focus groups.

College Campuses - II

■■ Adv ertising in the college newspaper
■■ Word of mouth among college acquaintances
■ Contacting the head of the psychology department (psychology departments are constantly conducting experiments using the student population as participants.)
You may need permission to post your needs on bulletin boards (analog or digital), and you may have to pay a fee to post to a web site or event listing. Because it is so easy to acquire college students and because they love the extra cash, you run the risk of overusing this population. As with any other group, college students have their own specific characteristics, which may bias your study if you use them exclusively. For example, college students are probably more willing to try new things than the general population, and they are less apt to purchase products in their more precarious financial state. Therefore, use them judiciously and not as your sole source of participants, unless they are the sole targeted user.
if you specifically want college students, you may have to test on or near campus if there is a great distance between campus and your workplace because many college students may not have their own cars.

College Campuses - I

College students are always looking for extra cash and are delighted to help out by participating in usability tests. If college students are part of your user profile, by all means set up relationships with local colleges. (College campuses can also be a rich source of international participants.) One way to acquire students at colleges is to work directly with the college's outreach department, which seeks out relationships with local businesses. In their desire to establish these relationships, these outreach departments will often bend over backward to help you acquire what you need. In the case where there is no department providing that service, you will need to find a way to contact the students directly. Several ways are through:
■ Job boards or web sites
■ Events listings
■ Bulletin boards

Craigslist

A web site called Craigslist (www.craigslist.org) is an online classifieds listing that includes posts for everything that newspaper classifieds used to contain and more. The site has listings in every state in the United State s, every province in Canada, a couple dozen international cities, and 50 or 60 countries
outside of North America.
Putting a listing to solicit participants in the ETC section of the Jobs part of the site for the target location can be extremely fruitful; for example, see http:/ sfbay.craigslist.org/etc. (Craigslist charges a small fee for job-
related postings in some cities.) Dana has received hundreds of email responses within hours of a posting going up on the site.
This has both great advantages and disadvantages. We recommend putting enough qualifying questions in the posting to prevent people who aren't qualified from responding, or so you can easily see who does and doesn't qualify as you make your first pass through the responses. After that, you must be ruthless in weeding people out. (For example, Dana has eliminated people from the pool if they don't answer all of the qualifying questions or if they give vague answers to questions about occupation such as "consultant"
or "engineer." She also eliminates people if their response was inaccurate or incorrect in referring to the activity for which they were signing up, such as calling a usability study a focus group.) The quickest way to start going through the responses is to set up filters in your email inbox to sort them for you automatically.

Referrals from Personal Networks, Coworkers, and Other Participants - II

Unless you must keep it secret that the test is happening at all, we have found that putting the word out among our professional and personal networks will often turn up a source of participants or individuals who fit the user profile.
You may want to use people in your test who have taken part in other activities for the company. If this is not your first test, you may want to invite participants from the earlier tests back or contact them to get referrals to other people they know who might be suitable for your next test.
You might even have a form that your participants complete at the end of each test where you ask them if they'd be willing to participate in future tests. On the same form, you can ask for referrals (phone numbers and/or email addresses) of friends and others who might be willing to participate in a similar study.

Referrals from Personal Networks, Coworkers, and Other Participants - I

For some user groups, the best way to find participants is through personal contacts. This is especially true for finding participants on either end of the age scale. For example, parents are usually reluctant to put their children in unknown situations with strangers. Likewise, many older adults are less likely to be found in online sources and are skeptical about strangers phoning them to take part in user research. Some older adults can feel quite vulnerable and will be very leery of sales scams. So, if you want to have consumers in your
test who are under age 17 or over age 65, the best way is to ask your friends, family, and coworkers to put the word out and to make introductions so you may then contact these prospective participants without scaring them off.

Societies and Associations

Professional societies and associations can be a good source of participants who do particular jobs or are in specific professions. If you or someone in your company is a member of an appropriate association, you may be able to put out a casual call for participants through listservs or email distribution lists
The groups may also have forums, blogs, or other technology-mediated social networking available that you could put out a discrete message or posting on. This may not be appropriate in some groups, though, so do check on the etiquette for making your approach. You may instead have to purchase a list from the association. You can usually do this by contacting its main office, paying a fee, and being sent a list electronically. The association will probably put restrictions on what you can use the list for.
If you want to include people with disabilities in your test, contacting organizations such as the American Foundation for the Blind, Lighthouse International, Disabled American Veterans, and National Council for Support of Disability Issues, or the American Disability Association can get you started.
For more on testing for accessibility, see Chapter 13

Societies and Associations

Professional societies and associations can be a good source of participants who do particular jobs or are in specific professions. If you or someone in your company is a member of an appropriate association, you may be able to put out a casual call for participants through listservs or email distribution lists
The groups may also have forums, blogs, or other technology-mediated social networking available that you could put out a discrete message or posting on. This may not be appropriate in some groups, though, so do check on the etiquette for making your approach. You may instead have to purchase a list from the association. You can usually do this by contacting its main office, paying a fee, and being sent a list electronically. The association will probably put restrictions on what you can use the list for.
If you want to include people with disabilities in your test, contacting organizations such as the American Foundation for the Blind, Lighthouse International, Disabled American Veterans, and National Council for Support of Disability Issues, or the American Disability Association can get you started.
For more on testing for accessibility, see Chapter 13

User Croups or Clubs, Churches, or Other Community Groups

Many companies sponsor product user groups who meet at conventions annually and hold other events locally or online throughout the year. Your company may do this for its products or services. There are also many enthusiasts' clubs that might be tapped as sources for participants for usability tests. (Be mindful of particular biases here because participants will have an avid interest in giving you feedback about something they have joined a group to engage in more deeply.)
You might also check for web sites devoted to particular interests, disciplines, or products to find sources for participants. There you may be able to contact members directly or post a call for participants on forums or message boards, for example, if the rules allow you to do so.
Typically, these groups are composed of people who range anywhere from novice to the most expert users. What makes them an especially valuable resource is the fact that you may be able to contact everyone in the group at once by contacting the head of the group directly or the planner of the group's meetings. You can then ascertain when the next meeting is and prepare a solicitation to be distributed at that meeting. Have all the people who are

Existing Customers through Sales Representatives

In larger companies the sales team may have access to the customers you would like to use for testing. However, they may not be particularly keen to give you direct access to their customers. And, because the sales reps already have relationships with customers around the country, they may have a different agenda than you do. In their zest to appear "in the know" to their customers about future products, they may say too much about the test. Or, they may simply be unaware of the usability testing process and the need for
"average" as well as excellent performers.
Be specific about the types of people you need, as well as the importance of being vague about the subject matter of the test. In one test that Jeff helped design that used existing customers and for which the sales reps served as the recruiters, one participant had obviously been primed by a sales rep. For his "test" of a new release of an existing product, he had diligently studied the entire 300-page user guide of the current product the night be/ore. It seems that he took the notion of a usability "test" a mite too far. The key to counteracting
these tendencies is to communicate clearly to your recruiters (the sales reps). That way you can nip these problems in the bud.

Existing Customers from In-House Lists

An excellent source of participant candidates is your own company's list of existing customers. If there is no formal list, explore other sources such as warranty card returns, various promotions to which customers have responded, or mailing lists. When contacting such customers, it is imperative, given that you are representing your company, that you make clear that there are no strings attached and there is no hidden agenda. That is, their participation is not a marketing strategy intended to eke out more sales but simply a means
to gather information about future products and improve them accordmgly You should say this in writing in the recruiting email, ment.oning that this research is not lied in any urn/ to a sales promotion.
Customer lists are a'great source of participants who are expenenced. you are running a study with both experienced end users and thosewhohave never used your product, then use the customer lists as a means of gathenng the experienced people.
NOTE  one note about using customer lists: Check with whoever owns thecustomer list to determine whether it is acceptable company pol.cy-to use customer list this way. Some companies decline to a.low usab, ty orr.her  researchers inside the company to use customer lists as a way to ontac customer-participants because the customers have not specially agree

Web Site Sign-Up

If you are planning to do extensive testing, you will probably want to develop a database of qualified candidates for your product so that you can move quickly when it is time for a test.
Setting up a web form and a database to gather and store information about potential participants may take some nontrivial setup time, but once it is set up, you can sit back and watch the panel of possible participants grow.
Many large companies include links on their web sites that invite people to "opt-in" to their participant databases. The forms collect basic demographic and behavioral data generally related to the company's product line in the online form. By doing this sort of thing, when a study comes up, you can do a simple search of your database for key characteristics to narrow the list of likely candidates, whom you can then screen more specifically for the test you're about to do.
There are several advantages to this approach. One is that the people who sign up at your web site are definitely motivated to take part in improving the products they use. They probably have some idea about what you might be doing, that is, they are probably familiar with marketing focus groups, at least.
Because they have joined your list of their own free will, you are free to contact them about taking part in your research. One caution is that your sample may be skewed toward people who are particular fans of your company, or who habitually sign up for surveys, focus groups, and other types of paid research.
Update the database after each test, showing the date of the last test. The advantage of this approach is that you are always ready with a list of potential participants who represent the market breakdown for your product on short notice.
NOTE
Itl'lM Please note that depending on your needs and how frequently you conduct tests, it may require a full-time administrator to manage the database and manage the participant contact prior to the test

Qualified Friends and Family

All of us know people who may be qualified to participate in a study. This is especially true of products that are used by the population at large and for whom the characteristic audience is very broad-based. You might have friends or family who share all the characteristics of your internal end users with the exception that they do not work for your company, which is a real advantage.
If you do end up using friends and family in your study, make sure that your relationship does not affect your professionalism. When testing, do everything exactly as you normally would, including reading the orientation script, being sure not to talk about the study until after it is completed, and not being overly chummy during the test.

Internal Participants - II

■ To conduct early exploratory research. Internal participants make good exploratory participants as long as they are unfamiliar with the product and meet the target user profile. With early exploratory research,

you are not as concerned with the subtle issues of later validation tests.

Instead, you are more concerned with overarching problems of the product, such as the mental model, primary navigation techniques, organization of documentation, organization and navigation of an online

help system, and so on. Internal participants can quickly help to determine whether you are on the right track, without the expense and time necessary to recruit outside participants.

- To conduct "best case" testing. In "best case" testing, which is often employed at an early stage of development, you see how someone who is very experienced and more familiar with the culture (i.e., a ringer) uses the product. The idea is that if this best case user has trouble, you have serious design problems. In some cases, the internal participant will be more critical than the external participant, more willing to point out problems and not hold back. If the product passes the best case scenario, you should quickly follow with a less experienced end user to verify the soundness of the design. If it does not "pass" the best case

scenario, do not bother with further testing. Instead, head back to the drawing board for further development work before testing again. Of course, in some cases the internal participant is the end user. By all means,

in that case, use them exclusively for your test. For example, in the banking  system example, the banking system may have been developed internally for internal use. The end users are certainly internal to the company.

Finding internal participants may be as straightforward as searching your company directory on the intranet and firing off an email to someone who seems to be in an appropriate position to see if he or she indeed meets the selection criteria. Otherwise, your company's human resources department may be able to direct you to qualified end users in your own organization. For example, if you are working on a product that requires a certain expertise and you cannot use people who are working on the project directly, the human
resources (HR) department or the company intranet can direct you to people within the company who would tend to have the appropriate experience.
You then could call them up yourself, do the screening, and make all the arrangements for using them as participants.
By the way, if you decide to use outside agencies to recruit test participants, the human resources department may want to coordinate that. HR may have already set up relationships with these companies through other people doing research studies in your company of whom you may not be aware.

Find Sources of Participants - II

Find Sources of Participants - I

Okay. So far so good. You have analyzed the user profile for your product, received confirmation that indeed you understand who the end user is, and have developed a list or matrix of the people within your user profile who you would like to test with a description of their characteristics. The next step is to
recruit the people who fit this description to participate in your test.
Now let's discuss some of the sources from which you can acquire participants. The following sections discuss the most common ones. Some sources require creativity and tenacity to exploit to the best advantage. Some sources generate large numbers of responses but don't always net quality responses. We have listed our ideas for sources in order of how much work or expense each takes for the test moderator or recruiter, from least work and expense to most expense and time.

Consider Creating an "Answer Sheet"

If your questionnaire format does not make answers obvious, develop an "answer sheet" separately or right on the questionnaire that provides the replies that qualify a participant and how many of each category you need.
This can be as simple as a table or spreadsheet. We like a spreadsheet such as the one in Figure 7-5 better because you can later filter and sort the data you have gathered. (For an expanded sample of such a spreadsheet, go to the web site that accompanies this book: ww.wiley.com/usabilicytescing.com.)

Consider Creating an "Answer Sheet"

If your questionnaire format does not make answers obvious, develop an "answer sheet" separately or right on the questionnaire that provides the replies that qualify a participant and how many of each category you need.
This can be as simple as a table or spreadsheet. We like a spreadsheet such as the one in Figure 7-5 better because you can later filter and sort the data you have gathered. (For an expanded sample of such a spreadsheet, go to the web site that accompanies this book: ww.wiley.com/usabilicytescing.com.)

Consider Creating an "Answer Sheet"

If your questionnaire format does not make answers obvious, develop an "answer sheet" separately or right on the questionnaire that provides the replies that qualify a participant and how many of each category you need.
This can be as simple as a table or spreadsheet. We like a spreadsheet such as the one in Figure 7-5 better because you can later filter and sort the data you have gathered. (For an expanded sample of such a spreadsheet, go to the web site that accompanies this book: ww.wiley.com/usabilicytescing.com.)

Test the Questionnaire on Colleagues and Revise It

(Yes, you are right. This is a usability test of the screener!) If it is a phone questionnaire, use the phone. You would be surprised, but it makes a difference. If it is an email or web-based questionnaire, test it that way. Observe as a colleague tries it out by answering the questions to see what types of answers you get. If they are not what you are expecting, you must revise the questions, so it is clear that future respondents will understand what you want to find out and so you will be able to determine easily whether they qualify for the
study.
Regardless of whom you use to help you acquire participants, control the process as much as possible by being as specific as possible about your needs, anticipating miscommunications, and writing down the exact words you want the procurers to use. It is your study, they are your results, and the whole affair is seriously compromised when the wrong people show up.

Develop a Format for Easy Flow through the Questionnaire - II

Develop a Format for Easy Flow through the Questionnaire - I

Allow the caller to branch from one question to the next without thinking too much about the answers. This is especially important if you use an outside agency and the callers do not have an in-depth understanding of your product. Figure 7-4 shows an example of a screening questionnaire that was developed for the usability test of an accounting package for small businesses.
This questionnaire could be administered by phone or with slight adjustments by email. Note how the questions are intended to screen for a user who does his or her own bookkeeping on a PC, and who has experience using other office-related software. In administering a questionnaire like the sample one, the caller should not offer the choices listed below the questions, but simply mark them based on the respondent's answers. Likewise, if the questionnaire is to be administered by email, only the questions should appear without the possible answers. Note also that the questions focus on behaviors, not demographics. Figure 7-5 shows data collected by using the sample screener. (For an expanded sample of questionnaires and spreadsheets for recruiting and screening, go to the web site that accompanies this book: www. wiley. com/usability testing. com.)

Organize the Questions in a Specific Order

It makes sense to place those questions that are most likely to disqualify a person first on a questionnaire, while being mindful of basic telephone etiquette. For example, if you are testing a graphic design product designed for seasoned end users with knowledge of color theory, you would not want to ask 20 very general questions only to find out in question 21 that the person you are speaking with does not know saturation from hue. You should establish that background early in the call to limit the phone time. 

On the other 

hand, that is not the type of high-level question you can ask first either. You need to work your way into it.

For the hotel reservations web site, you might want to eliminate candidates who don't book their own hotel rooms at all, so you would ask a question such as: "For your last three (or five; pick a number) trips, who made the arrangements?"

Formulate Screening Questions - II

You might ask respondents for the test of the hotel reservations web site to list all of the rewards programs they belong to: "Tell me about rewards memberships. Is that something you do? What memberships do you have?"
If people have trouble remembering, you might suggest they take out their wallet to flip through membership cards.
The case of the bank employees is slightly different. You can find out how long workers have been in their jobs from the human resources department or from the group supervisor. If you want to include workers who have mastered particular tasks or operations procedures, you may also be able to find that out from a supervisor. To be sure, you could ask candidates to tell you about their jobs, what their specialties are, what they like and don't like doing and why (you may want people who don't like doing the task you will be testing, to see if the new system improves their attitudes about it), and when the last time was that someone asked them for help on the task or procedure (perhaps an indication of expertise). Or, you could develop an assessment test.

Formulate Screening Questions - I

Once you have isolated the different factors for which you are screening you need to formulate questions to ascertain whether a person has that expertise and other characteristics defined for the test. Use the same approach you would use for creating any sort of questionnaire, especially if you will be handing your screener over to someone else to do the recruiting and scheduling We present a number of question formats in Chapter 8. Besides those question formats, you will have to operationalize a definition of expertise Determine

levels of expertise by creating a way of checking proficiency (what we call a criterion test) such as an actual quiz or a matrix similar to the one we show in Figure 7-1. Because people are often reluctant to divulge exact personal details if you must gather personal information, such as age or salary, frame your questions in ranges. For example, instead of asking, "What is your annual household income?" ask instead:

"Into what category does your annual household income fall?"

$20,000- $45,000 $46,000-$55,000
$56,000-$75,000 $76,0000-$100,000

Identify Specific Selection Criteria

Focus on and pull out those characteristics that are unique to your product and not just general categories of the population. For example, experience using a cell phone and a camera may qualify as a requirement for a product that combines the two. But, because the product will be used by all age groups, you need not screen for a particular age group to include in your test sample.
Later, as you are acquiring participants, you can simply make sure that you get a mix of different ages.
To follow the example for the hotel reservations web site, your objective is to learn what differences there are in how different types of travelers make hotel reservations. The team decides it wants to see whether there are differences between rewards members and travelers who are not members of the rewards
program. To select participants, then, some criteria are:
■ Are they members of our hotel rewards program? We want some of each, split into equal sized groups, if possible.
■ Are they members of nny rewards program? For participants who are not members of our program, do they ever belong to rewards programs? For this test, we want one group of people who do not join rewards
programs.
Can you see where you might end up with questions to ask potential participants from this?

For the banking example, let's focus on just the back office folks for now. The objective of the test is to learn whether users who are new to the back office can use the new system as efficiently and effectively as those who have extensive knowledge of the particular operations of this back office. Is it possible that the new system could cut down on training time? To select participants to help you find this out, you will need workers who have been in their jobs for a long time (whatever you and your team determine is an appropriate amount of time), but also some who have been through little or no training. So, some selection criteria might be:

Review the Profile to Understand Users' Backgrounds

Doing usability testing is just one of the reasons you develop the user profile. However, the user profile should prove a useful tool for visualizing the person you want to be in the testing room so you can articulate the requirements and classifiers and use those to form screening questions. If you don't have a user profile that documents the characteristics of the target users you want to have in the usability test, go back to the beginning of this chapter to "Characterize Users."

Write the Screening Questionnaire

The screening questionnaire is the means for qualifying and selecting participants to participate in the test. Its content is typically obtained from the characteristics, requirements, and classifiers that you included in the user profile and test plan. The questionnaire is usually presented over the phone, although you can also give it to potential participants by email or in a web-based questionnaire. If you will be using a colleague, an outside consultant, or an agency to help you recruit participants, the screening questionnaire is your primary way of describing the people you need in an unambiguous format. While some agencies will develop their own screeners based on information you provide, you must revieiv their questionnaires before they begin using
them, to ensure that your instructions have been understood
A screening questionnaire can be extremely simple or quite involved, depending on the variability and background of the potential participants of the product Either way, developing the screening questionnaire exposes
- Review the profile to understand users' backgrounds.
■ Identify specific selection criteria.
■ Formulate screening questions.
■ Organize the questions in a specific order.
- Develop a format for easy flow through the questionnaire.
■ Test the questionnaire and revise it.
■ Consider creating an "answer sheet."

Determine the Number of Participants to Test

The number of participants you decide to test depends on many factors, including:

■ The degree of confidence in the results that you require

■ The number of available resources to set up and conduct the test

■ The availability of the type of participants you require

■ The duration of the test session

Ultimately, you have to balance your need for acquiring participants with these (and probably other) practical constraints of time and resources. If you require statistically valid results, you will need to test enough participants to conduct the appropriate analyses and generalize to your specific target population, as well as to rigorously control for potentially biasing conditions and factors.

If, however, you are simply attempting to expose as many usability problems as possible in the shortest amount of time, then test at least four to five participants of each user or audience cell. Research indicates that testing four to five participants of each type or cell will expose the vast majority of usability problems. However, we add that we are a bit uncomfortable testing only four participants, and we try to test at least eight participants if possible. While you may expose most of the usability problems with four participants, there is still a good chance you may overlook a problem that could have severe ramifications.

An important consideration is whether you will conduct more than one test during the product development life cycle. 1/ you are going to conduct multiple tests, then you can feel more confident testing fewer participants in each test. For example, if you conduct three tests with five participants each, you end up testing fifteen people. If you will be conducting a single test, however, then you will be better served by having more participants for that lone test.

Consider a Matrix Test Design

Because you want to see a range of behavior and performance within your small group of participants consider a matrix design for the test that balance different variables or classifiers in such a way that no particular user group or cell of the matrix is left out. 6 p

Take the banking example. These days you can assume that having some computer experience is a requirement for getting a job either at the teller vvicket or in the back office, so that's probably not a useful way of classifying these participants. What could determine levels of expertise in the two roles that would be important to your test? Possibly a combination of time on the job and types of responsibility may describe an "expert" teller from a "novice" one or an "expert" back office clerk from a "novice" one. You could look at the factors you use to determine expertise as domain knowledge (for example, what the participant knows about being a teller) or tools knowledge (for example, the efficiency and effectiveness with which the participant uses the software or other technology required in her job). You'll want some of each and you'll

have to create some type of operationalized definition for determining who falls into which cell.
Good test design mea ns that you balance the mixtures of user groups in such a way that the major categories or cells are all represented. For the banking example, you might test 16 participants whose job and experience level look like the ones in the table below (assuming that you have clearly defined the criteria for "novice" and "expert"; see "Define Expertise," above).

Divide the User Profile into Distinct Categories

The user profile describes the range of skills, behaviors, and experience that make up the entire universe of target end users. Every end user should fall somewhere within that spectrum However, you are not interested in individuals alone. Instead, you should identify entire groups of end users who share many of the same characteristics on the user profile. These groups, sometimes identified by similar occupations or job titles but often as roles (for example, "frequent traveler/' "club member," "voter," or even "patient," "parent," "vendor," or "client"), will use the product and its support materials in different ways and for different purposes. This is also where your classifiers come in. Use those to help you group users into categories. It is important that you understand the user profile for each target audi- ence. For example, the hotel reservations web site previously mentioned is intended for both rewards members and travelers who are not rewards mem-

bers (our classifiers); each group may use the site differently. If you were testing that web site, you would want to select some members and some non-members. The proportion depends on the priorities of the business and the objectives and goals for the study. You don't have to try to match the proportions in the target market segments.

Or, suppose you were responsible for testing a banking system that both tellers and "back office" staff members will use. You will want to have some tellers who will use the system for performing daily customer transactions. 

You will also want to have some "back office" staff members who will use the system to generate reports and conduct different analyses. The tasks of the two groups are different.

Each group will have similar characteristics when compared to other members of their group, although with varying levels of expertise. With each group, make sure that you are representing these varying expertise levels, if any. For example, test some novice tellers, some experienced tellers, some novice "back

office" staff, and some expert "back office" staff. Suggestions for how many of each type to include are discussed later in this chapter.

In the hotel reservations web site example, rewards members are likely to be very experienced with the site and non-members are likely to have less experience with the site. But you want both groups of users to cover a range of levels of previous experience with the site. Or you may assume that participants'

overall travel booking experience is more important than their direct experience with your site. In that case, you might want to classify the participants based on the frequency with which they book accommodations in the course of a year, or whether most of their travel is leisure or business-related, again depending on the objective and goals of the test. To represent these different skill levels, you may want to use a matrix test design, discussed in the next section.

Document the User Profile - III

Document the User Profile - II