Screen and Select Participants

As we have mentioned a couple of times, you'll need to put together some type of process to determine whether the candidates you have qualify for the study. This usually takes the form of a document that contains a table of qualifications (see the sections "Define the Criteria for Each User Group," and "Choose the Number of Participants to Test," earlier in this chapter) and a set of multiple choice questions (see the previous section "Write the Screening Questionnaire"). The table lists requirements and classifiers along with a range of the number of participants you would like to possess these attributes. The multiple choice questions operationalize the selection/elimination process. The document, typically called a screcner, includes scripted language for starting a phone interview process, getting through the questions, scheduling

the participant (if qualified), and closing the interview.

Newspaper Advertisements

If you have plenty of lead time and want to specifically recruit people who are less technology savvy, advertise in your local newspaper either in a display ad area or the classified ad section for "participants needed for research study."

(Response rates are likely to be higher for display ads than for classifieds.) You might also try online versions of the same newspaper; some newspapers offer special rates for advertising in both the print and online editions. 

In the advertisement, be vague. (Let's be precise about what we mean by vague. Just refer to market research or product testing without providing precise details about what the person will be doing.) Only state the qualifications necessary and the approximate amount of money you are willing to pay for qualified participants. You can have interested candidates either communicate to a web site address or respond by phone.

Responding by phone is much more labor-intensive on your part, especially if you get many more interested people than you have slots for, but it is faster if you are working against a deadline. If you can, use a phone line with an explanatory message on voice mail or a standard answering machine. That way you can return calls at your leisure.

Including a link in the ad to a web site with an online questionnaire not only allows you greater control over the process but also controls the number of responses. Either way, people have to be quite motivated to take part because they have one extra step to go through to respond to the ad. Make it clear in your ad that people must qualify to participate. Of course, this assumes that your participants are computer and Internet users.

Employment Agencies

Employment agencies can be a good source of some types of participants.

There are temporary and employment agencies for almost every position in the work force, from file clerk to programmers to system analysts.

If you would like to conduct international usability tests but are unable to travel to other countries, employment agencies might be able to assist you in finding people who have been in your country for a very short tune and have not completely assimilated yet. While having them in your study is not the same as testing in their home country, you can get some useful data by using foreign nationals from the target country. (For a d.scuss.on abou whether geography matters, see the section "Testing in Multiple Geograph.c

Locations" in Chapter 6.)

Pricing varies by location in the country and of course, by the s i the person you require. Expect to Pay for end users at an hourly "'^f includes the agency fee and the person minimum of time, usually 4 hours. Once you establish a relationship agency and use them on an ongoing basis, you can negotiate both the hourly

rate and the minimum time required for people.

Because agencies are used to placing people in jobs, some training may be necessary to help the person or agency you are working with to understand what you need. Be clear about what you are doing and what your needs are. 

Do not expect the agency to understand the discipline of usability testing. Having explained usability testing to a number of different agencies, we have had mixed results. We try to keep it very simple. Simplicity and brevity are also important, so that the agency cannot inadvertently describe the testing procedure to potential candidates. In short, tell agencies only as much as they need to know to acquire the right participants and no more.

Ln addition to the screening questionnaire that you give the agencies to use, give them a written description of the exact way in which you would like your study described to potential candidates. Ask them to read your description verbatim. Keep it simple, keep it short, and, if necessary, keep it vague, such as, "We're conducting research." Personally, we do not even like to use terms such as "We'll be looking at ease of use." We prefer to fill in the participant about the exact nature of the research once he or she arrives.

One last point. It may be necessary (and advantageous) for you to go through vour own human resources department in working with temporary agencies. Some companies prohibit their employees from contacting agencies directly. If that is the case, spend some time to educate the HR person who will work the agencies on all the potential pitfalls mentioned previously.

Market Research Firms or Recruiting Specialists - II

There is much less of a concern of a professional firm biasing the participant and saying the wrong thing than there is with other sources. However, it is critical that you provide a clear, concise screening questionnaire because the marketing firm still will not understand your product or possibly much about the person that you are looking for. So, be precise, clear in your communication, and specific in your screening questions.
Market research firms can also supply the facility for conducting the usability test. They typically are set up with focus group observation rooms complete with microphones, digital video/audio output, video conferencing, and one-way mirrors. This is definitely the way to go when it is important that you
conduct an anonymous study of your product (e.g., the company who made the product is not mentioned) and cannot have participants come to your own location.
Expect to pay a premium for acquiring participants from these firms, especially if you need people quickly. They may ask anywhere from 5125 to $300 and up per person, depending on the difficulty of obtaining the
person and the amount of contact required. This does not include the actual compensation that you pay to the participant. However, once again, if you use the firm on a regular basis, you can negotiate the price down considerably. Typically, the firm will want to over recruit by 20 percent in order to ensure that you get the right number of people. If you say you need 10 end users, the firm typically will want to line up 12 people and have 2 of them act as substitutes if another person cancels or is a "no-show." (The lingo is "recruit
12 for 10.")

Market Research Firms or Recruiting Specialists - I

Market research firms recruit people for their client's marketing studies. A few firms specialize in recruiting participants for usability studies as well. Consequently, they will know exactly what you are talking about when you tell them that you need particular end users for a research study. Some market research firms may not understand usability testing per se, but bringing in computer users, scientists, doctors, lawyers, chemists, general consumers, and so on, for the purpose of gathering opinions and preferences is their business.
The larger ones can get any type of person you ask for, for a price.
Almost all major cities now have at least one market research firm. Because they network with each other in order to conduct studies around the country and internationally, once you find a good one, you can usually find another agency in a different city if you need to. With so much communication being done electronically, a recruiting specialist based in one city can easily recruit participants in other places. These firms therefore are the vehicle to use when it is important that you conduct usability testing in different geographical
locations. They can also help with some of the other types of research tasks, such as surveys and focus groups.

College Campuses - II

■■ Adv ertising in the college newspaper
■■ Word of mouth among college acquaintances
■ Contacting the head of the psychology department (psychology departments are constantly conducting experiments using the student population as participants.)
You may need permission to post your needs on bulletin boards (analog or digital), and you may have to pay a fee to post to a web site or event listing. Because it is so easy to acquire college students and because they love the extra cash, you run the risk of overusing this population. As with any other group, college students have their own specific characteristics, which may bias your study if you use them exclusively. For example, college students are probably more willing to try new things than the general population, and they are less apt to purchase products in their more precarious financial state. Therefore, use them judiciously and not as your sole source of participants, unless they are the sole targeted user.
if you specifically want college students, you may have to test on or near campus if there is a great distance between campus and your workplace because many college students may not have their own cars.

College Campuses - I

College students are always looking for extra cash and are delighted to help out by participating in usability tests. If college students are part of your user profile, by all means set up relationships with local colleges. (College campuses can also be a rich source of international participants.) One way to acquire students at colleges is to work directly with the college's outreach department, which seeks out relationships with local businesses. In their desire to establish these relationships, these outreach departments will often bend over backward to help you acquire what you need. In the case where there is no department providing that service, you will need to find a way to contact the students directly. Several ways are through:
■ Job boards or web sites
■ Events listings
■ Bulletin boards

Craigslist

A web site called Craigslist (www.craigslist.org) is an online classifieds listing that includes posts for everything that newspaper classifieds used to contain and more. The site has listings in every state in the United State s, every province in Canada, a couple dozen international cities, and 50 or 60 countries
outside of North America.
Putting a listing to solicit participants in the ETC section of the Jobs part of the site for the target location can be extremely fruitful; for example, see http:/ sfbay.craigslist.org/etc. (Craigslist charges a small fee for job-
related postings in some cities.) Dana has received hundreds of email responses within hours of a posting going up on the site.
This has both great advantages and disadvantages. We recommend putting enough qualifying questions in the posting to prevent people who aren't qualified from responding, or so you can easily see who does and doesn't qualify as you make your first pass through the responses. After that, you must be ruthless in weeding people out. (For example, Dana has eliminated people from the pool if they don't answer all of the qualifying questions or if they give vague answers to questions about occupation such as "consultant"
or "engineer." She also eliminates people if their response was inaccurate or incorrect in referring to the activity for which they were signing up, such as calling a usability study a focus group.) The quickest way to start going through the responses is to set up filters in your email inbox to sort them for you automatically.

Referrals from Personal Networks, Coworkers, and Other Participants - II

Unless you must keep it secret that the test is happening at all, we have found that putting the word out among our professional and personal networks will often turn up a source of participants or individuals who fit the user profile.
You may want to use people in your test who have taken part in other activities for the company. If this is not your first test, you may want to invite participants from the earlier tests back or contact them to get referrals to other people they know who might be suitable for your next test.
You might even have a form that your participants complete at the end of each test where you ask them if they'd be willing to participate in future tests. On the same form, you can ask for referrals (phone numbers and/or email addresses) of friends and others who might be willing to participate in a similar study.

Referrals from Personal Networks, Coworkers, and Other Participants - I

For some user groups, the best way to find participants is through personal contacts. This is especially true for finding participants on either end of the age scale. For example, parents are usually reluctant to put their children in unknown situations with strangers. Likewise, many older adults are less likely to be found in online sources and are skeptical about strangers phoning them to take part in user research. Some older adults can feel quite vulnerable and will be very leery of sales scams. So, if you want to have consumers in your
test who are under age 17 or over age 65, the best way is to ask your friends, family, and coworkers to put the word out and to make introductions so you may then contact these prospective participants without scaring them off.

Societies and Associations

Professional societies and associations can be a good source of participants who do particular jobs or are in specific professions. If you or someone in your company is a member of an appropriate association, you may be able to put out a casual call for participants through listservs or email distribution lists
The groups may also have forums, blogs, or other technology-mediated social networking available that you could put out a discrete message or posting on. This may not be appropriate in some groups, though, so do check on the etiquette for making your approach. You may instead have to purchase a list from the association. You can usually do this by contacting its main office, paying a fee, and being sent a list electronically. The association will probably put restrictions on what you can use the list for.
If you want to include people with disabilities in your test, contacting organizations such as the American Foundation for the Blind, Lighthouse International, Disabled American Veterans, and National Council for Support of Disability Issues, or the American Disability Association can get you started.
For more on testing for accessibility, see Chapter 13

Societies and Associations

Professional societies and associations can be a good source of participants who do particular jobs or are in specific professions. If you or someone in your company is a member of an appropriate association, you may be able to put out a casual call for participants through listservs or email distribution lists
The groups may also have forums, blogs, or other technology-mediated social networking available that you could put out a discrete message or posting on. This may not be appropriate in some groups, though, so do check on the etiquette for making your approach. You may instead have to purchase a list from the association. You can usually do this by contacting its main office, paying a fee, and being sent a list electronically. The association will probably put restrictions on what you can use the list for.
If you want to include people with disabilities in your test, contacting organizations such as the American Foundation for the Blind, Lighthouse International, Disabled American Veterans, and National Council for Support of Disability Issues, or the American Disability Association can get you started.
For more on testing for accessibility, see Chapter 13

User Croups or Clubs, Churches, or Other Community Groups

Many companies sponsor product user groups who meet at conventions annually and hold other events locally or online throughout the year. Your company may do this for its products or services. There are also many enthusiasts' clubs that might be tapped as sources for participants for usability tests. (Be mindful of particular biases here because participants will have an avid interest in giving you feedback about something they have joined a group to engage in more deeply.)
You might also check for web sites devoted to particular interests, disciplines, or products to find sources for participants. There you may be able to contact members directly or post a call for participants on forums or message boards, for example, if the rules allow you to do so.
Typically, these groups are composed of people who range anywhere from novice to the most expert users. What makes them an especially valuable resource is the fact that you may be able to contact everyone in the group at once by contacting the head of the group directly or the planner of the group's meetings. You can then ascertain when the next meeting is and prepare a solicitation to be distributed at that meeting. Have all the people who are

Existing Customers through Sales Representatives

In larger companies the sales team may have access to the customers you would like to use for testing. However, they may not be particularly keen to give you direct access to their customers. And, because the sales reps already have relationships with customers around the country, they may have a different agenda than you do. In their zest to appear "in the know" to their customers about future products, they may say too much about the test. Or, they may simply be unaware of the usability testing process and the need for
"average" as well as excellent performers.
Be specific about the types of people you need, as well as the importance of being vague about the subject matter of the test. In one test that Jeff helped design that used existing customers and for which the sales reps served as the recruiters, one participant had obviously been primed by a sales rep. For his "test" of a new release of an existing product, he had diligently studied the entire 300-page user guide of the current product the night be/ore. It seems that he took the notion of a usability "test" a mite too far. The key to counteracting
these tendencies is to communicate clearly to your recruiters (the sales reps). That way you can nip these problems in the bud.

Existing Customers from In-House Lists

An excellent source of participant candidates is your own company's list of existing customers. If there is no formal list, explore other sources such as warranty card returns, various promotions to which customers have responded, or mailing lists. When contacting such customers, it is imperative, given that you are representing your company, that you make clear that there are no strings attached and there is no hidden agenda. That is, their participation is not a marketing strategy intended to eke out more sales but simply a means
to gather information about future products and improve them accordmgly You should say this in writing in the recruiting email, ment.oning that this research is not lied in any urn/ to a sales promotion.
Customer lists are a'great source of participants who are expenenced. you are running a study with both experienced end users and thosewhohave never used your product, then use the customer lists as a means of gathenng the experienced people.
NOTE  one note about using customer lists: Check with whoever owns thecustomer list to determine whether it is acceptable company pol.cy-to use customer list this way. Some companies decline to a.low usab, ty orr.her  researchers inside the company to use customer lists as a way to ontac customer-participants because the customers have not specially agree

Web Site Sign-Up

If you are planning to do extensive testing, you will probably want to develop a database of qualified candidates for your product so that you can move quickly when it is time for a test.
Setting up a web form and a database to gather and store information about potential participants may take some nontrivial setup time, but once it is set up, you can sit back and watch the panel of possible participants grow.
Many large companies include links on their web sites that invite people to "opt-in" to their participant databases. The forms collect basic demographic and behavioral data generally related to the company's product line in the online form. By doing this sort of thing, when a study comes up, you can do a simple search of your database for key characteristics to narrow the list of likely candidates, whom you can then screen more specifically for the test you're about to do.
There are several advantages to this approach. One is that the people who sign up at your web site are definitely motivated to take part in improving the products they use. They probably have some idea about what you might be doing, that is, they are probably familiar with marketing focus groups, at least.
Because they have joined your list of their own free will, you are free to contact them about taking part in your research. One caution is that your sample may be skewed toward people who are particular fans of your company, or who habitually sign up for surveys, focus groups, and other types of paid research.
Update the database after each test, showing the date of the last test. The advantage of this approach is that you are always ready with a list of potential participants who represent the market breakdown for your product on short notice.
NOTE
Itl'lM Please note that depending on your needs and how frequently you conduct tests, it may require a full-time administrator to manage the database and manage the participant contact prior to the test

Qualified Friends and Family

All of us know people who may be qualified to participate in a study. This is especially true of products that are used by the population at large and for whom the characteristic audience is very broad-based. You might have friends or family who share all the characteristics of your internal end users with the exception that they do not work for your company, which is a real advantage.
If you do end up using friends and family in your study, make sure that your relationship does not affect your professionalism. When testing, do everything exactly as you normally would, including reading the orientation script, being sure not to talk about the study until after it is completed, and not being overly chummy during the test.

Internal Participants - II

■ To conduct early exploratory research. Internal participants make good exploratory participants as long as they are unfamiliar with the product and meet the target user profile. With early exploratory research,

you are not as concerned with the subtle issues of later validation tests.

Instead, you are more concerned with overarching problems of the product, such as the mental model, primary navigation techniques, organization of documentation, organization and navigation of an online

help system, and so on. Internal participants can quickly help to determine whether you are on the right track, without the expense and time necessary to recruit outside participants.

- To conduct "best case" testing. In "best case" testing, which is often employed at an early stage of development, you see how someone who is very experienced and more familiar with the culture (i.e., a ringer) uses the product. The idea is that if this best case user has trouble, you have serious design problems. In some cases, the internal participant will be more critical than the external participant, more willing to point out problems and not hold back. If the product passes the best case scenario, you should quickly follow with a less experienced end user to verify the soundness of the design. If it does not "pass" the best case

scenario, do not bother with further testing. Instead, head back to the drawing board for further development work before testing again. Of course, in some cases the internal participant is the end user. By all means,

in that case, use them exclusively for your test. For example, in the banking  system example, the banking system may have been developed internally for internal use. The end users are certainly internal to the company.

Finding internal participants may be as straightforward as searching your company directory on the intranet and firing off an email to someone who seems to be in an appropriate position to see if he or she indeed meets the selection criteria. Otherwise, your company's human resources department may be able to direct you to qualified end users in your own organization. For example, if you are working on a product that requires a certain expertise and you cannot use people who are working on the project directly, the human
resources (HR) department or the company intranet can direct you to people within the company who would tend to have the appropriate experience.
You then could call them up yourself, do the screening, and make all the arrangements for using them as participants.
By the way, if you decide to use outside agencies to recruit test participants, the human resources department may want to coordinate that. HR may have already set up relationships with these companies through other people doing research studies in your company of whom you may not be aware.

Find Sources of Participants - II

Find Sources of Participants - I

Okay. So far so good. You have analyzed the user profile for your product, received confirmation that indeed you understand who the end user is, and have developed a list or matrix of the people within your user profile who you would like to test with a description of their characteristics. The next step is to
recruit the people who fit this description to participate in your test.
Now let's discuss some of the sources from which you can acquire participants. The following sections discuss the most common ones. Some sources require creativity and tenacity to exploit to the best advantage. Some sources generate large numbers of responses but don't always net quality responses. We have listed our ideas for sources in order of how much work or expense each takes for the test moderator or recruiter, from least work and expense to most expense and time.

Consider Creating an "Answer Sheet"

If your questionnaire format does not make answers obvious, develop an "answer sheet" separately or right on the questionnaire that provides the replies that qualify a participant and how many of each category you need.
This can be as simple as a table or spreadsheet. We like a spreadsheet such as the one in Figure 7-5 better because you can later filter and sort the data you have gathered. (For an expanded sample of such a spreadsheet, go to the web site that accompanies this book: ww.wiley.com/usabilicytescing.com.)

Consider Creating an "Answer Sheet"

If your questionnaire format does not make answers obvious, develop an "answer sheet" separately or right on the questionnaire that provides the replies that qualify a participant and how many of each category you need.
This can be as simple as a table or spreadsheet. We like a spreadsheet such as the one in Figure 7-5 better because you can later filter and sort the data you have gathered. (For an expanded sample of such a spreadsheet, go to the web site that accompanies this book: ww.wiley.com/usabilicytescing.com.)

Consider Creating an "Answer Sheet"

If your questionnaire format does not make answers obvious, develop an "answer sheet" separately or right on the questionnaire that provides the replies that qualify a participant and how many of each category you need.
This can be as simple as a table or spreadsheet. We like a spreadsheet such as the one in Figure 7-5 better because you can later filter and sort the data you have gathered. (For an expanded sample of such a spreadsheet, go to the web site that accompanies this book: ww.wiley.com/usabilicytescing.com.)

Test the Questionnaire on Colleagues and Revise It

(Yes, you are right. This is a usability test of the screener!) If it is a phone questionnaire, use the phone. You would be surprised, but it makes a difference. If it is an email or web-based questionnaire, test it that way. Observe as a colleague tries it out by answering the questions to see what types of answers you get. If they are not what you are expecting, you must revise the questions, so it is clear that future respondents will understand what you want to find out and so you will be able to determine easily whether they qualify for the
study.
Regardless of whom you use to help you acquire participants, control the process as much as possible by being as specific as possible about your needs, anticipating miscommunications, and writing down the exact words you want the procurers to use. It is your study, they are your results, and the whole affair is seriously compromised when the wrong people show up.

Develop a Format for Easy Flow through the Questionnaire - II

Develop a Format for Easy Flow through the Questionnaire - I

Allow the caller to branch from one question to the next without thinking too much about the answers. This is especially important if you use an outside agency and the callers do not have an in-depth understanding of your product. Figure 7-4 shows an example of a screening questionnaire that was developed for the usability test of an accounting package for small businesses.
This questionnaire could be administered by phone or with slight adjustments by email. Note how the questions are intended to screen for a user who does his or her own bookkeeping on a PC, and who has experience using other office-related software. In administering a questionnaire like the sample one, the caller should not offer the choices listed below the questions, but simply mark them based on the respondent's answers. Likewise, if the questionnaire is to be administered by email, only the questions should appear without the possible answers. Note also that the questions focus on behaviors, not demographics. Figure 7-5 shows data collected by using the sample screener. (For an expanded sample of questionnaires and spreadsheets for recruiting and screening, go to the web site that accompanies this book: www. wiley. com/usability testing. com.)