Organize the Questions in a Specific Order

It makes sense to place those questions that are most likely to disqualify a person first on a questionnaire, while being mindful of basic telephone etiquette. For example, if you are testing a graphic design product designed for seasoned end users with knowledge of color theory, you would not want to ask 20 very general questions only to find out in question 21 that the person you are speaking with does not know saturation from hue. You should establish that background early in the call to limit the phone time. 

On the other 

hand, that is not the type of high-level question you can ask first either. You need to work your way into it.

For the hotel reservations web site, you might want to eliminate candidates who don't book their own hotel rooms at all, so you would ask a question such as: "For your last three (or five; pick a number) trips, who made the arrangements?"

Formulate Screening Questions - II

You might ask respondents for the test of the hotel reservations web site to list all of the rewards programs they belong to: "Tell me about rewards memberships. Is that something you do? What memberships do you have?"
If people have trouble remembering, you might suggest they take out their wallet to flip through membership cards.
The case of the bank employees is slightly different. You can find out how long workers have been in their jobs from the human resources department or from the group supervisor. If you want to include workers who have mastered particular tasks or operations procedures, you may also be able to find that out from a supervisor. To be sure, you could ask candidates to tell you about their jobs, what their specialties are, what they like and don't like doing and why (you may want people who don't like doing the task you will be testing, to see if the new system improves their attitudes about it), and when the last time was that someone asked them for help on the task or procedure (perhaps an indication of expertise). Or, you could develop an assessment test.

Formulate Screening Questions - I

Once you have isolated the different factors for which you are screening you need to formulate questions to ascertain whether a person has that expertise and other characteristics defined for the test. Use the same approach you would use for creating any sort of questionnaire, especially if you will be handing your screener over to someone else to do the recruiting and scheduling We present a number of question formats in Chapter 8. Besides those question formats, you will have to operationalize a definition of expertise Determine

levels of expertise by creating a way of checking proficiency (what we call a criterion test) such as an actual quiz or a matrix similar to the one we show in Figure 7-1. Because people are often reluctant to divulge exact personal details if you must gather personal information, such as age or salary, frame your questions in ranges. For example, instead of asking, "What is your annual household income?" ask instead:

"Into what category does your annual household income fall?"

$20,000- $45,000 $46,000-$55,000
$56,000-$75,000 $76,0000-$100,000

Identify Specific Selection Criteria

Focus on and pull out those characteristics that are unique to your product and not just general categories of the population. For example, experience using a cell phone and a camera may qualify as a requirement for a product that combines the two. But, because the product will be used by all age groups, you need not screen for a particular age group to include in your test sample.
Later, as you are acquiring participants, you can simply make sure that you get a mix of different ages.
To follow the example for the hotel reservations web site, your objective is to learn what differences there are in how different types of travelers make hotel reservations. The team decides it wants to see whether there are differences between rewards members and travelers who are not members of the rewards
program. To select participants, then, some criteria are:
■ Are they members of our hotel rewards program? We want some of each, split into equal sized groups, if possible.
■ Are they members of nny rewards program? For participants who are not members of our program, do they ever belong to rewards programs? For this test, we want one group of people who do not join rewards
programs.
Can you see where you might end up with questions to ask potential participants from this?

For the banking example, let's focus on just the back office folks for now. The objective of the test is to learn whether users who are new to the back office can use the new system as efficiently and effectively as those who have extensive knowledge of the particular operations of this back office. Is it possible that the new system could cut down on training time? To select participants to help you find this out, you will need workers who have been in their jobs for a long time (whatever you and your team determine is an appropriate amount of time), but also some who have been through little or no training. So, some selection criteria might be:

Review the Profile to Understand Users' Backgrounds

Doing usability testing is just one of the reasons you develop the user profile. However, the user profile should prove a useful tool for visualizing the person you want to be in the testing room so you can articulate the requirements and classifiers and use those to form screening questions. If you don't have a user profile that documents the characteristics of the target users you want to have in the usability test, go back to the beginning of this chapter to "Characterize Users."

Write the Screening Questionnaire

The screening questionnaire is the means for qualifying and selecting participants to participate in the test. Its content is typically obtained from the characteristics, requirements, and classifiers that you included in the user profile and test plan. The questionnaire is usually presented over the phone, although you can also give it to potential participants by email or in a web-based questionnaire. If you will be using a colleague, an outside consultant, or an agency to help you recruit participants, the screening questionnaire is your primary way of describing the people you need in an unambiguous format. While some agencies will develop their own screeners based on information you provide, you must revieiv their questionnaires before they begin using
them, to ensure that your instructions have been understood
A screening questionnaire can be extremely simple or quite involved, depending on the variability and background of the potential participants of the product Either way, developing the screening questionnaire exposes
- Review the profile to understand users' backgrounds.
■ Identify specific selection criteria.
■ Formulate screening questions.
■ Organize the questions in a specific order.
- Develop a format for easy flow through the questionnaire.
■ Test the questionnaire and revise it.
■ Consider creating an "answer sheet."

Determine the Number of Participants to Test

The number of participants you decide to test depends on many factors, including:

■ The degree of confidence in the results that you require

■ The number of available resources to set up and conduct the test

■ The availability of the type of participants you require

■ The duration of the test session

Ultimately, you have to balance your need for acquiring participants with these (and probably other) practical constraints of time and resources. If you require statistically valid results, you will need to test enough participants to conduct the appropriate analyses and generalize to your specific target population, as well as to rigorously control for potentially biasing conditions and factors.

If, however, you are simply attempting to expose as many usability problems as possible in the shortest amount of time, then test at least four to five participants of each user or audience cell. Research indicates that testing four to five participants of each type or cell will expose the vast majority of usability problems. However, we add that we are a bit uncomfortable testing only four participants, and we try to test at least eight participants if possible. While you may expose most of the usability problems with four participants, there is still a good chance you may overlook a problem that could have severe ramifications.

An important consideration is whether you will conduct more than one test during the product development life cycle. 1/ you are going to conduct multiple tests, then you can feel more confident testing fewer participants in each test. For example, if you conduct three tests with five participants each, you end up testing fifteen people. If you will be conducting a single test, however, then you will be better served by having more participants for that lone test.

Consider a Matrix Test Design

Because you want to see a range of behavior and performance within your small group of participants consider a matrix design for the test that balance different variables or classifiers in such a way that no particular user group or cell of the matrix is left out. 6 p

Take the banking example. These days you can assume that having some computer experience is a requirement for getting a job either at the teller vvicket or in the back office, so that's probably not a useful way of classifying these participants. What could determine levels of expertise in the two roles that would be important to your test? Possibly a combination of time on the job and types of responsibility may describe an "expert" teller from a "novice" one or an "expert" back office clerk from a "novice" one. You could look at the factors you use to determine expertise as domain knowledge (for example, what the participant knows about being a teller) or tools knowledge (for example, the efficiency and effectiveness with which the participant uses the software or other technology required in her job). You'll want some of each and you'll

have to create some type of operationalized definition for determining who falls into which cell.
Good test design mea ns that you balance the mixtures of user groups in such a way that the major categories or cells are all represented. For the banking example, you might test 16 participants whose job and experience level look like the ones in the table below (assuming that you have clearly defined the criteria for "novice" and "expert"; see "Define Expertise," above).

Divide the User Profile into Distinct Categories

The user profile describes the range of skills, behaviors, and experience that make up the entire universe of target end users. Every end user should fall somewhere within that spectrum However, you are not interested in individuals alone. Instead, you should identify entire groups of end users who share many of the same characteristics on the user profile. These groups, sometimes identified by similar occupations or job titles but often as roles (for example, "frequent traveler/' "club member," "voter," or even "patient," "parent," "vendor," or "client"), will use the product and its support materials in different ways and for different purposes. This is also where your classifiers come in. Use those to help you group users into categories. It is important that you understand the user profile for each target audi- ence. For example, the hotel reservations web site previously mentioned is intended for both rewards members and travelers who are not rewards mem-

bers (our classifiers); each group may use the site differently. If you were testing that web site, you would want to select some members and some non-members. The proportion depends on the priorities of the business and the objectives and goals for the study. You don't have to try to match the proportions in the target market segments.

Or, suppose you were responsible for testing a banking system that both tellers and "back office" staff members will use. You will want to have some tellers who will use the system for performing daily customer transactions. 

You will also want to have some "back office" staff members who will use the system to generate reports and conduct different analyses. The tasks of the two groups are different.

Each group will have similar characteristics when compared to other members of their group, although with varying levels of expertise. With each group, make sure that you are representing these varying expertise levels, if any. For example, test some novice tellers, some experienced tellers, some novice "back

office" staff, and some expert "back office" staff. Suggestions for how many of each type to include are discussed later in this chapter.

In the hotel reservations web site example, rewards members are likely to be very experienced with the site and non-members are likely to have less experience with the site. But you want both groups of users to cover a range of levels of previous experience with the site. Or you may assume that participants'

overall travel booking experience is more important than their direct experience with your site. In that case, you might want to classify the participants based on the frequency with which they book accommodations in the course of a year, or whether most of their travel is leisure or business-related, again depending on the objective and goals of the test. To represent these different skill levels, you may want to use a matrix test design, discussed in the next section.

Document the User Profile - III

Document the User Profile - II

Document the User Profile - I

You have gathered information about target users from various parts of your company. TTien you interpreted that information to develop requirements and classifiers for the usability test. Having a written profile of the target audience (either supplied to you or formed by your own research) should help you, the designers, and the'developers throughout the development of the product. Such a snapshot is not only beneficial for designing a usability test but also helps the team to visualize the person for whom they are designing 

the product, another benefit of doing this type of analysis. We use the term per person in mind when they are designing, rather than many different types of people. Unfortunately, the person they visualize, in lieu of real data, is often himself or herself, even though the developers may not even remotely reflect an actual end user. So, an accurate picture of the user can be a real boon to the design process because it helps to keep the designers and developers on track. Figure 7-2 shows a demographic profile for the hotel reservation system

web site. This is the type of data you may get from marketing, research and development, or product managers. Figure 7-3 shows an example narrative persona for an archetypical user of the site. Someone in human factors, technical communication, or interface design or even marketing may have created something like it. The user profile you develop may end up being a combination of these and other types of information. If you are using personas, you should develop one for each distinct user group that typifies an aggregate

of several representative individuals.

Specify Requirements and Classifiers for Selection - II

For a company's hypothetical online hotel reservation system, there are many possible audiences for the web site: business travelers, leisure travelers (who could be the same people as the business travelers), administrative assistants who book travel for others, spouses of trav elers, hotel employees, rewards program members, meeting or event planners, and so on. To keep this test (and our example) simple, let's say that you want to concentrate on two user groups to see if there are differences in how successful the) are with the reservations system: people who are members of your rewards program and people who are not. These are classifiers.

TT,« breakdown has implications for the test design and implications ror fiJJn* oarS Its You know that most users of the web site reservations svstnfare Srds members. Should you then make sure that most of the

participants are rewards members? It depends on the business requirements and the coals of the test. Is i t the goal of the test to make the s.te easier to use for rewards members or for all users? Do you want to compare error rates or other measures between the two types of users? People who are rewards members 

mav indeed use the site more frequently than non-members, giving them many more opportunities to learn its quirks and to invent their own workarounds for any hindrances. These people could reveal interesting findings that may suggest to the design team things they could do to improve the interface.

But if one of the business goals is to attract more people to the s.te - people who are not rewards members - certainly you want a significant number of non-members to take part in your test.

Specify Requirements and Classifiers for Selection - I

Having visualized whom you want to be vour from interna, sources about the groups, now try to think in terms of requin-.uc^ and Wass.user profile. Requ.rements describe what participants must come im^h

session knowing already. Classifiers he.r you put studv participant iltS categor.es (or groups) that may explain some effects you see in the E To make the concept of a user profile or user characterization more concrete you can take the traveler profile a bit further than the original visualization In this test, a participant must:

- Stay at a hotel a few times a year: We want people who would nor malty stay ,n a hotel rather than with friends or family or at a hostel, rather than a vacation home or time-share.

- Book some travel and accommodations online: Because this is an online reservation system, we want people who are familiar and com fortable with doing searches and credit card transactions on the web.

These are requirements. There's a temptation to get more specific What kinds of hotels do they stay in? How long are their stays? Which web sites do they use to make reservations through? But none of that really matters for selecting participants for this test, You can find out those interesting things later from a background questionnaire or in the session. Doing so may give you some context for analyzing the data from the test, but knowing these things ahead of time will not help you know why participants performed the

way they did with your reservation system. It may help to be more specific in the timeliness of the requirements — that is, participants should have done these things recently.

Define Expertise - III

Define Expertise - II

Dana changed the operational definition of "novice" and "expert" for that usability test to include knowledge of using a web browser and evidence of the person doing a range of complex tasks on the web. This matrix of expertise (shown in Figure 7-1J also avoided the possible mismatch of expectations when participants are asked to rate their own expertise.

Jt's more accurate to use criteria that are measurable, objective "facts"  rather than subjective self-classifications on the part of potential end users. A seif-cJassification has the end user rate his or her own abilities and skill level, which is more subject to misinterpretation and wide variance.

For example, when attempting to classify a group of end users as "novice," before using them in a usability test, do not simply accept their own rating of themselves as "novice," "expert," and so on. Instead, find out by using a method something like the matrix in Figure 7-1. Although it still relies on participants reporting what they remember doing, at least it does gather more objective data about what they have done. This latter method has the advantage of using criteria on which everyone can readily agree. The team can determine a "score" for how each combination of tasks and frequencies defines different levels of expertise. For example, someone who marked all the tasks in the "Every week" row would likely be an expert user of the 

Internet. Likewise, someone who marked all of the tasks in the "Never" row probably would not qualify for most usability tests of web sites (depending on the objectives and research questions). People marking boxes between those extremes then become defined as "novice" or "intermediate/' again,

Define Expertise - I

It is tempting to use time and frequencies as a measure of expertise. While these are measures of experience, they do not always describe expertise. For example, in one study Dana ran, she looked for people with a range of experience using the web. The original screening criteria looked at the average amount of time spent on the web over a week and how long the participant bad been using the web overall. Though Dana expected that someone who had been using the web for a few years for 60 hours per week would be an expert web user, this was not the case. One notable participant did nothing on the web but play four specific games, getting to each from shortcut icons set up for her by her grandson. She didn't know how to use a web browser at all.

Define the Criteria for Each User Group

Competitive Benchmarking and Analysis Croup

Some organizations assign a eroun marking of their own cJpeM^Z^^d° eXt6nsive a good handle on not only who is usine th^tS"^"^ ^ willhave competitor's products as well. This can be a wond.f . Wh° Usin» their
for establishing the user profile. t rtul S0l"«! of informaHon

Product Manager (Marketing)

Product Manager (R&D)

If you are unable to get your hands on specific documents, analyses or reports there are certain individuals who should be able to help determent the user profile. Among them is the product manager who sould have his or her finger on the pulse of the marketplace and should have a clear understainding of the end users for whom the product is intended . the product manager may also have access to reports and surveys that describe the user profile in the detail or she may be able to describe example customers whom she met.

Structured Analyses or Marketing Studies

Dev elopers, technical writers, or human factors specialists may hav e already completed task analyses or similar analyses prior to any desil workS usually mdude the skill and knowledge set required* £ the produ !
effectively. In larger corporations, there is aJmost sure to have been am arl« tine study conducted for each unique product, which has a simt ar br'kdown

Requirements and Specification Documents

There are several documents created in the early stages of product development that may be useful in beginning to generate a user profile: the functional specification, business requirements document, design specification, or use cases.

- The functional specification or product requirements document is the blueprint for the product. It describes the product's intended functionality , as well as the tasks that the end user will perform. Most functional

specifications include a comprehensive description of the intended user population (if an interna] product) or market {if an external product) for the product, which is the basis for your user profile.

- I f all you find in the functional specification is specifications for functionality, you may find market segment information in a business requirements document (often referred to as a BRD). The BRD outlines

the business case for the product or its planned upgrade, including information about whom the product is targeting and why. 

- A design specification typically describes the system archjtecture for supporting a software product or web site but may include end-user characteristics as part of the assumptions and dependencies it covers.

■ Use cases typically include information about end users as "actors" someone who interacts with part of the proposed system. The actor wants to achieve some goal, so you may learn useful things about tasks

that the team envisions users doing.

Look for Information about Users

We assume that there is no established user profile (a written set of characteristics) or set of profiles in place as you begin to think about usability testing. So, where shall you look for information to develop a detailed user profile Let's discuss some of the potential sources of this type of information in your organization. Once you and your team agree on a profile or two or three, then you can determine who among those should be included as participants in the usability test. The following sections list some typical places and people from
whom to ascertain the user profile. Keep in mind that the format or content of the information you get may not make an obvious user profile. You may have to apply some analysis and critical thinking to establish those characteristics that make up realistic human traits and behavior.