Overview of the Methodology - Graphic

Overview of the Methodology -

Exploratory tests usually dictate extensive interaction between the participant and test moderator toestablish the efficacy of preliminary design concepts. One way to answer very fundamental questions, similar to those listed previously, is to develop preliminary versions of the product's interface and/or its support materials for evaluation by representative users. For software, this would typically involve a prototype simulation or mockup of the product that represents its basic layout, organization of functions, and high-level operations.

Even prior to a working prototype, one might use static screen representations or even paper drafts of screens. For hardware representations, one might use two-dimensional or three-dimensional foamcore, clay, or plastic models. For user support materials, one might provide very rough layouts of manuals, training materials, or help screens. When developing a prototype, one need not represent the entire function-

ality of the product. Rather, one need only show enough functionality to address the particular test objective. For example, if you want to see how the user responds to the organization of your pull-down menus, you need only show the menus and one layer of options below. If the user proceeds deeper than the first layer, you might show a screen that reads, "Not yet implemented," or something similar and ask what the participant was looking for or expecting next.

This type of prototype is referred to as a "horizontal representation," since the user can move left or right but is limited in moving deeper. However, if your test objective requires seeing how well a user can move down several menu layers, you will need to prototype several functions "vertically," so users can proceed deeper. You might achieve both objectives with a horizontal representation of rt// major functions, and a vertical representation of two of the functions.
During the test of such a prototype, the user would attempt to perform representative tasks. Or if it is too early to perform tasks, then the user can simply "walk through" or review the product and answer questions under the guidance of a test moderator. Or, in some cases, the user can even do both. The technique depends on the point in the development cycle and the sophistication of the mockups. the testing process for an exploratory test is usually quite informal and almost a collaboration between participant and test moderator, with much interaction between the two. Because so much of what you need to know is cognitive in nature, an exploration of the user's thought process is vital.
The test moderator and participant might explore the product together, with the test moderator conducting an almost ongoing interview or encouraging the participant to "think aloud" about his or her thought process as much as possible. Unlike later tests where there is much less interaction, the test moderator and participant can sit side by side as shown in Figure 3-2

Exploratory or Formative Study - II

Objective
The main objective of the exploratory study is to examine the effectiveness of preliminary design concepts. If one thinks of a user interface or a document as being divided into a high-level aspect and a more detailed aspect, the exploratory study is concerned with the former.
For example, designers of a Web application interface would benefit greatly knowing early on whether the user intuitively grasps the fundamental and distinguishing elements of the interface. For example, designers might want to know how well the interface:
■ Supports users' tasks within a goal.
■ Communicates the intended workflow.
- Allows the user to navigate from screen to screen and within a screen.  Or, using the task-oriented user guide of a software product as an example, technical writers typically might want to explore the following high-level issues:
■ Overall organization of subject matter
■ Whether to use a graphic or verbal approach
■ How well the proposed format supports findability
■ Anticipated points of assistance and messaging
■ How to address reference information
The implications of these high-level issues go beyond the product, because you are also interested in verifying your assumptions about the users. Understanding one is necessary to define the other. Some typical user-oriented questions that an exploratory study would attempt to answer might include the following:
■ What do users conceive and think about using the product?
■ Does the product's basic functionality have value to the user?
■ How easily and successfully can users navigate?
■ How easily do users make inferences about how to use this user interface, based on their previous experience?
■ What type of prerequisite information does a person need to use the product?
■ Which functions of the product are "walk up and use" and which will probably require either help or written documentation?
■ How should the table of contents be organized to accommodate both novice and experienced users?

Exploratory or Formative Study - I

When

The exploratory study is conducted quite early in the development cycle, when a product is still in the preliminary stages of being defined and designed (hence the reason it is sometimes called "formative"). By this point in the development cycle, the user profile and usage model (or task analysis) of the product will have (or should have) been defined. The project team is probably wrestling with the functional specification and early models of the product. Or perhaps the requirements and specifications phase is completed, and the design phase is just about to begin.

When Should You Test?

Our Types of Tests: An Overview

The literature is filled with a variety of testing methodologies, each with a lightly diferent purpose. Often, different terms are used to describe identical testing techniques. Needless tosay, thiscan be extremely confusing. In deciding which tests to discuss and emphasize, the most beneficial approach might be to use the product development lifecycle as a reference point for describing severa different types of tests. Associating a test with a particular phase in the lifecycle should help you understand the test's purpose and benefits.

We discuss three tests - exploratory (or formative), assessment (or summative), and validation (or verification) tests —at a high level, according to the approximate point in the product development lifecycle at which each would be administered. The fourth type of test, the comparison test, can be used as an integral part of any of the other three tests and is not associated with any specific lifecycle phase.

Hie basic methodology for conducting each test is roughly the same and is described in detail in Chapter 5. However, each test will vary in its emphasis on qualitative vs. quantitative measures, and by the amount of interaction

Limitations of Testing

Now, having painted a rather glorified picture of what usability testing is intended to accomplish, let's splash a bit of cold water on the situation. Testing is neither the end-all nor be-all for usability and product success, and it is important to understand its limitations. Testing does not guarantee success or even prove that a product will be usable. Even the most rigorously conducted formal test cannot, with 100 percent certainty, ensure that a product will be usable when released. Here are some reasons why:

- Testing is always an artificial situation. Testing in the lab, or even testing in the field, still represents a depiction of the actual situation of usage and not the situation itself. The very act of conducting a study can itself affect the results.

- Test results do not prove that a product works. Even if one conducts the type o test that acquires statistically significant results, this still does not prove that a product works. Statistical significance is simply a measure of the probability that one's results were not due to chance It is not a guarantee, and it is very dependet upon the way in which the test was conducted

Basic Elements of Usability Testing

■ Development of research questions or test objectives rather than hypotheses.

« Use of a representative sample of end users which may or may not be randomly chosen.

• Representation of the actual work environment.

■ Observation of end users who either use or review a representation of the product.

■ Controlled and sometimes extensive interviewing and probing of the participants by the test moderator.

■ Collection of quantitative and qualitative performance and preference measures.

■ Recommendation of improvements to the design of the product. We detail the "how-to" of this approach in the chapters that follow

Basics of the Methodology - II

The preceding approach is the basis for conducting classical experiments, and when conducting basic research, it is the method of choice. However, it is not the method expounded in litis book for the following reason. 

■ It is often impossible or inappropriate to use such a methodology to conduct usability tests in the fast-paced, highly pressurized development environment in which most readers will find themselves. It is impossible because of the many organizational constraints, political and otherwise. 

It is inappropriate because the purpose of usability- testing is not necessarily to formulate and test specific hypotheses, that is, conduct research, but rather to make informed decisions about design to improve products.

■ The amount of prerequisite knowledge of experimental method and statistics required in order to perform these kinds of studies properly is considerable and better left to an experienced usability or human fac-

tors specialist. Should one attempt to conduct this type of tight research without the appropriate background and training, the results can often be very misleading, and lead to a worse situation than if no research had

been conducted.

■ In the environment in which testing most often takes place, it is often very difficult to apply the principle of randomly assigning participants because one often has little control over this factor. This is especially true

as it concerns the use of existing customers as participants.

■ Still another reason for a less formal approach concerns sample size. To achieve generalizable results for a given target population, one's sample size is dependent on knowledge of certain information about that population, which is often lacking (and sometimes the precise reason for the test). Lacking such information, one mav need to test 10 to 12 participants per condition to be on the safe side, a factor that might require one to test 40 or more participants to ensure statistically significant results.

Basics of the Methodology - I

The basic methodology for conducting a usability test has its origin in the classical approach for conducting a controlled experiment. With this formal approach, often employed to conduct basic research, a specific hypothesis is formulated and then tested by isolating and manipulating variables under controlled conditions. Cause-and-effect relationships are then carefully examined, often through the use of the appropriate inferential statistical technique(s), and the hypothesis is either confirmed or rejected. Employing a true experimental design, these studies require that:

-A hypothesis must be formulated. A hypothesis states what you expect to occur when testing. For example, "Help as designed in format A will improve the speed and error rate of experienced users more than help as designed in format B." It is essential that the hypothesis be as specific as possible.

-Randomly chosen (using a very systematic method) participants must be assigned to experimental conditions. One needs to understand the characteristics of the target population, and from that larger pop-

ulation select a representative random sample. Random sampling is often difficult, especially when choosing from a population of existing customers.

-Tight controls must be employed. Experimental controls are crucial or else the validity of the results can be called into question, regardless or whether statistical significance is the goal, All participants should have

nearly the identical experience as each other prior to and during the In addition, the amount of interaction with the test moderator must be controlled.

- Control groups must be employed. In order to validate results, a control group must be employed; its treatment should vary only on the single variable being tested.

- The sample (of users) must be of sufficient size to measure statistically significant differences between groups. In order to measure differences between groups statistically, a large enough sample size must be used. Too small a sample can lead to erroneous conclusions.

Improving Profitability

Goals or benefits of testing for your organization are;

- Creating a historical record of usability benchmarks for future releases. By keeping track of test results, a company can ensure that future products either improve on or at least maintain current usability standards.

- Minimizing the cost of service and support calls. A more usable product will require fewer service calls and less support from the company.

- Increasing sales and the probability of repeat sales. Usable products create happy customers who talk to other potential buyers or users. Happy customers also tend to stick with future releases of the product,

rather than purchase a competitor's product.

-Acquiring a competitive edge because usability has become a market separator for products. Usability has become one of the main ways to separate one's product from a competitor's product in the customer's

mind. One need only scan the latest advertising to see products described using phrases such as "simple" and "easy" among others. Unfortunately, this information is rarely truthful when put to the test.

-Minimizing risk. Actually, all companies and organizations have conducted usability testing for years. Unfortunately, the true name for this type of testing has been "product release," and the "testing"

involved trying the product in the marketplace. Obviously, this is a very risky strategy, and usability testing conducted prior to release can minimize the considerable risk of releasing a product with serious usability

problems.

Eliminating Design Problems and Frustration

One side of the profitability coin is the ease with which customers can use the product. When you minimize the frustration of using a product for your target audience by remedying flaws in the design ahead of product release, you also accomplish these goals:

■ Set the stage for a positive relationship between your organization and your customers.

■ Establish the expectation that the products your organization sells are high quality and easy to use.

■ Demonstrate that the organization considers the goals and priorities of its customers to be important.

* Release a product that customers find useful, effective, efficient, and satisfying.

Informing Design

The overall goal of usability testing is to inform design by gathering data from which to identify and rectify usability deficiencies existing in products and their accompanying support materials prior to release. The intent is to ensure the creation of products that:

* Are useful to and valued by the target audience

■ Are easy to learn

■ Help people be effective and efficient at what they want to do

■ Are satisfying (and possibly even delightful) to use

Why Test? Goals of Testing

From the point of view of some companies, usability testing is part of a larger effort to improve the profitability of products. There are many aspects to doing so, which in the end also benefits users greatly: design decisions are informed by data gathered from representative users to expose design issues so they can

be remedied, thus minimizing or eliminating frustration for users.

What Is Usability Testing?

The term usability testing is often used rather indiscriminately to refer to any technique used to evaluate a product or system. Many times it is obvious that the speaker is referring to one of the other techniques discussed in Chapter 1 Throughout this book we use the term usability testing to refer to a process that

employs people as testing participants who are representative of the target audience to evaluate the degree to which a product meets specific usability criteria. This inclusion of representative users eliminates labeling as usability testing such techniques as expert evaluations, walk-throughs, and the like that do not require representative users as part of the process.

Usability testing is a research tool, with its roots in classical experimental methodology. The range of tests one can conduct is considerable, from true classical experiments with large sample sizes and complex test designs to very informal qualitative studies with only a single participant. Each testing approach has different objectives, as well as different time and resource requirements. The emphasis of this book is on more informal, less complex tests designed for quick turnaround of results Ln industrial product development

environments.

Follow-Up Studies

A follow-up study occurs after formal release of the product. The idea is to collect data for the next release, using surveys, interviews, and observations. Structured follow-up studies are probably the truest and most accurate appraisals of usability, because the actual user, product, and environment are all in place and interacting with each other. That follow-up studies are so rare is unfortunate because designers would benefit immensely from learning what happened to the product that they spent two years of their lives perfecting

Sales figures, while helpful, add nothing to one's knowledge of the product's strengths and weaknesses.

This is not a definitive list of methods by any means, and it is meant merely to provide the reader with an appreciation for the wealth of techniques available and the complexity involved in implementing a UCD approach It is a rare organization that performs all of these techniques, and just as few conduct them in their pure form. Typically, they are used in altered and combined form, as the specific needs and constraints of a project dictate For more about these techniques, check out our list of resources on the web site ^ accompanies this book at www.wiley.com/ao/usabilitytesting Now lets take a closer look at one of the most renowned techniques of all the ones discussed, and the focus of this book, usability testing, in Chapter 2

Usability Testing

Usability testing, the focus of this book, employs techniques to collect empirical data while observing representative end users using the product to perform realistic tasks. Testing is roughly divided into two main approaches. The firs approach involves formal tests conducted as true experiments, in order o confirm or refute specific hypotheses. The second approach, a Jess formal but still rigorous one (and the one we emphasize in this book), employs an iterative cycle of tests intended to expose usability deficiencies and gradually shape or mold the product in question.

Expert or Heuristic Evaluations

Expert evaluations involve a review of a product or system, usually by a usability specialist or human factors specialist who has little or no involvement in the project. The specialist performs his or her review according to accepted usability principles (heuristics) from the body of research, human factors literature, and previous professional experience. The viewpoint is that of the specific target population that will use the product.

A "double" specialist, that is, someone who is an expert in usability principles or human factors as well as an expert in the domain area (such as healthcare, financial services, and so on, depending on the application), or in the particular technology employed by the product, can be more effective than one without such knowledge.

Paper Prototyping

In this technique users are shown an aspect of a product on paper and asked questions about it, or asked to respond in other ways. To learn whether the flow of screens or pages that you have planned supports users' expectations, you may mock up pages with paper and pencil on graph paper, or create line drawings or wireframe drawings of screens, pages, or panels, with a version of the page for each state. For example, if the prototype is for a shopping cart for an e-commerce web site, you can show the cart with items, as items are being changed, and then with shipping and taxes added. (Or, you may simply dec.de to have the participant or the "computer" fill these items in as the session progresses.)

To learn whether the labels help users know what to expect next, and if the categories you have planned reflect how users think and talk about tasks you can show the top-level navigation. As the participant indicates the top-level choice, you then show the next level of navigation for that choice. The process

continues until the user has gone as deeply into the navigation as you have designed and prepared for the sessions.

Or, you may simply ask participants about the prototype you have created. The questions can range from particular attributes, such as organization and layout, to where one might find certain options or types of information. 

The value of the paper prototype or paper-and-pencil evaluation is that critical information can be collected quickly and inexpensively. One can ascertain those functions and features that are intuitive and those that are not, before one line of code lins been written. In addition, technical writers might use the technique to evaluate the intuitiveness of their table of contents before writing one word of text. The technique can be employed again and again with minimal drain on resources.

Open and Closed Card Sorting

Use card sorting to design in "findability" of content or functionality. This is a very inexpensive method for getting user input on content organization, vocabulary, and labeling in the user interface. You can either give participants cards showing content without titles or categories and have the users do the naming (an open card sort), or give participants preliminary or preexisting categories and ask participants to sort content or functions into those (a closed sort

Walk-Th roughs

Once you hav e a good idea who your target users are and the task goals they have, walk-throughs are used to explore how a user might fare with a product by envisioning the user's route through an early concept or prototype of the product. Usually the designer responsible for the work guides his or her colleagues through actual user tasks (sometimes even playing the role for the user), while another team member records difficulties encountered or concerns of the team. In a structured walk-through, as first developed by IBM to perform code reviews, the participants assume specific roles (e.g., moderator, recorder) and follow explicit guidelines (e.g., no walk-through longer than two hours) to ensure the effectiveness of the effort Rather than the designer taking on the role of the user, you may want to bring in a real user, perhaps someone from

a fav ored client.

Surveys

By administering surveys you can begin to understand the preferences of a broad base of users about an existing or potential product. While the survey cannot match the focus group in its ability to plumb for in-depth responses and rationale, it can use larger samples to generalize to an entire population.

For example, the Nielsen ratings, one of the most famous ongoing surveys, are used to make multimillion-dollar business decisions for a national population based on the preferences of about 1500 people. Surveys can be used at any time in the lifecycle but are most often used in the early stages to better understand the potential user. An important aspect of surveys is that their language must be crystal clear and understood in the same way by all readers, a task impossible to perform without multiple tested iterations and adequate

preparation time. Again, asking people about what they do or have done is no substitute for observing them do it in a usability test.

Focus Croup Research

Use focus group research at the very early stages of a project to evaluate preliminary concepts with representative users. It can be considered part of "proof of concept" review. In some cases it is used to identify and confirm the characteristics of the representative user altogether. All focus group research

employs the simultaneous involvement of more than one participant, a key factor in differentiating this approach from many other techniques.

The concepts that participants evaluate in these group sessions can be presented in the most preliminary form, such as paper-and-pencil drawings, storyboards, and/or more elaborate screen-based prototypes or plastic models. 

The objective is to identify how acceptable the concepts are, in what ways they are unacceptable or unsatisfactory, and how they might be made more acceptable and useful. The beauty of the focus group is its ability to explore a few people's judgments and feelings in great depth, and in so doing learn something about how end users think and feel. In this way, focus groups  are very different from — and no substitute for — usability tests. A focus group is good for general, qualitative information but not for learning about performance issues and real behaviors. Remember, people in focus groups are reporting what they feel like telling you, which is almost always different from what they actually do. Usability tests are best for observing behaviors and measuring performance issues, while perhaps gathering some qualitative information along the way.

Defined Usability Goals and Objectives

Designing a product to be useful must be a structured and systematic process, beginning with high-level goals and moving to specific objectives. You cannot achieve a goal — usability or otherwise — if it remains nebulous and ill -conceived. Even the term usability itself must be defined with your organization. An operational definition of what makes your product usable (tied to successful completion criteria, as we will talk about in Chapter 5) may include: 

■ Usefulness
■ Efficiency
■ Effectiveness
■ Satisfaction
■ Accessibility

Thus bringing us full circle to our original description of what makes a product usable. Now let's review some of the major techniques and methods a usability specialist uses to ensure a user-centered design

A 'Learn as You Co" Perspective

UCD is an evolutionary process whereby the final product is shaped over time. Tt requires designers to take the attitude that the optimum design is acquired through a process of trial and error, discovery, and refinement. Assumptions

Concerned, Enlightened Management

Typically, the degree to which usability is a true corporate concern is the degree to which a company's management is committed to following its own lifecycle and giving its guidelines teeth by holding the design team accountable. Management understands that there are financial benefits to usability and market share to be won

A Multidisciplinary Team Approach

No longer can design be the province of one person or even of one specialty.

While one designer may take ultimate responsibility for a product's design, he or she is not all-knowing about how to proceed. There are simply too many factors to consider when designing very complex products for less technical end users. User-centered design requires a variety of skills, knowledge, and, most importantly, information about the intended user and usage. Today, teams composed of specialists from many fields, such as engineering, marketing, training, user-interface design, human factors, and multimedia, are becoming

the norm. In turn, many of these specialists have training in complementary areas, so cross-discipline work is easier and more dynamic than ever before.

Attributes of Organizations That Practice UCD

User-centered design demands a rethinking of the way in which most companies do business, develop products, and think about their customers. While currently there exists no cookie-cutter formula for success, there are common attributes that companies practicing UCD share. For example.

■ Phases that include user input
■ Multidisciplinary teams
■ Concerned, enlightened management
■ A "learn as you go" perspective
■ Defined usability goals and objectives

Phases That Include User Input

Unlike the typical phases we have all seen in traditional development methodologies, a user-centered approach is based on receiving user feedback or input during each phase, prior to moving to the next phase. This can involve a variety of techniques, usability testing being only one of these.

Today, most major companies that develop technology-based products or systems have product lifecycles that include some type of usability engineering/human factors process. In that process, questions arise. These questions and some suggested methods for answering them appear in Figure 1-4.

Within each phase, there will be a variety of usability engineering activities Xote that, although this particular lifecycle is written from the viewpoint of the human factors specialist's activities, there are multiple places where collaboration is required among various team members. This leads to our next attribute of organizations practicing UCD.