Basics of the Methodology - I

The basic methodology for conducting a usability test has its origin in the classical approach for conducting a controlled experiment. With this formal approach, often employed to conduct basic research, a specific hypothesis is formulated and then tested by isolating and manipulating variables under controlled conditions. Cause-and-effect relationships are then carefully examined, often through the use of the appropriate inferential statistical technique(s), and the hypothesis is either confirmed or rejected. Employing a true experimental design, these studies require that:

-A hypothesis must be formulated. A hypothesis states what you expect to occur when testing. For example, "Help as designed in format A will improve the speed and error rate of experienced users more than help as designed in format B." It is essential that the hypothesis be as specific as possible.

-Randomly chosen (using a very systematic method) participants must be assigned to experimental conditions. One needs to understand the characteristics of the target population, and from that larger pop-

ulation select a representative random sample. Random sampling is often difficult, especially when choosing from a population of existing customers.

-Tight controls must be employed. Experimental controls are crucial or else the validity of the results can be called into question, regardless or whether statistical significance is the goal, All participants should have

nearly the identical experience as each other prior to and during the In addition, the amount of interaction with the test moderator must be controlled.

- Control groups must be employed. In order to validate results, a control group must be employed; its treatment should vary only on the single variable being tested.

- The sample (of users) must be of sufficient size to measure statistically significant differences between groups. In order to measure differences between groups statistically, a large enough sample size must be used. Too small a sample can lead to erroneous conclusions.