Beware of Inadvertently Testing Only the "Best" People - I

Very often when acquiring participants, and especially when you do not have primary control of participant selection, you will be sent only the very "best" people by your contact or recruiter. By "best," we mean that regardless of the category of end user, whether it is novice, expert, or whatever, you are sent the cream of the crop, the high achievers. Your first reaction to this might be very positive, but on close inspection, it is easy to see the potential problem here. The "best" end users typically possess the skills to plow through even the most hard-to-use products and perform admirably. Consequently, the product "tests out" much better than it should, and provides a false sense of confidence to the design and marketing team. Later, after release, when average and poor performers use the product, many of the design flaws that exist, but did not
come out during the test, are exposed.

This situation is especially apt to occur when you are acquiring participants under the following conditions:
■ When you will be testing end users from within your own company, usually from a department with which you are not familiar.

■ When you are acquiring people directly from an established customer and participation is seen as an enviable perk. In this case, the manager responsible for providing people sends participants to your company as a reward for a job well done at their company. Invariably, these are their best performers.