Beware of Inadvertently Testing Only the "Best" People - II

« When you are acquiring people from an established customer through your own sales force. In this case, similar to the previous one, there is also a hidden agenda. Your sales rep very often has strong relationships

that he or she would like to further by using the testing as a perk. These relationships are often with the best performers and most influential people in your customer's organization. You never see the average or poor performers.

We mention these situations as warnings because "best" people are hard to predict. We have learned this the hard way, having had people show up as participants who were so accomplished that they should have been designing the product, never mind using it. Such users were able to foresee and work around the most troublesome areas, almost as if they knew they were there.

These participants also downplayed whatever problems they encountered, although they were critical ones. They can also be the most critical. One or at most two of these people during a test is reasonable, but if you see three or four, it plays havoc with your test results.

What makes the situation even worse is when the development team attending the test does not agree that these people are "ringers." The team is more than haPPv to bask in the glow of receiving unexpected but excellent results. Now, if you could just get all "ringers" to buy the product in sufficient numbers, everyone would be happy.