On the NYC-CHI mailing list today, someone from a big web shop wrote this:
My company is dismantling its focus group and usability testing lab, due to a chronic lack of space and clients’ growing reluctance to allow our agency to do testing on our own work.
Behavior‘s thinking has always been that it takes an awful lot of chutzpah to try to sell in-house usability testing services as part of a web design and development process. We build test prototypes, write test plans, advise on the creation of test screeners and suggest types of participants for recruiting, and of course we observe the testing and take notes — but the actual recruitment of subjects, the proctoring/facilitating of the sessions, the recording of the sessions, and the synthesis and reporting of the test results is done by a third party, always.
How do other consulting firms “get away” with testing their own designs? If your company does offer design and in-house usability testing services, have you heard clients express distrust of the model? If so, how do you get over it?
UPDATE: I should distinguish between the different scales of user testing here. When a design team conducts quick and informal usability testing (i.e., non-lab-based, such as with colleagues and friends), well, somehow to me that’s a lot easier to swallow than a when a large-scale formal lab study is done by that design team. It’s a healthy part of a design process to build in informal testing, and the benefits of stepping back and reviewing a site in this way far outweigh the risks of bias or glossing over problems.
It’s funny how wildly different the two ends of this spectrum seem, at least to me: Low-fi informal testing done by the design consultant seems, to me, healthy and honest and worth paying extra for… while major formal lab testing done by the same design consultant seems highly vulnerable to bias. Maybe it’s because the stakes seem so much higher in the formal testing.