Okay, in this post I’m going to get a little down and dirty. I’ll show some examples of research which do not seem to really enable a design team to learn more about their users, nor convince stakeholders about correct design decisions, but which seem instead intended to put up a facade of scientific truth in order to create a more expensive service offering.
Have a Beer in One of Our “Persona Rooms”
I really really don’t want to take cheap shots at my company’s competitors, nor do I feel comfortable about the idea of ridiculing a leading industry researcher… but when I read about this, my jaw dropped to the floor.
Organic, an excellent top-tier web design agency, was recently featured in a Forrester research paper for their innovative work in the area of user personas, and they’ve been getting a lot of positive buzz about it.
Their innovation? “Persona Rooms“: In addition to building standard user personas, Organic also hires professional set designers to build actual living room spaces in their office, one living room for each user persona. The design team then spends time hanging out in these rooms, gaining a deep understanding of what it’s like to be “Bob, the single dude” or “Mary, the homemaker”. They then use this experience to inform their design process.
Any alarms going off yet?
Forrester is a leading technology and market research company, and they have a research specialization in the web design industry. They have, admirably, been a leader in promoting user-centric practices in web design. If you have a large client who is familiar with the idea of user personas, there’s a good chance that they might have read one of Forrester’s reports on the topic.
But their user-centric boosterism may have gone too far in this case. Forrester’s review of Organic’s persona rooms service offering was positively gushing: “Persona rooms are clearly the next big thing in the art of persona creation”.
Huh? Everyone I’ve shown this too — everyone — has said “Huh?” and asked me if this wasn’t a joke. Are clients actually paying to create hang-out rooms as if these were a legitimate form of user research? I mean, it sounds like a cool idea, and again (like chicken soup) it can’t hurt. Well, it can’t hurt the design process, but is it worth the money? How can Forrester recommend this to their clients with a straight face? Where is the common sense incredulity at the concept?
Forrester writes “Visitors can walk out of the world of the PT Cruiser buyer and, in two steps, enter the world of a Jeep customer â€” a far more compelling representation of market segments than any set of slides or flip charts could ever achieve.” This may be true, but are they tens of thousands of dollars more compelling?
Magically Transforming the Subjective into the Objective
The Forrester research paper is available for download from the Organic web site, and you should certainly read it to understand it before passing judgement. While you’re there, you might also want to look at Forrester’s industry analysis of Organic’s service offering just out of curiosity. (These papers cost, respectively, $350 and $800 if you want to get them directly from Forrester’s…)
Forrester’s broader analysis of the web design industry is, by the way, an excellent example of how researchers can take a perfectly good subjective analysis (such as how good one firm is at branding services versus another), attach an arbitrary number to it, process that number with a bunch of other subjectively-derived numbers, and deliver a final result which has all the bells and whistles of a objective statistical analysis of empirical data but which is, ultimately, still subjective.
Just look at this chart on the right, and tell me if it isn’t reasonable for one to conclude that R/GA is twice as good at branding than Avenue A|Razorfish, or three times as good as FCBi. The “it depends” factor is totally absent in this diagram, even though all of these numbers are based on “measurements” exactly like the branding example above. Forrester’s actual research and analysis is no doubt excellent, but expressing their conclusions quantitatively gives the illusion of a level of authority and accuracy that simply doesn’t exist. The text of the report is insightful and obviously useful for anyone considering hiring a web design firm — the charts, however, are meaningless.
The IDEO Shopping Cart Legend
Many designers are familiar with ABC Nightline’s 1999 feature story about the product design firm IDEO, in which IDEO’s use of playful ethnographic research was shown to be the key to their great design process. Specifically, the special followed IDEO around as they addressed the challenge of designing a better shopping cart.
Almost every designer I know has watched this video with great envy, wondering how they can make their own design process as inspirational and perfect as IDEO’s process. We wish we worked at a fun cool company like IDEO! Well, Steve Portigal, in a recent interview with Luke Wroblewski, throws some much-appreciated cold water on this story:
I’m glad you refer to the IDEO example as “infamous” – because it’s not a real example. It’s a made-for-TV special that is filled with fakery. In one shot, a woman is walking through a grocery store supposedly documenting the environment with a digital camera. But she’s an IDEO staffer, not an actor, so her body language as she pretends that the camera is not watching her pivot right and left snapping away carefree outs her (and indeed the whole process) as manufactured.
The camera did not document IDEO doing what they do, IDEO agreed to stage an event specifically to be documented. They created (and the producers edited) an idealized process that none of our work will ever live up to, because it can’t, because it’s real and constrained and challenged in ways that real projects always are.
I don’t blame IDEO for this — it’s PR, and exaggeration is in the nature of it. I only mention this because I want to give courage to designers who think that real user research requires a huge investment of time, money, and people. The IDEO story is incredibly inspirational, but like most heroics, it is idealized.
Real projects are messy and imperfect. I suspect the same is true at IDEO. I also suspect that IDEO’s great product designs ultimately result more from having great designers on their team — their use of research helps the designers make decisions, but it’s not, as Nightline would have you think, the very foundation of their design process — their designers are the foundation of their design process.
Graphic Designers Do it, Too
It’s not just web designers who pretend to have empirical reasons for design decisions — graphic designers do it as well. Pentagram’s Michael Beirut said in a recent interview with Peter Merholz:
Itâ€™s a dirty secret that much of what we admire in the design world is a byproduct not of â€œstrategyâ€ but of common sense, taste and luck. Some clients are too unnerved by ambiguity to accept this, and create garganuan superstructures of bullshit to provide a sense of security. Not only do designers enthusiastically collude in this process, but many have found ways to bill for it.
Graphic designers aren’t fine artists driven by personal idiosyncratic obsessions or inner demons, nor are they cognitive researchers or statistical analysts making design decisions based on MRI scans of the optic lobe of the brain. They are professional communicators whose experience and talent largely define their effectiveness. But they are often asked to provide, and sometimes they willingly construct on their own initiative, a convoluted justification for their design that stretches the limits of credibility. This is a waste of the designer’s time and their client’s money. Designers should do research, yes, but it’s just wrong to make stuff up.
Anyway, as I’ve said before, I could be the one who’s actually full of shit here. Some of these examples come damn near being cheap shots, but I’m just calling ’em how I see ’em. If you think I am wrong, please let’s discuss.