(This is Part 4. Please read Part 1 , Part 2, and Part 3 first.)
Okay, in this post I’m going to get a little down and dirty. I’ll show some examples of research which do not seem to really enable a design team to learn more about their users, nor convince stakeholders about correct design decisions, but which seem instead intended to put up a facade of scientific truth in order to create a more expensive service offering.
Have a Beer in One of Our “Persona Rooms”
I really really don’t want to take cheap shots at my company’s competitors, nor do I feel comfortable about the idea of ridiculing a leading industry researcher… but when I read about this, my jaw dropped to the floor.
Organic, an excellent top-tier web design agency, was recently featured in a Forrester research paper for their innovative work in the area of user personas, and they’ve been getting a lot of positive buzz about it.
Their innovation? “Persona Rooms“: In addition to building standard user personas, Organic also hires professional set designers to build actual living room spaces in their office, one living room for each user persona. The design team then spends time hanging out in these rooms, gaining a deep understanding of what it’s like to be “Bob, the single dude” or “Mary, the homemaker”. They then use this experience to inform their design process.
Any alarms going off yet?
Forrester is a leading technology and market research company, and they have a research specialization in the web design industry. They have, admirably, been a leader in promoting user-centric practices in web design. If you have a large client who is familiar with the idea of user personas, there’s a good chance that they might have read one of Forrester’s reports on the topic.
But their user-centric boosterism may have gone too far in this case. Forrester’s review of Organic’s persona rooms service offering was positively gushing: “Persona rooms are clearly the next big thing in the art of persona creation”.
Huh? Everyone I’ve shown this too — everyone — has said “Huh?” and asked me if this wasn’t a joke. Are clients actually paying to create hang-out rooms as if these were a legitimate form of user research? I mean, it sounds like a cool idea, and again (like chicken soup) it can’t hurt. Well, it can’t hurt the design process, but is it worth the money? How can Forrester recommend this to their clients with a straight face? Where is the common sense incredulity at the concept?
Forrester writes “Visitors can walk out of the world of the PT Cruiser buyer and, in two steps, enter the world of a Jeep customer — a far more compelling representation of market segments than any set of slides or flip charts could ever achieve.” This may be true, but are they tens of thousands of dollars more compelling?
Magically Transforming the Subjective into the Objective
The Forrester research paper is available for download from the Organic web site, and you should certainly read it to understand it before passing judgement. While you’re there, you might also want to look at Forrester’s industry analysis of Organic’s service offering just out of curiosity. (These papers cost, respectively, $350 and $800 if you want to get them directly from Forrester’s…)
Forrester’s broader analysis of the web design industry is, by the way, an excellent example of how researchers can take a perfectly good subjective analysis (such as how good one firm is at branding services versus another), attach an arbitrary number to it, process that number with a bunch of other subjectively-derived numbers, and deliver a final result which has all the bells and whistles of a objective statistical analysis of empirical data but which is, ultimately, still subjective.
Just look at this chart on the right, and tell me if it isn’t reasonable for one to conclude that R/GA is twice as good at branding than Avenue A|Razorfish, or three times as good as FCBi. The “it depends” factor is totally absent in this diagram, even though all of these numbers are based on “measurements” exactly like the branding example above. Forrester’s actual research and analysis is no doubt excellent, but expressing their conclusions quantitatively gives the illusion of a level of authority and accuracy that simply doesn’t exist. The text of the report is insightful and obviously useful for anyone considering hiring a web design firm — the charts, however, are meaningless.
The IDEO Shopping Cart Legend
Many designers are familiar with ABC Nightline’s 1999 feature story about the product design firm IDEO, in which IDEO’s use of playful ethnographic research was shown to be the key to their great design process. Specifically, the special followed IDEO around as they addressed the challenge of designing a better shopping cart.
Almost every designer I know has watched this video with great envy, wondering how they can make their own design process as inspirational and perfect as IDEO’s process. We wish we worked at a fun cool company like IDEO! Well, Steve Portigal, in a recent interview with Luke Wroblewski, throws some much-appreciated cold water on this story:
I’m glad you refer to the IDEO example as “infamous” – because it’s not a real example. It’s a made-for-TV special that is filled with fakery. In one shot, a woman is walking through a grocery store supposedly documenting the environment with a digital camera. But she’s an IDEO staffer, not an actor, so her body language as she pretends that the camera is not watching her pivot right and left snapping away carefree outs her (and indeed the whole process) as manufactured.
The camera did not document IDEO doing what they do, IDEO agreed to stage an event specifically to be documented. They created (and the producers edited) an idealized process that none of our work will ever live up to, because it can’t, because it’s real and constrained and challenged in ways that real projects always are.
I don’t blame IDEO for this — it’s PR, and exaggeration is in the nature of it. I only mention this because I want to give courage to designers who think that real user research requires a huge investment of time, money, and people. The IDEO story is incredibly inspirational, but like most heroics, it is idealized.
Real projects are messy and imperfect. I suspect the same is true at IDEO. I also suspect that IDEO’s great product designs ultimately result more from having great designers on their team — their use of research helps the designers make decisions, but it’s not, as Nightline would have you think, the very foundation of their design process — their designers are the foundation of their design process.
Graphic Designers Do it, Too
It’s not just web designers who pretend to have empirical reasons for design decisions — graphic designers do it as well. Pentagram’s Michael Beirut said in a recent interview with Peter Merholz:
It’s a dirty secret that much of what we admire in the design world is a byproduct not of “strategy†but of common sense, taste and luck. Some clients are too unnerved by ambiguity to accept this, and create garganuan superstructures of bullshit to provide a sense of security. Not only do designers enthusiastically collude in this process, but many have found ways to bill for it.
Graphic designers aren’t fine artists driven by personal idiosyncratic obsessions or inner demons, nor are they cognitive researchers or statistical analysts making design decisions based on MRI scans of the optic lobe of the brain. They are professional communicators whose experience and talent largely define their effectiveness. But they are often asked to provide, and sometimes they willingly construct on their own initiative, a convoluted justification for their design that stretches the limits of credibility. This is a waste of the designer’s time and their client’s money. Designers should do research, yes, but it’s just wrong to make stuff up.
Anyway, as I’ve said before, I could be the one who’s actually full of shit here. Some of these examples come damn near being cheap shots, but I’m just calling ’em how I see ’em. If you think I am wrong, please let’s discuss.
Comments
13 responses to “User Research Smoke & Mirrors, Part 4: Research as Bullshit”
man, when i was at organic we had friday massages and kegs, pool tables and video games. now an IA can be a couch potato for $250 per hour? what a joke. the “getting real” boys are gonna have a field day with this one…
i couldn’t imagine pitching this to a client as a valuable component of any research process. forrester says it’s “a far more compelling representation of market segments than any set of slides or flip charts could ever achieve” and therein lies the problem.
design personas are not representations of market segments; they’re representations of the desires, goals, needs, behaviors, quirks, pain-points, etc. of numerous people across one or more market segments.
yes, that sounds like a subtle difference, but if forrester doesn’t understand the difference between a stereotype and an archetype… well hell, i’d call *that* bullshit.
You wrote: “Designers should do research, yes, but it’s just wrong to make stuff up.” After reading that paragraph several times I realize now you are saying that it is wrong to make up a bunch of research or some method to justify the design decision.
At first, however, I thought you were saying it was wrong for the design team to make up stuff. That is, to make up stuff that is the design. I was thinking that sentence should read: “Designers should do research, yes, but it’s just wrong NOT to make stuff up.” This staement doesn’t belong in that paragraph, but stands on its own. Why?
Because our job is to make stuff up. Of course it has to serve the needs of the client, the business and the users. I had an interesting response from the director of the Database unit at a client to this idea. She asked where the REQUIREMENT was for a particular feature. I explained that the users had a NEED and the business had a GOAL and the feature we MADE UP addressed both.
She freaked out over “MADE UP”. No matter how many times I explained that our job was to come up with/make up/create/invent solutions that met the needs, goals and requirements of the customer, she could not accept that we had “made up” a feature that wasn’t specifically detailed in a customer-written requirements doc. Egads. Sometimes we wonder why we even try to explain it.
Good one, Ralph. Yes, making up is what designers do. We synthesize solutions often out of thin air. Design is not deduced entirely from research the way proofs are derived from axioms in geomettry. Design, and indeed all creative thinking including technology and business strategy, includes a hefty dose of reaching back into the idiosycrasies of one’s mysterious brain and finding clever and non-obvious solutions.
Your director sounds like someone who lives in constant fear of being accused of the crime of fostering innovation.
Is it really wrong to make up stuff (I don’t mean create, but I mean lying, bullshit…) if in fact it’s what the client wants? Let’s say you build the perfect solution, but it doesn’t sound sexy enough. Is it realoly wrong to lie IF you’re simply telling the truth in a new way? Like, let’s say your point is design X is now more efficient than design Y, because the server prove it; but the client wants to know that users find the site faster. You already know it is, so you claim user testing proves it. Client is happy, your happy, users are happy. Isn’t this a white lie?
(btw, this is just speculation and I may actually still consider this wring.)
That’s a tough one, Mario. I’ve stood by and watched my client lie to their clueless colleagues to the benefit of all, saying that usability testing showed X when maybe it only at best suggested X.
Instead of making up research to battle clueless people who will only listen to research, however, I often draw on my experience with the UI design community, saying something like “the consensus in the UI community is that tag clouds, while having a cool factor, are not clear for most normal web users” — I’m not lying, I’m not citing non-existent research, but I am making an argument that suggests empiricism (that is, I mean the listener to think I have some scientific authority behind me) perhaps a little deceptively. But it’s 100% true, and if the client wants me to spend time looking up and citing research studies (or, worse, to conduct research) even though the results are already held as a basic assumption in the profession, then it’s a real waste of money and I want them to know that.
“Graphic designers…are professional communicators whose experience and talent largely define their effectiveness.”
First, allow me to take issue with the reference to ‘graphic’ — these are design issues (graphic or otherwise). That aside, the real underlying talent that is largely misunderstood is an ability to recognize, identify and reshape patterns. And it is an ability that is innately more prevalent in females — it is the princple that drives ‘female intuition’. What is being ‘intuited’ is a recognition and connection to prior experiences from which a similar conclusion is drawn. It’s an ability to connect the dots; it’s an ability to traverse the ‘in-between’ — to find order where there is none — to exercise the divine power by which the universe was created — to organize matter.
That’s the fundamental principle of design and design thinking.
The great landscape architect and philosopher Lawrence Halprin got it right when, at a seminar in Seattle, he was asked the secret of his and other notable LAs’ success. He replied,
“Design not with forms but with forces.”
A more productive line of collective reasoned argument, now that bad research has been thoroughly debunked (but not banished), is to identify the forces that we must design with.
Paula: Michael Beirut is a graphic designer, and I was talking about him and talking about the field of graphic design. Why on earth do people still seem to view the term “graphic designer” as pejorative? That’s so 1990’s.
Making up research blurs the line with the techno-babble that is so pervasive in this industry. I’ve been in more than one meeting where
–person A and person B were talking about entirely different topics but didn’t know it (and I didn’t interfere)
–person A was making up bullshit but person B was afraid to call him on it
–person A was trying to make up bullshit but failing and person B wanted him to save face so he played along
all this can happen with the non-techie observers thinking everything is very scientific and under control (oh, and we’ll need more time and money…)
[…] Don’t get me wrong, I love qualitative research. But for execs who seek cold, hard numbers, qualitative research is often meaningless and untrustworthy. It is seen as fluffy psychobabble or artistic/creative posturing. So when a designer or a researcher wants their insights to be taken seriously, they often feel the need to “translate” their extensive subjective insights into objective numbers, a process that I think is just another flavor of bullshitting. […]
People who make up research findings and tote crap methods risk putting the rest of us who actually do good work out of business. If you can’t justify your work without lying, please take your dishonesty to another field. I’m sure you can steal from someone in a more profitable fashion if you used your imagination as so called “designers” and “user research experts”, so please move on. I hope that clients will wise up and start asking for references and qualifications before these charlatans give user-centered methods a bad name.
where is the dam graph paper
In this pioneering field of interactive, every company wants to provided a competitive edge because they think thats important. There are a lot of uncharted waters and we continue to define what this all means to ourselves and our clients. This is where ideas like “persona rooms” come in. Ok, sure… I guess. But what value does it bring to users and clients? Is it the next great value-added idea? Or, is it like a lot of the things that have been invented over the years that have gone to the wayside? I do have an opinion about it (that I’m not going to share :). Anyway, keep in mind that we are all pioneers in the interactive/internet medium. We still dont have a clue what we are doing (yet) and it will be sometime before we do. In the meantime, we keep innovating and thats a good thing. Persona rooms? Um… maybe…