Peter Morville’s well-known “honeycomb” diagram (and accompanying article) illustrates seven qualities or “facets” of user experience design, going beyond just usability into six other areas where the user experience designer’s work is cut out for them. It’s a great diagram — I use it with clients to describe all the things we need to address, and I use it in my classes to help my students see the full scope of the UX designer’s responsibilities.

It occurred to me that a lot of what is included in this diagram is, in fact, entirely subjective in nature. Of these seven facets of UX design, how many of them can actually be measured by objective, empirical, or quantitative means during the design process?

I’m limiting the question to the design process itself, because once a product is released to the market it’s clearly a lot easier to measure its performance than it is to measure it while you are still building it

• Useful: Immeasurable. Sure, focus groups might help us understand what users will find useful, but ultimately even those studies are subjective. Maybe market analysis or customer surveys could approach quasi-quantitative results (i.e., a number of qualitative results can be run through a formula and articificially transformed into quantitative results, such as “21% of customers said they would find an auto-save feature useful”), but I don’t count that as a true measure of the empirical usefulness of a feature.
• Usable: Probably the easiest to measure (such as with clocks and performance counters), but still prone to subjective conclusions and flawed methodologies (as I’ve just written about extensively).
• Desirable: Immeasurable. Again, this can be researched but not truly measured. See “useful” above.
• Findable: Similar to “usable”, this is possible to measure with empirical performance metrics.
• Accessible: Often this is measurable, at the very least legally, according to a series of subjective but expertly-determined standards. There are gray areas, of course, and sometimes the standards are wrong and easily misinterpreted (putting ALT tags on bullet point GIFs for example), but in general I’d say accessibility is measurable.
• Credible: Immeasurable.
• Valuable: Immeasurable.

So it looks like at least 50% of the user experience designer’s job is completely immeasurable during the design process, leaving the designer and his or her colleagues to make their decisions based entirely on their own professional and expert opinions.

• Leave a Note (3)

The classic Skinner box experiment.

A comment by Steve Baty on my recent series about design research got me thinking about the fine art of experimental research design:

Where research (in all forms) becomes a waste of time and effort is when the research design and methodology applied invalidate the conclusions _before_ they can be drawn.

One of the things that defines a great scientific thinker is their ability to design and construct meaningful experimental methodologies. Defining the terms and tools of an experiment in such a way as to focus on just the right question while excluding bad data, well, this is where science becomes a high art. Whether it’s physics, medicine, psychology/neurology, human behavior, whatever, the structure of a great experiment is often as interesting, surprising, and elegant as the plot of a great novel — and the insights such experiments reveal are often astonishing.

Sometimes the results of powerful experiments open more questions (often deeper, and more significant questions) than they answer.

I think of things like the Turing Test (measuring the quality of an Artificial Intelligence) or the Double-Slit Experiment (illustrating the quantum nature of light). One of my favorites is a kind of test for autism:

Typically, the child watches two characters in a scene. The first person (A) places some chocolate in a drawer. A leaves the room and B (the second person) moves the chocolate to another drawer. A returns and the observer/child is asked to guess where A believes the chocolate is. Young children (< 3 years) and persons with autism mistakenly think that A will look in the new drawer, whereas older children will correctly claim that A will look in the old hiding place.

Another favorite of mine is the controversial experiment by Benjamin Libet in which it is shown that humans begin an action a full half a second after before their conscious mind makes the decision to act. A shocking conclusion, one that throws our understanding of consciousness and free will for quite a loop. But how in the world can this be measured? Libet’s methodology is ingenius:

Libet asked his experimental subjects to move one hand at an arbitrary moment decided by them, and to report when they made the decision (they timed the decision by noticing the position of a dot circling a clock face). At the same time the electrical activity of their brain was monitored. Now it had already been established by much earlier research that consciously-chosen actions are preceded by a pattern of activity known as a Readiness Potential (or RP). The surprising result was that the reported time of each decision was consistently a short period (some tenths of a second)after the RP appeared.

What are your favorite scientific experiments? I’d love to hear about them.

• Leave a Note (3)

(This is Part 5 — the final part. Please read Part 1 , Part 2 , Part 3 , and Part 4 first.)

I would certainly agree that more rigorous methodologies can’t hurt in our field. But at the same time, I think that we need to be a little more honest about the value of some less-rigorous methodologies, techniques that are (and always have been) extremely helpful to the user experience designer. Card sorting, focus groups, guerilla (or gorilla — thanks Keith!) usability testing, and user personas (even the ad hoc kind) can provide invaluable insights and useful tools for a design team, even as they are entirely subjective and even a little touchy-feely in nature.

For example: I’ve been working closely lately with a major branding agency who conducted two weeks of field research into how their client (our client, too) was perceived by their potential audience. The research was well-organized, the participants were carefully selected, the questions were well-crafted, and the results were carefully organized and reported.

What was particularly refreshing to me, as a “customer” of their research results, was that the branding agency was upfront and honest that the final report, despite their care, was “not scientific”. It was research, but it was entirely qualitative and subjective. The final report was comprised of the branding experts’ opinions and interpretations of the data they collected. And it is incredibly useful to the rest of us working on the project because it give us a better foundation upon which to build our creative ideas for the design, the UI, and the business strategies.
MORE…

• Leave a Note (12)

(This is Part 4. Please read Part 1 , Part 2, and Part 3 first.)

Okay, in this post I’m going to get a little down and dirty. I’ll show some examples of research which do not seem to really enable a design team to learn more about their users, nor convince stakeholders about correct design decisions, but which seem instead intended to put up a facade of scientific truth in order to create a more expensive service offering.

Have a Beer in One of Our “Persona Rooms”

I really really don’t want to take cheap shots at my company’s competitors, nor do I feel comfortable about the idea of ridiculing a leading industry researcher… but when I read about this, my jaw dropped to the floor.

Organic, an excellent top-tier web design agency, was recently featured in a Forrester research paper for their innovative work in the area of user personas, and they’ve been getting a lot of positive buzz about it.

Their innovation? “Persona Rooms“: In addition to building standard user personas, Organic also hires professional set designers to build actual living room spaces in their office, one living room for each user persona. The design team then spends time hanging out in these rooms, gaining a deep understanding of what it’s like to be “Bob, the single dude” or “Mary, the homemaker”. They then use this experience to inform their design process.

Any alarms going off yet?
MORE…

• Leave a Note (13)

(This is Part 3. Please read Part 1 and Part 2 first.)

Explaining it to the boss.

Next time you read an article about a user research success story, ask yourself if the conclusions of that research weren’t just common sense (or at least common sense to good UI designers) to begin with. Ask yourself if a good designer couldn’t have concluded the same conclusion that the user research seemed to reach.

Then ask yourself if you could articulate your “common sense” recommendation to a person who doesn’t understand design at all. To someone who may, in fact, be hostile to your so-called “expert” recommendations?

This is one area where research can help: explaining a user interface design strategy to stakeholders, peers, and bosses who have their own agendas and biases.
MORE…

• Leave a Note (11)

Here’s a great and succint case study of how user research data can be easily misinterpreted, and a great example about why we should always be suspicious of statistics.

The marketing blog at FutureLab (which I do recommend) has a short post today entitled “Study Shows Fear of MySpace Predators is Overblown“.

The research paper (PDF) referenced in the post makes the following conclusions:

• Only 7% of those teens interviewed were ever approached by anyone with a sexual intent and nearly all of them simply ignored the person and blocked him from their page.
• Two-thirds of the parents were sure that there were many sexual predators on MySpace, while only one-third of the teenagers shared this concern.
• When asked about media coverage, 66% of the parents felt that it was either understated or close to the truth.
• Conversely, 58% of the teens felt it was vastly overblown.

At this point, several alarm bells should be going off in your head:
MORE…

• Leave a Note (3)

(This is Part 2. Please read Part 1 first.)

An eyetracking “heatmap” showing in red where users’ eyes were pointing for the longest time during a page-view.

There is a limit, I think, to what a so-called “empirical” user interface test can tell you. At some point, the results must be interpreted in order to be useful as a design tool — and interpretations can easily go wrong. They can overlook a critical objective or even reach the wrong conclusions, especially when interpreted by people without the appropriate design skills.

Eyetracking (breathlessly called the “future of web design” by one writer) is a great example of a “design tool” that is getting a lot of buzz lately. Jared Spool says it’s probably not worth it, and I tend to agree with him there. My first objection is that an eyetracker can tell you what people are looking at, but not necessarily what they are seeing (or why they are looking at it). Secondly, the results generated by an eyetracking study are, to a good UI designer, rarely surprising. Finally, as with any analysis of subjective experience, the results can be easily misinterpreted.
MORE…

• Leave a Note (18)

Research-based design is a noble and widely-admired approach to building good products, especially in the web design field.

Like a great many other user experience design firms, at Behavior we conduct research whenever possible, to whatever degree our clients’ budgets and timelines will allow. Our projects frequently involve usability testing (both lab-based and informal), card-sorting exercises, stakeholder interviews, user polls and quantitative analysis, direct ethnographic studies and contextual analysis, and/or secondary market research.

In short, we try to know as much as possible about our clients, their customers, and their competitors, and we use this knowledge to inform our design process.

Many web designers and consultancies, however, feel it’s not enough to use research to inform their design process. They go further: they try to make “scientific” user research the very foundation of their design process.

I use the word “try” because I suspect that the ideal of empirical, science-based user-centered design is something that we aspire to but never reach.

That’s me being generous. The cynic in me would not have used the word “try”. The cynic would say “pretend”, as in “many firms pretend to use scientific user research as the foundation of their design process”. I don’t want to seem like I’m taking petty swipes at competitors, but honestly there’s no way to say this without being plain about it: I suspect that a lot of user research in this industry is a sham.
MORE…

• Leave a Note (18)

A slight change of focus…

(also check out Part 1 Part 2 and Part 3)

You may recall that a nefarious global spying program called “Total Information Awareness”, spearheaded by convicted Iran-Contra criminal Vice Adm. John M. Poindexter, was exposed in 2002 by the New York Times.

After this program was made public, there was a great outcry by Senators, Congressmen, and pundits both from the left and the right. The consensus at the time was that the program, as conceived, was illegal and would require new legislation in order to become legal.

The Republican-led Congress decided they did not want to change our existing civil liberties legislation (except for what they passed in the Patriot Act), so in 2003 they banned the TIA program and Poindexter was let go. In fact, Congress even banned programs like TIA, specifying that they would not fund…

“…the program known either as Terrorism Information Awareness or Total Information Awareness, or any successor program, funded by the Defense Advanced Research Projects Agency, or any other Department or element of the Federal Government…”

Much of the new NSA program resembles Total Information Awareness, and many suspect they simply moved the program from the Defense department to the NSA, where Congress wouldn’t know about it.

It looks like the Bush Administration learned a valuable lesson: If you want something that Congress or existing laws won’t let you have, then just do it on your own and don’t tell anyone about it. MORE…

• Leave a Note (1)